Cavium QL41112HLCU-BK, User Manual

Page 1: ...AH0054601 00 B User s Guide Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series ...

Page 2: ...clarification Supported Operating Systems and OFED on page 57 Updated Step 2 to add the Dynamic option Preparing the Adapter on page 59 Updated Table 6 2 to add the Quality of Service property Configuring RoCE on the Adapter for Windows Server on page 64 Added Configuring RoCE on the Adapter for ESX to chapter 6 Configuring RoCE on the Adapter for ESX on page 80 Updated the procedure introduction ...

Page 3: ...ase xxi Legal Notices xxi Warranty xxi Laser Safety FDA Notice xxii Agency Certification xxii EMI and EMC Requirements xxii KCC Class A xxiii Product Safety Compliance xxiv 1 Product Overview Functional Description 1 Features 1 Adapter Management 3 QLogic Control Suite CLI 4 QLogic QConvergeConsole GUI 4 QLogic QConvergeConsole vCenter Plug in 4 QConvergeConsole PowerKit 5 FastLinQ ESXCLI VMware P...

Page 4: ... 17 Linux Driver Optional Parameters 18 Linux Driver Operation Defaults 18 Linux Driver Messages 19 Statistics 19 Installing Windows Driver Software 19 Installing the Windows Drivers 20 Removing the Windows Drivers 20 Managing Adapter Properties 21 Setting Power Management Options 22 Installing Drivers for Windows Nano Server 22 Creating a Nano ISO Image Injecting Drivers and Updating the Multiboo...

Page 5: ...itch 61 Configuring the Arista 7060X Ethernet Switch 63 Configuring RoCE on the Adapter for Windows Server 64 Configuring RoCE on the Adapter for Linux 68 RoCE Configuration for RHEL 68 RoCE Configuration for SLES 69 RoCE Configuration for Ubuntu 69 Verifying the RoCE Configuration on Linux 72 VLAN Interfaces and GID Index Values 74 RoCE v2 Configuration for Linux 75 Identifying RoCE v2 GID Index ...

Page 6: ...04 Configuring the Initiator 110 Optimizing Linux Performance 111 Configuring CPUs to Maximum Performance Mode 111 Configuring Kernel sysctl Settings 112 Configuring IRQ Affinity Settings 112 Configuring Block Device Staging 112 9 iSCSI Configuration iSCSI Boot 114 iSCSI Boot Setup 114 Selecting the Preferred iSCSI Boot Mode 114 Configuring the iSCSI Target 114 Configuring iSCSI Boot Parameters 11...

Page 7: ...h Dump 147 iSCSI Offload in Linux Environments 148 Differences from bnx2i 148 Configuring qedi ko 149 Verifying iSCSI Interfaces in Linux 149 Open iSCSI and Boot from SAN Considerations 152 10 FCoE Configuration FCoE Boot from SAN 157 Preparing System BIOS for FCoE Build and Boot 158 Specifying the BIOS Boot Protocol 158 Configuring Adapter UEFI Boot Mode 158 Windows FCoE Boot from SAN 163 Windows...

Page 8: ...E over Switch Embedded Teaming 203 Creating a Hyper V Virtual Switch with SET and RDMA Virtual NICs 204 Enabling RDMA on SET 204 Assigning a VLAN ID on SET 204 Running RDMA Traffic on SET 205 Configuring QoS for RoCE 205 Configuring QoS by Disabling DCBX on the Adapter 205 Configuring QoS by Enabling DCBX on the Adapter 210 Configuring VMMQ 213 Enabling VMMQ on the Adapter 214 Setting the VMMQ Max...

Page 9: ...Hosts 235 Starting the Remote Windows PowerShell Session 235 Managing QLogic Adapters on a Windows Nano Server 236 RoCE Configuration 236 14 Troubleshooting Troubleshooting Checklist 239 Verifying that Current Drivers Are Loaded 240 Verifying Drivers in Windows 240 Verifying Drivers in Linux 240 Verifying Drivers in VMware 241 Testing Network Connectivity 241 Testing Network Connectivity for Windo...

Page 10: ...User s Guide Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 41000 Series x AH0054601 00 B Index ...

Page 11: ...iSCSI Offload 51 5 16 Partitions Configuration Page with FCoE Offload and iSCSI Offload 52 5 17 Global Bandwidth Allocation Page 53 5 18 Partition 1 Configuration 54 5 19 Partition 2 Configuration FCoE Offload 55 5 20 Partition 3 Configuration iSCSI Offload 56 5 21 Partition 4 Configuration Ethernet 56 6 1 Configuring RoCE Properties 66 6 2 Switch Settings Server 78 6 3 Switch Settings Client 79 6...

Page 12: ...r Discover Target Portal 142 9 22 Target Portal IP Address 143 9 23 Selecting the Initiator IP Address 144 9 24 Connecting to the iSCSI Target 145 9 25 Connect To Target Dialog Box 146 10 1 System Utilities 158 10 2 System Configuration Port Selection 159 10 3 Port Level Configuration 159 10 4 Boot Mode in Port Level Configuration 160 10 5 FCoE Offload Enabled 161 10 6 Selecting General Parameters...

Page 13: ...NetAdapter 204 13 9 Advanced Properties Enable QoS 206 13 10 Advanced Properties Setting VLAN ID 207 13 11 Advanced Properties Enabling QoS 211 13 12 Advanced Properties Setting VLAN ID 212 13 13 Advanced Properties Enabling Virtual Switch RSS 214 13 14 Advanced Properties Setting VMMQ 215 13 15 Virtual Switch Manager 216 13 16 Windows PowerShell Command Get VMSwitch 217 13 17 Advanced Properties ...

Page 14: ...rs 28 3 8 VMware Driver Parameter Defaults 29 3 9 QLogic 41000 Series Adapter VMware FCoE Driver 30 3 10 QLogic 41000 Series Adapter iSCSI Driver 31 5 1 Adapter Properties 38 6 1 OS Support for RoCE RoCEv2 iWARP iSER and OFED 57 6 2 Advanced Properties for RoCE 65 9 1 Configuration Options 115 9 2 DHCP Option 17 Parameter Definitions 129 9 3 DHCP Option 43 Sub option Definitions 130 9 4 DHCP Optio...

Page 15: ...This guide is intended for system administrators and other technical staff members responsible for configuring and managing adapters installed on servers in Windows Linux or VMware environments What Is in This Guide Following this preface the remainder of this guide is organized into the following chapters and appendices Chapter 1 Product Overview provides a product functional description a list o...

Page 16: ... describes Fibre Channel over Ethernet FCoE boot from SAN and booting from SAN after installation Chapter 11 SR IOV Configuration provides procedures for configuring single root input output virtualization SR IOV on Windows Linux and VMware systems Chapter 12 NVMe oF Configuration with RDMA demonstrates how to configure NVMe oF on a simple network Chapter 13 Windows Server 2016 describes the Windo...

Page 17: ...rgeConsole FastLinQ PowerKit to manage the QLogic FastLinQ adapters on the system User s Guide QConvergeConsole Plug ins for vSphere part number SN0054677 00 provides details for using the two plug ins to extend the capabilities of VMware vCenter Server and the vSphere Web Client User s Guide FastLinQ ESXCLI VMware Plug in part number BC0151101 00 describes the plug in that extends the capabilitie...

Page 18: ...h or command line text For example To return to the root directory from anywhere in the file structure Type cd root and press ENTER Issue the following command sh install bin Key names and key strokes are indicated with UPPERCASE Press CTRL P Press the UP ARROW key Text in italics indicates terms emphasis variables or document titles For example For a complete listing of license agreements refer t...

Page 19: ... repeated For example x means one or more instances of x x means zero or more instances of x Vertical ellipses within command example output indicate where portions of repetitious output data have been intentionally omitted parentheses and braces are used to avoid logical ambiguity For example a b c is ambiguous a b c means a or b followed by c a b c means either a or b c License Agreements Refer ...

Page 20: ...re that you have the correct firmware software or documentation For additional information click Read Me and Release Notes under Support Files 5 Click Download Now 6 Save the file to your computer 7 If you have downloaded firmware software drivers or boot code follow the installation instructions in the Read Me file Instead of typing a model name in the search box you can perform a guided search a...

Page 21: ...of information in the database to provide answers to your most urgent questions Access the database from the QLogic Support Center support qlogic com Legal Notices Legal notices covered in this section include warranty laser safety FDA notice agency certification and product safety compliance Warranty For warranty details please check the QLogic Web site www qlogic com Support Pages Warranty aspx ...

Page 22: ... cause undesired operation ICES 003 Compliance Class A This Class A digital apparatus complies with Canadian ICES 003 Cet appareil numériqué de la classe A est conformé à la norme NMB 003 du Canada CE Mark 2014 30 EU 2014 35 EU EMC Directive Compliance EN55032 2012 CISPR 32 2015 Class A EN55024 2010 EN61000 3 2 Harmonic Current Emission EN61000 3 3 Voltage Fluctuation and Flicker Class 1 Laser Pro...

Page 23: ...s A Korea RRA Class A Certified Korean Language Format Class A Product Name Model Converged Network Adapters and Intelligent Ethernet Adapters Certification holder QLogic Corporation Manufactured date Refer to date code listed on product Manufacturer Country of origin QLogic Corporation USA A class equipment Business purpose info telecommunications equipment As this equipment has undergone EMC reg...

Page 24: ... 14 CSA C22 2 No 60950 1 07 2nd Edition A1 A2 2014 10 Use only with listed ITE or equivalent Complies with 21 CFR 1040 10 and 1040 11 2014 30 EU 2014 35 EU 2006 95 EC low voltage directive TUV EN60950 1 2006 A11 A1 A12 A2 2nd Edition TUV IEC 60950 1 2005 2nd Edition Am1 2009 Am2 2013 CB CB Certified to IEC 60950 1 2nd Edition ...

Page 25: ...our network into virtual LANs VLANs as well as group multiple network adapters together into teams to provide network load balancing and fault tolerance For more information about teaming see your operating system documentation Features The 41000 Series Adapters provide the following features Some features may not be available on all adapters NIC partitioning NPAR FCoE offload iSCSI offload Univer...

Page 26: ...t offload LSO Generic segment offload GSO HW based Generic Receive Offload HW GRO Large receive offload LRO Receive segment coalescing RSC Microsoft dynamic virtual machine queue VMQ and Linux multiqueue Adaptive interrupts Transmit receive side scaling TSS RSS Stateless offloads for Network Virtualization using Generic Routing Encapsulation NVGRE and virtual LAN VXLAN L2 L3 GRE tunneled traffic1 ...

Page 27: ...Serial flash NVRAM memory PCI Power Management Interface v1 1 64 bit base address register BAR support EM64T processor support iSCSI and FCoE boot support2 Adapter Management The following applications are available to manage 41000 Series Adapters QLogic Control Suite CLI QLogic QConvergeConsole GUI QLogic QConvergeConsole vCenter Plug in QConvergeConsole PowerKit FastLinQ ESXCLI VMware Plug in Fo...

Page 28: ...g QCC GUI see the Installation Guide QConvergeConsole GUI For information about using the QCC GUI see the online help QLogic QConvergeConsole vCenter Plug in The QConvergeConsole vCenter Plug in is a Web based management tool that is integrated into the VMware vCenter Server for configuring and managing QLogic Fibre Channel adapters Converged Network Adapters and Intelligent Ethernet Adapters in a...

Page 29: ...ple instructions together In addition to being a powerful scripting tool the QLogic PowerKit comes with a selection of preconfigured cmdlets scripts that perform a single function to monitor and manage your QLogic FastLinQ adapters For information about installing and using the QConvergeConsole PowerKit refer to the User s Guide PowerShell FastLinQ ESXCLI VMware Plug in The FastLinQ ESXCLI plug in...

Page 30: ...ss Card Electromechanical Specification rev 3 0 PCI Bus Power Management Interface Specification rev 1 2 IEEE Specifications 802 3 2012 IEEE Standard for Ethernet flow control 802 1q VLAN 802 1AX Link Aggregation 802 1ad QinQ 802 1p Priority Encoding 1588 2002 PTPv1 Precision Time Protocol 1588 2008 PTPv2 IEEE 802 3az Energy Efficient Ethernet EEE IPv4 RFQ 791 IPv6 RFC 2460 ...

Page 31: ...7 AH0054601 00 B 2 Hardware Installation This chapter provides the following hardware installation information System Requirements Safety Precautions Preinstallation Checklist Installing the Adapter ...

Page 32: ...Ie Gen2 x8 2x10G NIC PCIe Gen3 x8 2x25G NIC Full dual port 25Gb bandwidth is supported on PCIe Gen3 x8 or faster slots Memory 8GB RAM minimum Cables and Optical Modules The 41000 Series Adapters have been tested for interoperability with a variety of 1G 10G and 25G cables and optical modules See Tested Cables and Optical Modules on page 247 Table 2 2 Minimum Host Operating System Requirements Oper...

Page 33: ...driverdownloads qlogic com WARNING The adapter is being installed in a system that operates with voltages that can be lethal Before you open the case of your system observe the following precautions to protect yourself and to prevent damage to the system components Remove any metallic objects or jewelry from your hands and wrists Make sure to use only insulated or nonconducting tools Verify that t...

Page 34: ...er can be seated into a greater width slot x8 in an x16 but a greater width adapter cannot be seated into a lesser width slot x8 in an x4 If you do not know how to identify a PCIe slot refer to your system documentation 3 Remove the blank cover plate from the slot that you selected 4 Align the adapter connector edge with the PCI Express connector slot in the system 5 Applying even pressure at both...

Page 35: ...ver Operation Defaults Linux Driver Messages Statistics The 41000 Series Adapter Linux drivers and supporting documentation are available on the QLogic Downloads and Documentation page driverdownloads qlogic com Table 3 1 describes the 41000 Series Adapter Linux drivers Table 3 1 QLogic 41000 Series Adapters Linux Drivers Linux Driver Description qed The qed core driver module directly controls th...

Page 36: ... for the 41000 Series Adapter This driver directly controls the hard ware and is responsible for sending and receiving Ethernet packets on behalf of the Linux host networking stack This driver also receives and processes device interrupts on behalf of itself for L2 networking The qede driver requires Linux kernel version 2 6 32 or later Testing was concentrated on the x86_64 architecture qedr Linu...

Page 37: ...Using the kmp kmod RPM Package Installing Linux Drivers Using the TAR File Removing the Linux Drivers There are two procedures for removing Linux drivers one for a non RDMA environment and another for an RDMA environment Choose the procedure that matches your environment NOTE For network installations through NFS FTP or HTTP using a network boot disk a driver disk that contains the qede driver may...

Page 38: ...e installed using a TAR file issue the following commands rmmod qede rmmod qed depmod a For RHEL and CentOS cd lib modules version extra qlgc fastlinq rm rf qed ko qede ko qedr ko For SLES cd lib modules version updates qlgc fastlinq rm rf qed ko qede ko qedr ko For Ubuntu issue the following command dpkg r fastlinq dkms To remove Linux drivers in an RDMA environment 1 Unload and remove the Linux ...

Page 39: ...ackage 1 Issue the following at a command prompt rpm ivh qlgc fastlinq version src rpm 2 Change the directory to the RPM path and build the binary RPM for the kernel For RHEL and CentOS cd root rpmbuild rpmbuild bb SPECS fastlinq version spec For SLES cd usr src packages rpmbuild bb SPECS fastlinq version spec 3 Install the newly compiled RPM rpm ivh RPMS arch qlgc fastlinq version arch rpm The dr...

Page 40: ...2 Reload the driver modprobe r qede modprobe qede Installing Ubuntu Linux Drivers To install Ubuntu Linux drivers issue the following command dpkg i fastlinq dkms_ version _all deb Installing Linux Drivers Using the TAR File To install Linux drivers using the TAR file 1 Create a directory and extract the TAR files to the directory tar xjvf fastlinq version tar bz2 2 Change to the recently created ...

Page 41: ...ge 13 4 Install the new Linux drivers using one of the following methods Installing Linux Drivers Using the kmp kmod RPM Package Installing Linux Drivers Using the TAR File 5 Install libqedr libraries to work with RDMA user space applications The libqedr RPM is available only for inbox OFED You must select which RDMA RoCE RoCEv2 or iWARP is used in UEFI until concurrent RoCE iWARP capability is su...

Page 42: ... ethtool s dev msglvl int_mode Controls interrupt mode other than MSI X gro_enable Enables or disables the hardware generic receive offload GRO feature This feature is similar to the kernel s software GRO but is only performed by the device hardware err_flags_override A bitmap for disabling or forcing the actions taken in case of a hardware error bit 31 An enable bit for this bitmask bit 0 Prevent...

Page 43: ...formation Installing Windows Driver Software For information on iWARP see Chapter 7 iWARP Configuration Installing the Windows Drivers Removing the Windows Drivers Managing Adapter Properties Setting Power Management Options Installing Drivers for Windows Nano Server Creating a Nano ISO Image Injecting Drivers and Updating the Multiboot Flash Image on a Nano Server Flow Control Auto negotiation wi...

Page 44: ...ify that the Windows drivers have been installed a Click Start and then click Control Panel b In the Control Panel click Programs and then click Programs and Features c In the installed programs list locate QLogic FastLinQ Driver Installer Removing the Windows Drivers To remove the Windows drivers 1 In the Control Panel click Programs and then click Programs and Features 2 In the list of programs ...

Page 45: ... change the 41000 Series Adapter properties 1 In the Control Panel click Device Manager 2 On the properties of the selected adapter click the Advanced tab 3 On the Advanced page Figure 3 1 select an item under Property and then change the Value for that item as needed Figure 3 1 Setting Advanced Adapter Properties ...

Page 46: ...Allow the computer to turn off the device to save power check box Figure 3 2 Figure 3 2 Power Management Options Installing Drivers for Windows Nano Server To install the drivers for Windows Nano Server 1 Add and install the driver package by issuing the following command pnputil exe i a a usbcam USBCAM INF NOTE The Power Management page is available only for servers that support power management ...

Page 47: ...nject the QLogic drivers into the Nano image a Download the Windows Driver Installer from the QLogic Downloads page and extract the relevant drivers driverdownloads qlogic com Table 3 4 Windows Drivers Windows Driver Description QeVBD Core driver QeND Ethernet networking driver QeOIS iSCSI Offload driver QeFCoE FCoE Offload driver QxDiag Diagnostics driver NOTE The oem0 inf may not always be the d...

Page 48: ... Upgrade Utility in a Nano Server environment a Copy the utility into the Nano Server b After creating Nano Server image mount it and copy the utility c After connected remotely navigate to the utility directory and then run the utility NOTE You will use the QLogic drivers files during Nano Server image creation For more information on a specific command refer to the Injecting Drivers section in t...

Page 49: ...ols The ESXi drivers are included as individual driver packages and are not bundled together Table 3 6 lists the ESXi versions and applicable driver versions Table 3 5 VMware Drivers VMware Drivers Description qedentv Native networking driver qedrntv Native RDMA Offload RoCE and RoCEv2 driver a a The certified RoCE driver is not included in this release The uncertified driver may be available as a...

Page 50: ...tire driver set from the same driver ZIP file Mixing drivers from different ZIP files will cause problems To install the VMware driver 1 Download the VMware driver for the 41000 Series Adapter from the VMware support page www vmware com support html 2 Power up the ESX host and then log into an account with administrator authority 3 Unzip the driver ZIP file and then extract the vib file 4 Use the ...

Page 51: ...are vib install v tmp qedentv 1 0 3 11 1OEM 550 0 0 1331820 x86_64 vi b To install the vib file using the VUM see the knowledge base article here Updating an ESXi ESX host using VMware vCenter Update Manager 4 x and 5 x 1019545 Option 2 Install all of the individual VIBs at one time by issuing the following command esxcli software vib install d tmp qedentv bundle 2 0 3 zip To upgrade an existing d...

Page 52: ... 4 or one of the following values 1 uses the default number of RX filters per queue 0 disables RX filters disable_tpa Enables 0 or disables 1 the TPA LRO feature disable_tpa 0 is the default max_vfs Specifies the number of virtual functions VFs per physical function PF max_vfs can be 0 disabled or 64 VFs on a single port enabled The 64 VF maximum support for ESXi is an OS resource allocation const...

Page 53: ...rt or multifunction configurations enable_vxlan_offld Enables 1 or disables 0 the VXLAN tunneled traffic checksum offload and TCP segmentation offload TSO capability The default is enable_vxlan_offld 1 You can specify multiple values delimited by commas for multiport or multifunction configurations Table 3 8 VMware Driver Parameter Defaults Parameter Default Speed Autonegotiation with all speeds a...

Page 54: ... VMware software package to support QLogic FCoE converged network interface controllers C NICs The FCoE and DCB feature set is supported on VMware ESXi 5 0 and later Number of Queues Numbered value Wake on LAN WoL Disabled Table 3 9 QLogic 41000 Series Adapter VMware FCoE Driver Driver Description qedf The QLogic VMware FCoE driver is a kernel mode driver that pro vides a translation layer between...

Page 55: ...ter iSCSI Driver Driver Description qedil The qedil driver is the QLogic VMware iSCSI HBA driver Similar to qedf qedil is a kernel mode driver that provides a translation layer between the VMware SCSI stack and the QLogic iSCSI firmware and hardware qedil leverages the services provided by the VMware iscsid infrastructure for session management and IP services ...

Page 56: ...ore information about these utilities their command syntax and parameters see their respective ReadMe files This chapter includes the following firmware upgrade information Image Verification Upgrading Adapter Firmware on Linux Upgrading Adapter Firmware on Windows Upgrading Adapter Firmware on Windows Nano Image Verification During firmware updates the image is verified by comparing a digital sig...

Page 57: ...lled All files should reside in the same folder 4 Confirm that the adapter port link status is up ifconfig ethx up 5 Run the Linux upgrade utility script by issuing the following command LnxQlgcUpg sh The flash update script should display information to the console indicating that it is proceeding If the utility is unable to identify the installed adapter ensure that the system is running the lat...

Page 58: ...nload the Firmware Upgrade Utility for Windows from QLogic driverdownloads qlogic com 3 Unzip the Firmware Upgrade Utility on the system where the adapter is installed All files should reside in the same folder 4 Copy the Firmware Upgrade Utility files onto the Windows Nano Server system 5 Remote into Windows Nano Server by PowerShell 6 Run the Windows upgrade utility script by issuing the followi...

Page 59: ...Infrastructure Interface HII application These tasks include the following Displaying Firmware Image Properties Configuring Device level Parameters Configuring Port level Parameters Configuring FCoE Boot Configuring iSCSI Boot Configuring Partitions NOTE The HII screen shots in this chapter are representative and may not match the screens that you see on your system ...

Page 60: ...or information about launching the System Setup consult the user guide for your system 2 In the System Setup window Figure 5 1 select Device Settings and then press ENTER Figure 5 1 System Setup 3 In the Device Settings window Figure 5 2 select the 41000 Series Adapter port that you want to configure and then press ENTER Figure 5 2 System Setup Device Settings ...

Page 61: ...B 4 The Main Configuration Page presents the adapter management options where you can set the partitioning mode If you are not using NPAR set the Partitioning Mode to Default as shown in Figure 5 3 Figure 5 3 Main Configuration Page Setting Default Partitioning Mode ...

Page 62: ...s on page 41 Port Level Configuration see Configuring Port level Parameters on page 42 iSCSI Configuration if iSCSI remote boot is available see Configuring iSCSI Boot on page 48 FCoE Configuration if FCoE boot from SAN is available see Configuring FCoE Boot on page 46 Partitions Configuration if NPAR is selected as the Partitioning Mode see Configuring Partitions on page 51 In addition the Main C...

Page 63: ...I Virtual MAC Address User defined device iSCSI Offload MAC address FCoE MAC Address Manufacturer assigned permanent device FCoE Offload MAC address FCoE Virtual MAC Address User defined device FCoE Offload MAC address FCoE WWPN Manufacturer assigned permanent device FCoE Offload WWPN world wide port name FCoE Virtual WWPN User defined device FCoE Offload WWPN FCoE WWNN Manufacturer assigned perma...

Page 64: ...n the Main Configuration Page and then press ENTER The Firmware Information page Figure 5 5 specifies the following view only data Family Firmware Version is the multiboot image version which comprises several firmware component images MFW Version is the management firmware version UEFI Driver Version is the unified extensible firmware interface UEFI driver version Figure 5 5 Firmware Information ...

Page 65: ...iguration Page 3 The SR IOV parameter enables Enabled or disables Disabled SR IOV 4 MFW Crash Dump Feature enables Enabled or disables Disabled the collecting of adapter specific running state information when specific system crashes occur 5 UEFI Driver Debug Level sets the UEFI runtime debug messages that are saved to the qdbg log file similar to the Linux dmesg logs Possible values are 0 to 0xFF...

Page 66: ...nk Up Delay RDMA Protocol Support To configure the port level parameters 1 On the Main Configuration Page select Port Level Configuration and then press ENTER 2 For Link Speed Figure 5 7 select one of the following values where the last specified value becomes the default until a different value is selected Auto Negotiated 10 Gbps 25 Gbps SmartAN see description below NOTE For a 10G SFP and 10GBAS...

Page 67: ... method to auto negotiate between a 10G switch and a 25G adapter or between DACs and optics Cavium s SmartAN provides an automatic and convenient method to detect the switch and to determine and set the link speed FEC types media type and length and so on Although SmartAN is most useful for connectivity with a 10G switch and for determining the type of transceiver that is inserted in the adapter c...

Page 68: ...ns 4 For Boot Mode select values for the port level parameters as shown in Figure 5 8 PXE enables PXE boot FCoE enables FCoE boot from SAN over the hardware offload pathway iSCSI SW enables iSCSI remote boot over the software pathway iSCSI HW enables iSCSI remote boot over the hardware offload pathway Disabled prevents this port from being used as a remote boot source Figure 5 8 Port Level Configu...

Page 69: ...fload 8 For FCoE Offload select either Enabled or Disabled This setting is only visible on 41000 Series Adapters that are enabled with FCoE Offload 9 For PXE VLAN Mode select either Enabled or Disabled 10 For PXE VLAN ID select a VLAN tag value from 1 to 4094 This value must match the network settings of the attached switch This control appears only when PXE VLAN Mode is Enabled PXE VLAN ID applie...

Page 70: ...try Count options include Connect 1 8 is Enabled or Disabled WWPN 1 8 shows the WWPN in use all zeros if the target is offline A indicates that the specified target is online while a indicates that the specified target is offline Boot LUN 1 8 shows the boot LUN 0 255 that is used by the target array To configure the FCoE boot configuration parameters 1 On the Main Configuration Page select FCoE Bo...

Page 71: ...054601 00 B 3 Choose values for the FCoE General or FCoE Target Configuration parameters Figure 5 9 FCoE General Parameters Figure 5 10 FCoE Target Configuration 4 Click Back 5 When prompted click Yes to save the changes Changes take effect after a system reset ...

Page 72: ... 3 Choose values for the appropriate iSCSI configuration parameters iSCSI General Configuration Figure 5 11 on page 50 TCP IP Parameters Via DHCP iSCSI Parameters Via DHCP CHAP Authentication CHAP Mutual Authentication IP Version ARP Redirect DHCP Request Timeout Target Login Timeout DHCP Vendor ID iSCSI Initiator Configuration Figure 5 12 on page 50 IPv4 Address IPv4 Subnet Mask IPv4 Default Gate...

Page 73: ...pter Preboot Configuration Configuring iSCSI Boot 49 AH0054601 00 B iSCSI Second Target Configuration Figure 5 14 on page 51 Connect IPv4 Address TCP Port Boot LUN iSCSI Name CHAP ID CHAP Secret 4 Click Back ...

Page 74: ...g iSCSI Boot 50 AH0054601 00 B 5 When prompted click Yes to save the changes Changes take effect after a system reset Figure 5 11 iSCSI General Configuration Figure 5 12 iSCSI Initiator Configuration Figure 5 13 iSCSI First Target Configuration ...

Page 75: ...apter To configure the maximum and minimum bandwidth allocations 1 On the Main Configuration Page select Partitions Configuration and then press ENTER 2 On the Partitions Configuration page select Global Bandwidth Allocation Figure 5 15 shows the Global Bandwidth Configuration page when FCoE Offload and iSCSI Offload are disabled Figure 5 15 Partitions Configuration Page No FCoE Offload or iSCSI O...

Page 76: ...guration Configuring Partitions 52 AH0054601 00 B Figure 5 16 shows the page when NPAR mode is enabled with FCoE Offload and iSCSI Offload enabled Figure 5 16 Partitions Configuration Page with FCoE Offload and iSCSI Offload ...

Page 77: ...dth value The total of the minimum TX bandwidth values of all partitions on a single port must equal 100 or be all zeros Setting the TX bandwidth to all zeros is similar to equally dividing the available bandwidth over every active partition however the bandwidth is dynamically allocated over all actively sending partitions A zero value when one or more of the other values are set to a non zero va...

Page 78: ...Figure 5 18 which shows the following parameters Personality NIC Mode always enabled PCI Device ID PCI bus Address Permanent MAC Address Virtual MAC Address Partition 1 is always present and cannot be disabled Figure 5 18 Partition 1 Configuration 3 To configure the second partition select Partition 2 Configuration to open the Partition 2 Configuration page If FCoE Offload is present the Partition...

Page 79: ...tition 3 Configuration to open the Partition 3 Configuration page Figure 5 18 If iSCSI Offload is present the Partition 3 Configuration shows the Personality as iSCSI Figure 5 20 and the following additional parameters NIC Mode Disabled iSCSI Offload Mode Enabled iSCSI Offload MAC Address1 Virtual iSCSI Offload MAC Address 1 PCI Device ID 1 This parameter is only present on the third partition of ...

Page 80: ... page for a partition 2 or greater Ethernet partition The Personality show as Ethernet Figure 5 21 and includes the following additional parameters NIC Mode Enabled or Disabled When disabled the partition is hidden such that it does not appear to the OS if fewer than the maximum quantity of partitions or PCI PFs are detected PCI Device ID PCI Address Permanent MAC Address Virtual MAC Address Figur...

Page 81: ... on the Adapter for Linux Configuring RoCE on the Adapter for ESX Supported Operating Systems and OFED Table 6 1 shows the operating system support for RoCE v1 RoCE v2 iWARP and OFED NOTE Some RoCE features may not be fully enabled in the current release For details refer to Appendix C Feature Constraints Table 6 1 OS Support for RoCE RoCEv2 iWARP iSER and OFED Operating System Inbox OFED 3 18 3 G...

Page 82: ...installing libibverbs devel Running user space applications in inbox OFED requires installing the InfiniBand Support group by yum groupinstall InfiniBand Support that contains libibcm libibverbs and more OFED and RDMA applications that depend on libibverbs also require the QLogic RDMA user space library libqedr Install libqedr using the libqedr RPM or source packages RoCE supports only little endi...

Page 83: ...ically 0 is used for the default lossy traffic class 3 is used for the FCoE traffic class and 4 is used for lossless iSCSI TLV over DCB traffic class 4 Click Back 5 When prompted click Yes to save the changes Changes will not take effect until after a system reset For Windows you can configure DCBX using the HII or QoS method The configuration shown in this section is through HII For QoS refer to ...

Page 84: ...s switch config policy map type qos roce switch config class type qos class roce switch config set qos group 3 6 Configure queuing policy maps to assign network bandwidth In this example use a value of 50 percent switch config policy map type queuing roce switch config class type queuing class roce switch config bandwidth percent 50 7 Configure network QoS policy maps to set priority flow control ...

Page 85: ...ol and assigning a VLAN ID to the switch port To configure the Dell switch 1 Create a DCB map Dell configure Dell conf dcb map roce Dell conf dcbmap roce 2 Configure two ETS traffic classes in the DCB map with 50 percent bandwidth assigned for RoCE group 1 Dell conf dcbmap roce priority group 0 bandwidth 50 pfc off Dell conf dcbmap roce priority group 1 bandwidth 50 pfc on 3 Configure the PFC prio...

Page 86: ...ity BW BW COMMITTED BW PEAK TSA Rate Mbps Burst KB Rate Mbps Burst KB 0 0 1 2 3 4 6 7 40 ETS 1 5 60 ETS 2 3 Dell Conf do show interfaces twentyFiveGigE 1 8 1 pfc detail Interface twentyFiveGigE 1 8 1 Admin mode is on Admin is enabled Priority list is 5 Remote is enabled Priority list is 5 Remote Willing Status is enabled Local is enabled Priority list is 5 Oper status is init PFC DCBX Oper status ...

Page 87: ...onfiguring RoCE APP TLVs for both RoCE and RoCE v2 The following is an example on configuring RoCE APP TLV for both RoCE and RoCE v2 Arista 7060X EIT config dcbx application ether 0x8915 priority 5 Arista 7060X EIT config dcbx application udp 4791 priority 5 Mapping Priority to Traffic Class In the following example Priority 5 is mapped to traffic class 1 and the rest of the priorities are mapped ...

Page 88: ...he PFC is enabled for Priority 5 Arista 7060X EIT config interface ethernet 26 1 Arista 7060X EIT config if Et26 1 dcbx mode cee Arista 7060X EIT config if Et26 1 priority flow control mode on Arista 7060X EIT config if Et26 1 priority flow control priority 5 no drop Arista 7060X EIT config if Et26 1 switchport mode trunk Arista 7060X EIT config if Et26 1 switchport trunk allowed vlan 1 5 Configur...

Page 89: ... the jumbo packet size RDMA Mode RoCE or RoCE v2 The iWARP value applies only when configuring ports for iWARP as described in Chapter 7 iWARP Configuration VLAN ID Assign any VLAN ID to the interface The value must be the same as is assigned on the switch Quality of Service Enables or disables QoS Select Enabled if you are controlling DCB through the Windows DCB QoS service For more informa tion ...

Page 90: ...oth ports are enabled PS C Users Administrator Get NetAdapterRdma Name InterfaceDescription Enabled SLOT 4 3 Port 1 QLogic FastLinQ QL41262 True SLOT 4 3 Port 2 QLogic FastLinQ QL41262 True 3 Using Windows PowerShell verify that NetworkDirect is enabled on the host operating system The Get NetOffloadGlobalSetting command shows NetworkDirect is enabled PS C Users Administrators Get NetOffloadGlobal...

Page 91: ...ps technet microsoft com en us library hh831795 v ws 11 aspx 5 By default Microsoft s SMB Direct establishes two RDMA connections per port which provides good performance including line rate at a higher block size for example 64KB To optimize performance you can change the quantity of RDMA connections per RDMA interface to four or greater To increase the quantity of RDMA connections to four or mor...

Page 92: ...t be installed and configured on the RHEL host To prepare inbox OFED for RHEL 1 While installing or upgrading the operating system select the InfiniBand and OFED support packages 2 Install the following RPMs from the RHEL ISO image libibverbs devel x x x x86_64 rpm required for libqedr library perftest x x x x86_64 rpm required for InfiniBand bandwidth and latency applications or using Yum install...

Page 93: ...for Ubuntu To configure RoCE on the adapter for an Ubuntu host RDMA must be installed and configured on the Ubuntu host To configure and set up RoCE for Ubuntu 14 04 5 16 04 1 Linux 1 When you begin installing the Ubuntu server verify if the basic packages modules and tools are available for Ethernet and RDMA Log in as a ROOT and install all required packages a Install the basic packages required ...

Page 94: ... unlimited root soft memlock unlimited root hard memlock unlimited 4 Log into the system again or verify after reboot Then issue the following command ulimit l You should get the output as unlimited 5 Reboot the system 6 To allow the device to be recognized as an InfiniBand device that can be used by OFED install the FastLinQ package by issuing the following commands cd fastlinq X X X X make clean...

Page 95: ...ules by issuing the following commands You must perform this step whenever you reboot the system modprobe rdma_cm modprobe ib_uverbs modprobe rdma_ucm modprobe ib_ucm modprobe ib_umad 10 To list RoCE devices issue the ibv_devinfo command ibv_devinfo 11 Assign IP addresses to the Ethernet interfaces To assign the static IP address edit the etc network interfaces file with the following entries auto...

Page 96: ...and service rdma start If RDMA does not start issue either of the following alternative commands etc init d rdma start or systemctl start rdma service 3 Verify that the RoCE devices were detected by examining the dmesg logs dmesg grep qedr 87910 988411 qedr discovered and registered 2 RoCE funcs 4 Verify that all of the modules have been loaded For example lsmod grep qedr qedr 89871 0 qede 96670 1...

Page 97: ...art_id 5684 hw_ver 0x0 phys_port_cnt 1 port 1 state PORT_ACTIVE 1 max_mtu 4096 5 active_mtu 1024 3 sm_lid 0 port_lid 0 port_lmc 0x00 link_layer Ethernet 7 Verify the L2 and RoCE connectivity between all servers one server acts as a server another acts as a client Verify the L2 connection using a simple ping command Verify the RoCE connection by performing an RDMA ping on the server or client On th...

Page 98: ...he following commands where X is the device number mount t debugfs nodev sys kernel debug cat sys kernel debug qedr qedrX stats VLAN Interfaces and GID Index Values If you are using VLAN interfaces on both the server and the client you must also configure the same VLAN ID on the switch If you are running traffic through a switch the InfiniBand applications must use the correct GID value which is b...

Page 99: ...with the sys or class parameters For example ibv_devinfo d qedr0 v grep GID GID 0 fe80 0000 0000 0000 020e 1eff fec4 1b20 GID 1 fe80 0000 0000 0000 020e 1eff fec4 1b20 GID 2 0000 0000 0000 0000 0000 ffff 1e01 010a GID 3 0000 0000 0000 0000 0000 ffff 1e01 010a GID 4 3ffe ffff 0000 0f21 0000 0000 0000 0004 GID 5 3ffe ffff 0000 0f21 0000 0000 0000 0004 NOTE The default GID value is zero 0 for back to...

Page 100: ...sh DEV PORT INDEX GID IPv4 VER DEV qedr0 1 0 fe80 0000 0000 0000 020e 1eff fec4 1b20 v1 p4p1 qedr0 1 1 fe80 0000 0000 0000 020e 1eff fec4 1b20 v2 p4p1 qedr0 1 2 0000 0000 0000 0000 0000 ffff 1e01 010a 30 1 1 10 v1 p4p1 qedr0 1 3 0000 0000 0000 0000 0000 ffff 1e01 010a 30 1 1 10 v2 p4p1 qedr0 1 4 3ffe ffff 0000 0f21 0000 0000 0000 0004 v1 p4p1 qedr0 1 5 3ffe ffff 0000 0f21 0000 0000 0000 0004 v2 p4...

Page 101: ... RoCE v2 Run over the same subnet and use the RoCE v2 GID index Server ib_send_bw d qedr0 F x 1 Client ib_send_bw d qedr0 F x 1 192 168 100 3 Verifying RoCE v2 Through Different Subnets To verify RoCE v2 through different subnets 1 Set the route configuration for the server and client using the DCBX PFC configuration System Settings Server VLAN IP 192 168 100 3 and Gateway 192 168 100 1 Client VLA...

Page 102: ...add 192 168 100 0 24 via 192 168 101 1 dev p4p1 101 2 Set the switch settings using the following procedure Use any flow control method Pause DCBX CEE or DCBX IEEE and enable IP routing for RoCE v2 See Preparing the Ethernet Switch on page 59 for RoCE v2 configuration or refer to the vendor switch documents If you are using PFC configuration and L3 routing run RoCE v2 traffic over the VLAN using a...

Page 103: ...ngs for RDMA_CM Applications To configure RoCE use the following scripts from the FastLinQ source package show_rdma_cm_roce_ver sh qedr0 is configured to IB RoCE v1 qedr1 is configured to IB RoCE v1 config_rdma_cm_roce_ver sh v2 configured rdma_cm for qedr0 to RoCE v2 configured rdma_cm for qedr1 to RoCE v2 Server Settings Figure 6 4 Configuring RDMA_CM Applications Server ...

Page 104: ... QLogic NIC and RoCE drivers 2 Using the module parameter enable the RoCE function from the NIC driver by issuing the following command esxcfg module s enable_roce 1 qedentv To apply the change reload the NIC driver or reboot the system 3 To view a list of the NIC interfaces issue the esxcfg nics l command For example esxcfg nics l Name PCI Driver Link Speed Duplex MAC Address MTU Description Vmni...

Page 105: ...te the QLogic NIC port to the vSwitch issue the following command esxcli network vswitch standard uplink add u uplink device v roce vswitch For example esxcli network vswitch standard uplink add u vmnic0 v roce_vs 3 To create a new port group on this vSwitch issue the following command esxcli network vswitch standard portgroup add p roce_pg v roce_vs For example esxcli network vswitch standard por...

Page 106: ...d Statistics For the RoCE mode ESXi requires concurrent support of both RoCE v1 and v2 The decision regarding which RoCE mode to use is made during queue pair creation The ESXi driver advertises both modes during registration and initialization To view RoCE statistics issue the following command esxcli rdma device stats get d vmrdma0 Packets received 0 Packets sent 0 Bytes received 0 Bytes sent 0 ...

Page 107: ...MA To configure PVRDMA using vCenter interface 1 Create and configure a new distributed virtual switch as follows a In the VMware vSphere Web Client right click the RoCE node in the left pane of the Navigator window b On the Actions menu point to Distributed Switch and then click New Distributed Switch c Select version 6 5 0 d Under New Distributed Switch click Edit settings and then configure the...

Page 108: ...e port group created in Step 1 3 Assign a vmknic for PVRDMA to use on ESX hosts a Right click a host and then click Settings b On the Settings page expand the System node and then click Advanced System Settings c The Advanced System Settings page shows the key pair value and its summary Click Edit d On the Edit Advanced System Settings page filter on PVRDMA to narrow all the setting to just Net PV...

Page 109: ...e 6 8 Setting the Firewall Rule 5 Set up the VM for PVRDMA as follows a Install one of the following supported guest OSs RHEL 7 2 Ubuntu 14 04 kernel version 4 0 b Install OFED 3 18 c Compile and install the PVRDMA guest driver and library d Add a new PVRDMA network adapter to VM as follows Edit the VM settings Add a new network adapter Select the newly added DVS port group as Network Select PVRDM...

Page 110: ...ter iWARP configuration using HII For more information about preboot adapter configuration see Chapter 5 Adapter Preboot Configuration To configure iWARP through HII in Default mode 1 Access the server BIOS System Setup and then click Device Settings 2 On the Device Settings page select a port for the 25G 41000 Series Adapter 3 On the Main Configuration Page for the selected adapter click NIC Conf...

Page 111: ...g iWARP verifying RDMA and verifying iWARP traffic on Windows For a list of OSs that support iWARP see Table 6 1 on page 57 To enable iWARP on the Windows host and verify RDMA 1 Enable iWARP on the Windows host a Open the Windows Device Manager and then open the 41000 Series Adapters NDIS Miniport Properties b On the FastLinQ Adapter properties click the Advanced tab c On the Advanced page under P...

Page 112: ... Get NetOffloadGlobalSetting command output Figure 7 2 shows NetworkDirect as Enabled Figure 7 2 Windows PowerShell Command Get NetOffloadGlobalSetting To verify iWARP traffic 1 Map SMB drives and run iWARP traffic 2 Launch Performance Monitor Perfmon 3 In the Add Counters dialog box click RDMA Activity and then select the adapter instances ...

Page 113: ...7 iWARP Configuration Configuring iWARP on the Windows 89 AH0054601 00 B Figure 7 3 shows an example Figure 7 3 Perfmon Add Counters ...

Page 114: ...ft Windows Network The command completed successfully b Issue the net xan command as follows where Share1 is mapped as an SMB share C Users Administrator net xan Active NetworkDirect Connections Listeners ShareEndpoints Mode IfIndex Type Local Address Foreign Address PID Kernel 56 Connection 192 168 11 20 16159 192 168 11 10 445 0 Kernel 56 Connection 192 168 11 20 15903 192 168 11 10 445 0 Kernel...

Page 115: ...figuring NFS RDMA iWARP RDMA Core Support on SLES 12 SP3 RHEL 7 4 and OFED 4 8x Installing the Driver Install the RDMA drivers as shown in Chapter 3 Driver Installation Configuring iWARP and RoCE To enable iWARP and RoCE 1 Unload all FastlinQ drivers modprobe r qedr or modprobe r qede 2 Use the following command syntax to change the RDMA protocol by loading the qed driver with a port interface PCI...

Page 116: ...ansport iWARP 1 transport iWARP 1 transport iWARP 1 Detecting the Device To detect the device 1 To verify whether RDMA devices are detected view the dmesg logs dmesg grep qedr 10500 191047 qedr 0000 04 00 0 registered qedr0 10500 221726 qedr 0000 04 00 1 registered qedr1 2 Issue the ibv_devinfo command and then verify the transport type If the command is successful each PCI function will show a se...

Page 117: ...er rdma_client rdma_xserver rdma_xclient rping NFS over RDMA NFSoRDMA iSER for details see Chapter 8 iSER Configuration NVMe oF for details see Chapter 12 NVMe oF Configuration with RDMA Running Perftest for iWARP All perftest tools are supported over the iWARP transport type You must run the tools using RDMA connection manager with the R option Example 1 On one server issue the following command ...

Page 118: ...000 2250 38 2250 36 0 036006 Configuring NFS RDMA NFS RDMA for iWARP includes both server and client configuration steps To configure the NFS server 1 In the etc exports file for the directories that you must export using NFS RDMA on the server make the following entry tmp nfs server fsid 0 async insecure no_root_squash Ensure that you use a different file system identification FSID for each direc...

Page 119: ...p nfs client 3 Verify that the file system is mounted by issuing the mount command Ensure that the RDMA port and file system versions are correct mount grep rdma iWARP RDMA Core Support on SLES 12 SP3 RHEL 7 4 and OFED 4 8x The user space library libqedr is part of the rdma core However the out of box libqedr does not support SLES 12 SP3 RHEL 7 4 OFED 4 8x Therefore these OS versions require a pat...

Page 120: ...c_pingpong iwpmd rdma_client rdma_xclient rping ucmatose umad_compile_test cmtime ibv_asyncwatch ibv_rc_pingpong ibv_ud_pingpong mckey rdma ndd rdma_xserver rstream udaddy umad_reg2 ibacm ibv_devices ibv_srq_pingpong ibv_xsrq_pingpong rcopy rdma_server riostream srp_daemon udpong umad_register2 Run applications from the current RDMA core master location For example rping c v C 5 a 192 168 21 3 pin...

Page 121: ...g 0 ABCDEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqr ping data rdma ping 1 BCDEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrs ping data rdma ping 2 CDEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrst ping data rdma ping 3 DEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrstu ping data rdma ping 4 EFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrstuv client DISCONNECT EVENT ...

Page 122: ...ing operating systems RHEL 7 2 and 7 3 SLES12 SP1 and SP2 Ubuntu 14 04 LTS and 16 04 LTS After logging into the targets or while running I O traffic unloading the Linux RoCE qedr driver may crash the system While running I O performing interface down up tests or performing cable pull tests can cause driver or iSER module errors that may crash the system If this happens reboot the system Configurin...

Page 123: ...dprobe qedr modprobe ib_iser modprobe ib_isert 5 Verify that all RDMA and iSER modules loaded on the initiator and target devices by issuing the lsmod grep qed and lsmod grep iser commands 6 Verify that there are separate hca_id instances by issuing the ibv_devinfo command as shown in Step 6 on page 73 7 Check the RDMA connection on the initiator device and the target device a On the initiator dev...

Page 124: ...tances 9 Install Linux iSCSI Initiator Utilities using the yum install iscsi initiator utils commands a To discover the iSER target issue the iscsiadm command For example iscsiadm m discovery t st p 192 168 100 99 3260 b To change the transport mode to iSER issue the iscsiadm command For example iscsiadm m node T iqn 2015 06 test target1 o update n iface transport_name v iser c To connect to or lo...

Page 125: ...Iface Transport is iser in the target connection as shown Figure 8 3 Issue the iscsiadm command for example iscsiadm m session P2 Figure 8 3 Iface Transport Confirmed e To check for a new iSCSI device as shown Figure 8 4 issue the lsscsi command Figure 8 4 Checking for New iSCSI Device ...

Page 126: ...0 1 4 10 noarch rpm python netifaces 0 8 6 55 x86_64 rpm python rtslib 2 2 6 6 noarch rpm python urwid 1 1 1 6 144 x86_64 rpm targetcli 2 1 3 8 x86_64 rpm 2 Before starting the targetcli load all RoCE device drivers and iSER modules as follows modprobe qed modprobe qede modprobe qedr modprobe ib_iser Initiator modprobe ib_isert Target 3 Before configuring iSER targets configure NIC interfaces and ...

Page 127: ... fb41 Copyright 2011 2013 by Datera Inc and others For help on commands type help 2 Issue the following commands backstores ramdisk create Ramdisk1 1 1g nullio true iscsi create iqn 2017 04 com org iserport1 target1 iscsi iqn 2017 04 com org iserport1 target1 tpg1 luns create backstores ramdisk Ramdisk1 1 iscsi iqn 2017 04 com org iserport1 target1 tpg1 portals create 192 168 21 4 ip_port 3261 isc...

Page 128: ...gin to iface iser target iqn 2017 04 com org iserport1 target1 portal 192 168 21 4 3261 successful 4 Ensure that those LUNs are visible by issuing the following command lsscsi 1 0 0 0 storage HP P440ar 3 56 1 1 0 0 disk HP LOGICAL VOLUME 3 56 dev sda 6 0 0 0 cd dvd hp DVD ROM DUD0N UMD0 dev sr0 7 0 0 0 disk LIO ORG Ramdisk1 1 4 0 dev sdb Configuring iSER for Ubuntu To configure iSER you must first...

Page 129: ...ze 1G named iSERPort1 1 by issuing the following command backstores rd_mcp create name iSERPort1 1 size 1G Generating a wwn serial Created rd_mcp ramdisk iSERPort1 1 with size 1G ls o o backstores o fileio 0 Storage Object o iblock 0 Storage Object o pscsi 0 Storage Object o rd_dr 0 Storage Object o rd_mcp 1 Storage Object o iSERPort1 1 ramdisk deactivated o ib_srpt 0 Targets o iscsi 0 Targets o l...

Page 130: ...iqn 2004 01 com qlogic iSERPort1 Target 1 TPG o tpgt1 enabled o acls 0 ACLs o luns 0 LUNs o portals 0 Portals o loopback 0 Targets o qla2xxx 0 Targets o tcm_fc 0 Targets 5 Create a LUN by issuing the following command Be sure that the RAM disk is activated iscsi iqn 2004 01 com qlogic iSERPort1 Target1 tpgt1 luns create backstores rd_mcp iSERPort1 1 Selected LUN 0 Successfully created LUN 0 ls o o...

Page 131: ...iscsi iqn 2004 01 com qlogic iSERPort1 Target1 tpgt1 portals create 192 168 10 5 Using default IP port 3260 Successfully created network portal 192 168 10 5 3260 ls o o backstores o fileio 0 Storage Object o iblock 0 Storage Object o pscsi 0 Storage Object o rd_dr 0 Storage Object o rd_mcp 1 Storage Object o iSERPort1 1 ramdisk activated o ib_srpt 0 Targets o iscsi 1 Target o iqn 2004 01 com qlogi...

Page 132: ...m qlogic iSERPort1 Target1 1 TPG o tpgt1 enabled o acls 0 ACLs o luns 1 LUN o lun0 rd_mcp iSERPort1 1 ramdisk o portals 1 Portal o 192 168 10 103 3260 OK iser enabled o loopback 0 Targets o qla2xxx 0 Targets o tcm_fc 0 Targets 8 Configure the quantity of access control lists ACLs by issuing the following command iscsi iqn 2004 01 com qlogic iSERPort1 Target1 tpgt1 set attribute authentication 0 de...

Page 133: ... configuration to disk will overwrite your boot settings The current target configuration will become the default boot config Are you sure Type yes yes Making backup of fc ConfigFS with timestamp 2015 06 09_19 07 37 855693 Successfully updated default config etc target fc_start sh Making backup of loopback ConfigFS with timestamp 2015 06 09_19 07 37 855693 Successfully updated default config etc t...

Page 134: ...2 target_core_iblock tcm_qla2xxx target_core_pscsi iscsi_target_mod tcm_fc ib_srpt target_core_file target_core_user tcm_loop ib_isert ib_iser 52919 0 rdma_cm 48739 3 ib_iser rdma_ucm ib_isert ib_core 98710 15 qedr rdma_cm ib_cm ib_sa iw_cm mlx4_ib ib_mad ib_ucm ib_iser ib_srpt ib_umad i b_uverbs rdma_ucm ib_ipoib ib_isert libiscsi 57498 3 libiscsi_tcp iscsi_tcp ib_iser scsi_transport_iscsi 100628...

Page 135: ...ux Performance Consider the following Linux performance configuration enhancements described in this section Configuring CPUs to Maximum Performance Mode Configuring Kernel sysctl Settings Configuring IRQ Affinity Settings Configuring Block Device Staging Configuring CPUs to Maximum Performance Mode Configure the CPU scaling governor to performance by using the following script to set all CPUs to ...

Page 136: ...v4 tcp_adv_win_scale 1 echo 0 proc sys vm nr_hugepages Configuring IRQ Affinity Settings The following example sets CPU core 0 1 2 and 3 to IRQ XX YY ZZ and XYZ respectively Perform these steps for each IRQ assigned to a port default is eight queues per port systemctl disable irqbalance systemctl stop irqbalance cat proc interrupts grep qedr Shows IRQ assigned to each port queue echo 1 proc irq XX...

Page 137: ...LES 12 Configuring iSCSI Boot from SAN for RHEL 7 4 iSCSI Offload in Windows Server iSCSI Offload in Linux Environments Differences from bnx2i Configuring qedi ko Verifying iSCSI Interfaces in Linux Open iSCSI and Boot from SAN Considerations NOTE iSCSI hardware offload is supported only on the QL4146x adapters Some iSCSI features may not be fully enabled in the current release For details refer t...

Page 138: ...des Selecting the Preferred iSCSI Boot Mode Configuring the iSCSI Target Configuring iSCSI Boot Parameters Selecting the Preferred iSCSI Boot Mode Boot mode option is listed under of the adapter and the setting is port specific Refer to the OEM user manual for direction on accessing the device level configuration menu under UEFI HII Configuring the iSCSI Target Configuring the iSCSI target varies ...

Page 139: ...pecific to either IPv4 or IPv6 are noted NOTE The availability of the IPv6 iSCSI boot is platform and device dependent Table 9 1 Configuration Options Option Description TCP IP parameters via DHCP This option is specific to IPv4 Controls whether the iSCSI boot host software acquires the IP address information using DHCP Enabled or use a static IP configuration Disabled iSCSI parameters via DHCP Co...

Page 140: ... Login Timeout Allows you to specify a maximum wait time in seconds for the ini tiator to complete target login DHCP Vendor ID Controls how the iSCSI boot host software interprets the Vendor Class ID field used during DHCP If the Vendor Class ID field in the DHCP offer packet matches the value in the field the iSCSI boot host software looks into the DHCP Option 43 fields for the required iSCSI boo...

Page 141: ...ey to access the System Utilities menu at boot time Figure 9 1 Figure 9 1 Systems Utilities at Boot Time 3 In System HII select the QLogic device Figure 9 2 Refer to the OEM user guide on accessing the PCI device configuration menu For example on a HPE Gen 9 server the System Utilities for QLogic devices are listed on the System Configuration menu Figure 9 2 Configuration Utility NOTE SAN boot is ...

Page 142: ...9 iSCSI Configuration iSCSI Boot 118 AH0054601 00 B 4 On the Main Configuration Page select Port Level Configuration Figure 9 3 and then press ENTER Figure 9 3 Selecting Port Level Configuration ...

Page 143: ...120 Dynamic iSCSI Boot Configuration on page 126 Configuring iSCSI Boot iSCSI boot configuration options include Static iSCSI Boot Configuration Dynamic iSCSI Boot Configuration Enabling CHAP Authentication NOTE The iSCSI HW option is not listed if the iSCSI Offload feature is disabled at port level If the preferred boot mode is iSCSI HW make sure the iSCSI offload feature is enabled Not all adapt...

Page 144: ...ss System s initiator IQN Target parameters obtained in Configuring the iSCSI Target on page 114 For information on configuration options see Table 9 1 on page 115 To configure the iSCSI boot parameters using static configuration 1 In the Device HII Main Configuration Page select iSCSI Configuration Figure 9 5 and then press ENTER Figure 9 5 Selecting iSCSI Boot Configuration ...

Page 145: ...RROW keys to select a parameter and then press the ENTER key to select or input the following values TCP IP Parameters via DHCP Disabled iSCSI Parameters via DHCP Disabled CHAP Authentication As required IP Version As required IPv4 or IPv6 ARP Redirect Not applicable for boot DHCP Request Timeout Default value or as required Target Login Timeout Default value or as required Figure 9 7 iSCSI Genera...

Page 146: ...g parameters and then type a value for each IPv4 Address IPv4 Subnet Mask IPv4 Default Gateway IPv4 Primary DNS IPv4 Secondary DNS VLAN ID Optional iSCSI traffic on the network may be isolated in a Layer 2 VLAN to segregate it from general traffic If this is the case make the iS2CSI interface on the adapter a member of that VLAN by setting this value iSCSI Name Corresponds to the iSCSI initiator n...

Page 147: ... 10 and then press ENTER Figure 9 10 iSCSI First Target Parameters NOTE Note the following for the preceding items with asterisks The label will change to IPv6 or IPv4 default based on the IP version set on the iSCSI General Parameters page Figure 9 7 on page 121 Carefully enter the IP address There is no error checking performed against the IP address to check for duplicates incorrect segment or ...

Page 148: ...meters for the iSCSI target and then press ENTER IPv4 Address TCP Port Boot LUN iSCSI Name CHAP ID CHAP Secret Figure 9 11 iSCSI First Target Configuration 11 Return to the iSCSI Boot Configuration page and then press ESC NOTE For the preceding parameters with an asterisk the label will change to IPv6 or IPv4 default based on IP version set on the iSCSI General Parameters page as shown in Figure 9...

Page 149: ... you want configure a second iSCSI target device select iSCSI Second Target Parameters Figure 9 12 and enter the parameter values as you did in Step 10 Otherwise proceed to Step 13 Figure 9 12 iSCSI Second Target Configuration 13 Press ESC once and a second time to exit ...

Page 150: ...changes to the adapter s running configuration Dynamic iSCSI Boot Configuration In a dynamic configuration ensure that the system s IP address and target or initiator information are provided by a DHCP server see IPv4 and IPv6 configurations in Configuring the DHCP Server to Support iSCSI Boot on page 129 Any settings on the following parameters are ignored and do not need to be cleared with the e...

Page 151: ...set the following options as shown in Figure 9 14 TCP IP Parameters via DHCP Enabled iSCSI Parameters via DHCP Enabled CHAP Authentication As required IP Version As required IPv4 or IPv6 ARP Redirect Not applicable for boot DHCP Request Timeout Default value or as required NOTE When using a DHCP server the DNS server entries are overwritten by the values provided by the DHCP server This override o...

Page 152: ...rs window type values for the following CHAP ID up to 255 characters CHAP Secret if authentication is required must be 12 to 16 characters in length 4 Press ESC to return to the iSCSI Boot configuration page 5 On the iSCSI Boot configuration page select iSCSI First Target Parameters 6 In the iSCSI First Target Parameters window type values used when configuring the iSCSI target CHAP ID optional if...

Page 153: ... Path DHCP Option 43 Vendor specific Information DHCP Option 17 Root Path Option 17 is used to pass the iSCSI target information to the iSCSI client The format of the root path as defined in IETC RFC 4173 is iscsi servername protocol port LUN targetname Table 9 2 lists the DHCP Option 17 parameters Table 9 2 DHCP Option 17 Parameter Definitions Parameter Definition iscsi A literal string servernam...

Page 154: ...on 43 requires more configuration than DHCP Option 17 but it provides a richer environment and more configuration options QLogic recommends that customers use DHCP Option 43 when performing dynamic iSCSI boot configuration targetname Target name in either IQN or EUI format For details on both IQN and EUI formats refer to RFC 3720 An example IQN name is iqn 1995 05 com QLogic iscsi target NOTE DHCP...

Page 155: ...alue is QLGC ISAN as shown in the General Parameters of the iSCSI Boot Configuration menu The content of Option 16 should be 2 byte length DHCP Vendor ID DHCPv6 Option 17 Vendor Specific Information DHCPv6 Option 17 vendor specific information provides more configuration options to the iSCSI client In this configuration three additional sub options are provided that assign the initiator IQN to the...

Page 156: ... case make the iSCSI interface on the adapter a member of that VLAN To configure VLAN for iSCSI boot 1 Go to the iSCSI Boot Configuration Menu for the port 2 Select iSCSI Initiator Configuration Table 9 4 DHCP Option 17 Sub option Definitions Sub option Definition 201 First iSCSI target information in the standard root path format iscsi servername protocol port LUN targetname 202 Second iSCSI targ...

Page 157: ...41000 Series Adapters for SLES 12 SP1 on UEFI based systems To perform iSCSI boot from SAN 1 Configure adapter ports for L2 iSCSI Boot Firmware Table iBFT configuration as follows a Open System Configuration select the adapter port and then select Port Level Configuration b On the Port Level Configuration page set the Boot Mode to iSCSI SW and set iSCSI Offload to Disabled c In the Device HII Main...

Page 158: ... On the iSCSI First Target Configuration page configure the following parameters Connect Set to Enabled IPv4 Address TCP Port Boot LUN iSCSI Name j Save the settings and reboot the server 2 Mount the OS ISO and proceed to install SL12 SP1 OS on iSCSI disk over L2 At the beginning of the installation pass the following boot parameter to inject latest drivers dud 1 3 To complete L2 BFS follow the on...

Page 159: ... dev sda multipath a dev sdb g To run the multipath utility issue the following commands multipath may not show the multipath devices because it is booted with single path on L2 multipath ll h To Inject the multipath module in initrd issue the following command dracut force add multipath include etc multipath i Reboot the server and enter system settings by pressing the F9 key during the POST menu...

Page 160: ...media install Red Hat Enterprise 7 x Troubleshooting Use the UP and DOWN keys to change the selection Press e to edit the selected item or c for a command prompt 2 To install an out of box driver type e Otherwise proceed to Step 7 3 Select the kernel line and then type e 4 Issue the following command and then press ENTER linux dd modprobe blacklist qed modprobe blacklist qede modprobe blacklist qe...

Page 161: ...shown in the Figure 9 17 example Figure 9 17 Prompt for Out of Box Installation 6 If required for your setup load the FastLinQ driver update disk when prompted for additional driver disks Otherwise type c if you have no other driver update disks to install 7 Continue with the installation You can skip the media test Click Next to continue with the installation ...

Page 162: ...da indicating a single path installation If you configured multipath the disk has a device mapper label 10 In the Specialized Network Disks section select the iSCSI LUN 11 Type the root user s password and then click Next to complete the installation 12 During the first boot add the following kernel command line rd iscsi firmware rd driver pre qed qede qedr qedf qedi selinux 0 13 After a successfu...

Page 163: ...tLinQ 41000 Series Adapter to offload iSCSI processing from the host processor The following sections describe how to enable the system to take advantage of QLogic s iSCSI offload feature Installing QLogic Drivers Installing the Microsoft iSCSI Initiator Configuring Microsoft Initiator to Use QLogic s iSCSI Offload iSCSI Offload FAQs Windows Server 2012 R2 and 2016 iSCSI Boot Installation iSCSI Cr...

Page 164: ...he iSCSI target using the QLogic iSCSI adapter For more details on Microsoft Initiator see the Microsoft user guide To configure the Microsoft Initiator 1 Open Microsoft Initiator 2 To configure the initiator IQN name according to your setup follow these steps a On the iSCSI Initiator Properties click the Configuration tab b On the Configuration page Figure 9 19 click Change to modify the initiato...

Page 165: ...Windows Server 141 AH0054601 00 B c In the iSCSI Initiator Name dialog box type the new initiator IQN name and then click OK Figure 9 20 Figure 9 20 iSCSI Initiator Node Name Change 3 On the iSCSI Initiator Properties click the Discovery tab ...

Page 166: ...9 iSCSI Configuration iSCSI Offload in Windows Server 142 AH0054601 00 B 4 On the Discovery page Figure 9 21 under Target portals click Discover Portal Figure 9 21 iSCSI Initiator Discover Target Portal ...

Page 167: ...2 a In the IP address or DNS name box type the IP address of the target b Click Advanced Figure 9 22 Target Portal IP Address 6 In the Advanced Settings dialog box Figure 9 23 complete the following under Connect using a For Local adapter select the QLogic name or model Adapter b For Initiator IP select the adapter IP address ...

Page 168: ...9 iSCSI Configuration iSCSI Offload in Windows Server 144 AH0054601 00 B c Click OK Figure 9 23 Selecting the Initiator IP Address 7 On the iSCSI Initiator Properties Discovery page click OK ...

Page 169: ...9 iSCSI Configuration iSCSI Offload in Windows Server 145 AH0054601 00 B 8 Click the Targets tab and then on the Targets page Figure 9 24 click Connect Figure 9 24 Connecting to the iSCSI Target ...

Page 170: ...ns page in QConvergeConsole GUI Question What tools should be used to create the connection to the target Answer Use Microsoft iSCSI Software Initiator version 2 08 or later Question How do I know that the connection is offloaded Answer Use Microsoft iSCSI Software Initiator From a command line type oiscsicli sessionlist From Initiator Name an iSCSI offloaded connection will display an entry begin...

Page 171: ...nection from the remote device Ensure that the target has sufficient disk space to hold the new OS installation 5 Configure the UEFI HII to set the iSCSI boot type offload or non offload correct initiator and target parameters for iSCSI boot 6 Save the settings and reboot the system The remote system should connect to the iSCSI target and then boot from the DVD ROM device 7 Boot from DVD and begin...

Page 172: ...information uio ko is the Linux Userspace I O interface used for light L2 memory mapping for iscsiuio These modules must be loaded before qedi can be functional Otherwise you might encounter an unresolved symbol error If the qedi module is installed in the distribution update path the requisite is automatically loaded by modprobe Differences from bnx2i Some key differences exist between qedi the d...

Page 173: ...es were detected correctly To verify iSCSI interfaces in Linux 1 To verify that the qedi and associated kernel modules are actively loaded issue the following command lsmod grep qedi qedi 114578 2 qed 697989 1 qedi uio 19259 4 cnic qedi libiscsi 57233 2 qedi bnx2i scsi_transport_iscsi 99909 5 qedi bnx2i libiscsi iscsi_boot_sysfs 16000 1 qedi 2 To verify that the iSCSI interfaces were detected prop...

Page 174: ...Loaded loaded usr lib systemd system iscsiuio service disabled vendor preset disabled Active active running since Fri 2017 01 27 16 33 58 IST 6 days ago Docs man iscsiuio 8 Process 3745 ExecStart usr sbin iscsiuio code exited status 0 SUCCESS Main PID 3747 iscsiuio CGroup system slice iscsiuio service 3747 usr sbin iscsiuio Jan 27 16 33 58 localhost localdomain systemd 1 Starting iSCSI UserSpace I...

Page 175: ...ze virtualun virtualun target 05000007 portal 192 168 25 100 3260 successful 7 To verify that the iSCSI session was created issue the following command iscsiadm m session qedi 297 192 168 25 100 3260 1 iqn 2003 04 com sanblaze virtualun virtualun target 05000007 non flash 8 To check for iSCSI devices issue the iscsiadm command iscsiadm m session P3 Attached SCSI devices Host Number 59 State runnin...

Page 176: ... the boot from SAN To boot from SAN using a software initiator 1 Complete the following in the adapter s preboot device configuration iSCSI Boot Configuration Menu a On all ports set iSCSI Offload to Disable b On all ports set the HBA Mode which uses the iSCSI offload boot pathway to Disabled c Set the Boot Mode to iSCSI 2 Configure the initiator and target entries 3 At the beginning of the instal...

Page 177: ...siuio systemctl restart iscsiuio systemctl restart iscsid 2 For the RHEL 7 x OS distro only complete the following steps a Create an iface record for an L4 interface iscsiadm m iface I qedi 00 0e 1e c4 ca 5d o new b Update the iface fields in the iface records For example iscsiadm m iface I qedi 00 0e 1e c4 ca 5d n iface hwaddress v 00 0e 1e c4 ca 5d o update iscsiadm m iface I qedi 00 0e 1e c4 ca...

Page 178: ...md system iscsi service Locate and comment out the following line ExecStart usr libexec iscsi mark root nodes 3 Issue the following command cat proc cmdline 4 Check if the OS has preserved any boot options such as ip ibft or rd iscsi ibft If there are preserved boot options continue with Step 5 If there are no preserved boot options skip to Step 5 c 5 Edit the etc default grub file and modify the ...

Page 179: ... initramfs image by issuing the following command dracut force 11 For the RHEL 7 x OS distro only complete the following steps a Restore the following iscsid configuration file vi usr lib systemd system iscsid service Locate the following lines ExecStart usr sbin iscsid ExecStop sbin iscsiadm k 0 2 Edit uncomment out the preceding lines as shown in the following ExecStart usr sbin iscsid ExecStop ...

Page 180: ... BindTo iscsid service c Reboot the server 12 On the adapter s preboot iSCSI Boot Configuration Menu change the value of the iSCSI offload a On the iSCSI Boot Configuration Menu set iSCSI Offload to Enable b Set HBA Mode to Enable NOTE The OS can now boot through the offload interface ...

Page 181: ... Boot from SAN Considerations FCoE Boot from SAN This section describes the installation and boot procedures for the Windows Linux and ESXi operating systems including Preparing System BIOS for FCoE Build and Boot Windows FCoE Boot from SAN NOTE FCoE offload is supported on all 41000 Series Adapters Some FCoE features may not be fully enabled in the current release For details refer to Appendix C ...

Page 182: ...ng the system BIOS configuration to UEFI Configuring Adapter UEFI Boot Mode To configure the boot mode to FCOE 1 Restart the system 2 Press the OEM hot key to enter System Setup or the configuration menu Figure 10 1 This is also known as UEFI HII For example the HPE Gen 9 system have F9 as a hotkey to access the System Utilities menu at boot time Figure 10 1 System Utilities NOTE FCoE BFS is not s...

Page 183: ... accessing PCI device configuration menu For example on an HPE Gen 9 server the System Utilities for QLogic devices are listed under the System Configuration menu Figure 10 2 System Configuration Port Selection 4 On the Main Configuration Page select Port Level Configuration Figure 10 3 and then press ENTER Figure 10 3 Port Level Configuration ...

Page 184: ...press ENTER to select FCoE as a preferred boot mode Figure 10 4 Boot Mode in Port Level Configuration NOTE FCoE is not listed as a boot option if the FCoE Offload feature is disabled at the port level If the Boot Mode preferred is FCoE make sure the FCoE Offload feature is enabled as shown in Figure 10 5 Not all adapter versions support FCoE ...

Page 185: ...CoE Boot Configuration Menu select FCoE General Parameters Figure 10 6 and then press ENTER Figure 10 6 Selecting General Parameters 3 In the FCoE General Parameters menu Figure 10 7 press the UP ARROW and DOWN ARROW keys to select a parameter and then press ENTER to select and input the following values FIP VLAN ID As required if not set adapter will attempt FIP VLAN discovery Fabric Login Retry ...

Page 186: ...CoE General Parameters 4 Return to the FCoE Boot Configuration page 5 Press ESC and then select FCoE Target Parameters 6 Press ENTER 7 In the FCoE Target Parameters Menu enable Connect to the preferred FCoE target 8 Type values for the following parameters Figure 10 8 for the iSCSI target and then press ENTER WWPN n Boot LUN n ...

Page 187: ...rom SAN information for Windows includes Windows Server 2012 R2 and 2016 FCoE Boot Installation Configuring FCoE FCoE Crash Dump Windows Server 2012 R2 and 2016 FCoE Boot Installation For Windows Server 2012R2 2016 boot from SAN installation QLogic requires the use of a slipstream DVD or ISO image with the latest QLogic drivers injected See Injecting Slipstreaming Adapter Drivers into Windows Imag...

Page 188: ... ROM device 7 Boot from DVD and begin installation 8 Follow the on screen instructions 9 On the window that shows the list of disks available for the installation the FCoE target disk should be visible This target is a disk connected through the FCoE boot protocol located in the remote FCoE target 10 To proceed with Windows Server 2012R2 2016 installation select Next and follow the on screen instr...

Page 189: ...er to which to extract the driver package For example type c temp d Follow the driver installer instructions to install the drivers in the specified folder In this example the driver files are installed here c temp Program File 64 QLogic Corporation QDrivers 3 Download the Windows Assessment and Deployment Kit ADK version 10 from Microsoft https developer microsoft com en us windows hardware windo...

Page 190: ...ocol FIP fabric login FLOGI libfc ko is the Linux FC kernel library needed for several functions including Name server login and registration rport session management scsi_transport_fc ko is the Linux FC SCSI transport library used for remote port and SCSI target management These modules must be loaded before qedf can be functional otherwise errors such as unresolved symbol can result If the qedf ...

Page 191: ...ce created by fipvlan for fcoemon qedf does not sit on top of the net_device qedf is not dependent on network drivers such as bnx2x and cnic qedf will automatically initiate FCoE discovery on link up because it is not dependent on fipvlan or fcoemon for FCoE interface creation Configuring qedf ko No explicit configuration is required for qedf ko The driver automatically binds to the exposed FCoE f...

Page 192: ... that the FCoE devices were detected properly In this example the two detected FCoE CNA devices are SCSI host numbers 4 and 5 dmesg grep qedf 235 321185 0000 00 00 0 qedf_init 3728 QLogic FCoE Offload Driver v8 18 8 0 235 322253 0000 21 00 2 __qedf_probe 3142 4 QLogic FastLinQ FCoE Module qedf 8 18 8 0 FW 8 18 10 0 235 606443 scsi host4 qedf 235 624337 0000 21 00 3 __qedf_probe 3142 5 QLogic FastL...

Page 193: ... is located in sys class fc_host hostX where X is the number of the SCSI host In the preceding example X could 4 or 5 The hostX file contains attributes for the FCoE function such as worldwide port name and fabric ID Boot from SAN Considerations FCoE boot from SAN should work like FC boot from SAN where the module simply needs to be injected by the driver update disk DUD into the installation envi...

Page 194: ... on Windows Configuring SR IOV on Linux Configuring SR IOV on VMware Configuring SR IOV on Windows To configure SR IOV on Windows 1 Access the server BIOS System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices 3 On the Integrated Devices page a Set the SR IOV Global Enable option to Enabled b Click Back 4 On the Main Configuration Page for the ...

Page 195: ...ration 6 On the Main Configuration Page click Finish 7 In the Warning Saving Changes message box click Yes to save the configuration 8 In the Success Saving Changes message box click OK 9 To enable SR IOV on the miniport adapter a Access Device Manager b Open the miniport adapter properties and then click the Advanced tab c On the Advanced properties page Figure 11 2 under Property select SR IOV a...

Page 196: ...ure 11 2 Adapter Properties Advanced Enabling SR IOV 10 To create a Virtual Machine Switch with SR IOV Figure 11 3 on page 173 a Launch the Hyper V Manager b Select Virtual Switch Manager c In the Name box type a name for the virtual switch d Under Connection type select External network ...

Page 197: ...54601 00 B e Select the Enable single root I O virtualization SR IOV check box and then click Apply Figure 11 3 Virtual Switch Manager Enabling SR IOV NOTE Be sure to enable SR IOV when you create the vSwitch This option is unavailable after the vSwitch is created ...

Page 198: ...tch capability issue the following Windows PowerShell command PS C Users Administrator Get VMSwitch Name SR IOV_vSwitch fl Output of the Get VMSwitch command includes the following SR IOV capabilities IovVirtualFunctionCount 96 IovVirtualFunctionsInUse 1 12 To create a virtual machine VM and export the virtual function VF in the VM a Create a virtual machine b Add the VMNetworkadapter to the virtu...

Page 199: ...ox Figure 11 4 Hardware Acceleration page under Single root I O virtualization select the Enable SR IOV check box and then click OK Figure 11 4 Settings for VM Enabling SR IOV NOTE After the virtual adapter connection is created the SR IOV setting can be enabled or disabled at any time even while traffic is running ...

Page 200: ...Figure 11 5 shows an example Figure 11 5 Device Manager VM with QLogic Adapter 14 To view the SR IOV VF details issue the following Windows PowerShell command PS C Users Administrator Get NetadapterSriovVf NOTE Be sure to use the same driver package on both the VM and the host system For example use the same qeVBD and qeND driver version on the Windows VM and in the Windows Hyper V host ...

Page 201: ...k Integrated Devices 3 On the System Integrated Devices page a Set the SR IOV Global Enable option to Enabled b Click Back 4 On the System BIOS Settings page click Processor Settings 5 On the Processor Settings page a Set the Virtualization Technology option to Enabled b Click Back 6 On the System Setup page select Device Settings 7 On the Device Settings page select Port 1 for the QLogic adapter ...

Page 202: ...igure the iommu parameter as shown in Figure 11 8 For Intel based systems add intel_iommu on For AMD based systems add amd_iommu on Figure 11 8 Editing the grub conf File for SR IOV b Save the grub conf file and then reboot the system c To verify that the changes are in effect issue the following command dmesg grep I iommu A successful input output memory management unit IOMMU command output shoul...

Page 203: ...e 0 function 0 root ah rh68 echo 8 sys devices pci0000 00 0000 00 02 0 0000 04 00 0 sriov_numvfs b Review the command output Figure 11 9 to confirm that actual VFs were created on bus 4 device 2 from the 0000 00 02 0 parameter functions 0 through 7 Note that the actual device ID is different on the PFs 8070 in this example versus the VFs 9090 in this example Figure 11 9 Command Output for sriov_nu...

Page 204: ...ample output Figure 11 10 Command Output for ip link show Command 13 Assign and verify MAC addresses a To assign a MAC address to the VF issue the following command ip link set pf device vf vf index mac mac address b Ensure that the VF interface is up and running with the assigned MAC address ...

Page 205: ...ach the VF Some OSs support hot plugging of VFs to the VM a In the Virtual Machine dialog box Figure 11 11 click Add Hardware Figure 11 11 RHEL68 Virtual Machine b In the left pane of the Add New Virtual Hardware dialog box Figure 11 12 click PCI Host Device c In the right pane select a host device ...

Page 206: ...re SR IOV on VMware 1 Access the server BIOS System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices 3 On the Integrated Devices page a Set the SR IOV Global Enable option to Enabled b Click Back 4 On the System Setup window click Device Settings 5 On the Device Settings page select a port for the 25G 41000 Series Adapter 6 On the Device Level C...

Page 207: ...c7 0000 05 02 0 Network controller QLogic Corp QLogic FastLinQ QL41xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_0 0000 05 02 1 Network controller QLogic Corp QLogic FastLinQ QL41xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_1 0000 05 02 2 Network controller QLogic Corp QLogic FastLinQ QL41xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_2 0000 05 02 3 Network controller QLogi...

Page 208: ...riovnic vf list n vmnic6 VF ID Active PCI Address Owner World ID 0 true 005 02 0 60591 1 true 005 02 1 60591 2 false 005 02 2 3 false 005 02 3 4 false 005 02 4 5 false 005 02 5 6 false 005 02 6 7 false 005 02 7 8 false 005 03 0 9 false 005 03 1 10 false 005 03 2 11 false 005 03 3 12 false 005 03 4 13 false 005 03 5 14 false 005 03 6 15 false 005 03 7 14 Attach VFs to the VM as follows a Power off ...

Page 209: ...0 B c For Physical Function select the QLogic VF d To save your configuration changes and close this dialog box click OK Figure 11 13 VMware Host Edit Settings 16 Power on the VM and then issue the ifconfig a command to verify that the added network interface is listed ...

Page 210: ...11 SR IOV Configuration Configuring SR IOV on VMware 186 AH0054601 00 B 17 If no inbox driver is available install the driver 18 As needed add more VFs in the VM ...

Page 211: ...the distance within a data center over which NVMe devices and NVMe subsystems can be accessed The NVMe oF configuration procedures and options described in this chapter apply to Ethernet based RDMA protocols including RoCE and iWARP The development of NVMe oF with RDMA is defined by a technical sub group of the NVMe organization This chapter demonstrates how to configure NVMe oF for a simple netwo...

Page 212: ...ning the Target Server 5 Testing the NVMe oF Devices 6 Optimizing Performance Installing Device Drivers on Both Servers After installing your operating system RHEL 7 4 or SLES 12 SP3 install device drivers on both servers 1 Install and load the FastLinQ drivers QED QEDE libqedr QEDR following all installation instructions in the README 2 Enable and start the RDMA service systemctl enable rdma syst...

Page 213: ...e nvmet modprobe nvmet rdma lsmod grep nvme confirm that the modules are loaded 2 Create the target subsystem NQN with the name indicated by nvme subsystem name mkdir sys kernel config nvmet subsystems nvme subsystem name cd sys kernel config nvmet subsystems nvme subsystem name 3 Create multiple unique NQNs for additional NVMe devices as needed 4 Set the target parameters as listed in Table 12 1 ...

Page 214: ...ausing the script using the wait command or something similar as needed to ensure that each command finishes before executing the next command To configure the initiator server 1 Load the NVMe modules Issue these commands after each server reboot modprobe qedr modprobe nvme rdma mkdir sys kernel config nvmet ports 1 cd sys kernel config nvmet ports 1 Create NVMe port 1 echo 1 1 1 1 addr_traddr Set...

Page 215: ... the installation version as follows nvme version 4 Discover the NVMe oF target as follows nvme discover t rdma a 1 1 1 1 s 1023 Make note of the subsystem NQN subnqn of the discovered target Figure 12 2 for use in Step 5 Figure 12 2 Subsystem NQN 5 Connect to the discovered NVMe oF target nvme qlogic tgt1 using NQN Issue the following command after each server reboot For example nvme connect t rd...

Page 216: ...refilled or preconditioned To precondition the target server 1 Secure erase the target server with vendor specific tools similar to formatting This test example uses an Intel NVMe SSD device which requires the Intel Data Center Tool that is available at the following link https downloadcenter intel com download 23931 Intel Solid State Drive Data Center Tool 2 Precondition the target server nvme0n1...

Page 217: ...4k iodepth 1 numjobs 1 runtime 60 group_reporting name temp out FIO reports two latency types submission and completion Submission latency slat measures application to kernel latency Completion latency clat measures end to end kernel latency The industry accepted method is to read clat percentiles in the 99 00th range In this example the initiator device NVMe oF latency is 30usec 3 Run FIO to meas...

Page 218: ...er Issue the following command fio verify crc32 do_verify 1 bs 8k numjobs 1 iodepth 32 loops 1 ioengine libaio direct 1 invalidate 1 fsync_on_close 1 randrepeat 1 norandommap time_based runtime 60 filename dev nvme0n1 name Write BW to NVMe Device rw randwrite where rw can be randread for reads only randwrite for writes only and randrw for reads and writes ...

Page 219: ...ave the grub file c Rebuild the grub file To rebuild the grub file for a legacy BIOS boot issue the following command grub2 mkconfig o boot grub2 grub cfg Legacy BIOS boot To rebuild the grub file for an EFI boot issue the following command grub2 mkconfig o boot efi EFI os grub cfg EFI boot d Reboot the server to implement the changes 3 Set the IRQ affinity for all 41000 Series Adapters The multi_...

Page 220: ...1 cut d f2 MAXCPUID echo 2 LASTCPU p dc OFFSET echo 2 OFFSET p dc FACTOR echo 2 FACTOR p dc CPUID 1 for eth in do NUM grep eth proc interrupts wc l NUM_FP NUM INT grep m 1 eth proc interrupts cut d f 1 echo eth NUM NUM_FP fast path starting irq INT CPUID CPUID OFFSET for A 1 A NUM_FP A A 1 do INT grep m A eth proc interrupts tail 1 cut d f 1 SMP echo CPUID 16 o p dc echo INT smp affinity set to SM...

Page 221: ...w net ipv4 tcp_mem 16777216 16777216 16777216 sysctl w net ipv4 tcp_wmem 4096 65536 16777216 sysctl w net ipv4 tcp_rmem 4096 87380 16777216 sysctl w net core wmem_max 16777216 sysctl w net core rmem_max 16777216 sysctl w net core wmem_default 16777216 sysctl w net core rmem_default 16777216 sysctl w net core optmem_max 16777216 sysctl w net ipv4 tcp_low_latency 1 sysctl w net ipv4 tcp_timestamps 0...

Page 222: ...Hyper V with Network Direct Kernel Provider Interface NDKPI Mode 2 host virtual network adapters host virtual NICs support RDMA RoCE configuration procedures in this section include Creating a Hyper V Virtual Switch with an RDMA Virtual NIC Adding a VLAN ID to Host Virtual NIC Verifying If RoCE is Enabled Adding Host Virtual NICs Virtual Ports Mapping the SMB Drive and Running RoCE Traffic NOTE DC...

Page 223: ...ager 2 Click Virtual Switch Manager see Figure 13 1 Figure 13 1 Enabling RDMA in Host Virtual NIC 3 Create a virtual switch 4 Select the Allow management operating system to share this network adapter check box In Windows Server 2016 a new parameter Network Direct RDMA is added in the Host virtual NIC To enable RDMA in a host virtual NIC 1 Open the Hyper V Virtual Ethernet Adapter Properties windo...

Page 224: ... PS C Users Administrator Enable NetAdapterRdma vEthernet New Virtual Switch PS C Users Administrator Adding a VLAN ID to Host Virtual NIC To add VLAN ID to a host virtual NIC 1 To find the host virtual NIC name issue the following Windows PowerShell command PS C Users Administrator Get VMNetworkAdapter ManagementOS Figure 13 3 shows the command output Figure 13 3 Windows PowerShell Command Get VM...

Page 225: ...l Command Get NetAdapterRdma Adding Host Virtual NICs Virtual Ports To add host virtual NICs 1 To add a host virtual NIC issue the following command Add VMNetworkAdapter SwitchName New Virtual Switch Name SMB ManagementOS 2 Enable RDMA on host virtual NICs as shown in To enable RDMA in a host virtual NIC on page 199 NOTE Note the following about adding a VLAN ID to a host virtual NIC A VLAN ID mus...

Page 226: ...apterName SMB VlanId 5 Access ManagementOS Mapping the SMB Drive and Running RoCE Traffic To map the SMB drive and run the RoCE traffic 1 Launch the Performance Monitor Perfmon 2 Complete the Add Counters dialog box Figure 13 5 as follows a Under Available counters select RDMA Activity b Under Instances of selected object select the adapter c Click Add Figure 13 5 Add Counters Dialog Box ...

Page 227: ...2016 Technical Preview SET integrates limited NIC Teaming functionality into the Hyper V Virtual Switch Use SET to group between one and eight physical Ethernet network adapters into one or more software based virtual network adapters These adapters provide fast performance and fault tolerance if a network adapter failure occurs To be placed on a team SET member network adapters must all be instal...

Page 228: ...ommand New VMSwitch Enabling RDMA on SET To enable RDMA on SET 1 To view the SET on the adapter issue the following Windows PowerShell command PS C Users Administrator Get NetAdapter vEthernet SET Figure 13 8 shows command output Figure 13 8 Windows PowerShell Command Get NetAdapter 2 To enable RDMA on SET issue the following Windows PowerShell command PS C Users Administrator Enable NetAdapterRdm...

Page 229: ...trol PFC enhanced transition services ETS and traffic classes configuration must be the same on the switch and server To configure QoS by disabling DCBX 1 Disable DCBX on the adapter 2 Using HII set the RoCE Priority to 0 3 To install the DCB role in the host issue the following Windows PowerShell command PS C Users Administrators Install WindowsFeature Data Center Bridging 4 To set the DCBX Willi...

Page 230: ...s Advanced Properties page Figure 13 9 under Property select Quality of Service and then set the value to Enabled c Click OK Figure 13 9 Advanced Properties Enable QoS 6 Assign the VLAN ID to the interface as follows a Open the miniport window and then click the Advanced tab b On the adapter s Advanced Properties page Figure 13 10 under Property select VLAN ID and then set the value ...

Page 231: ...g VLAN ID 7 To enable priority flow control for RoCE on a specific priority issue the following command PS C Users Administrators Enable NetQoSFlowControl Priority 4 NOTE The preceding step is required for priority flow control PFC NOTE If configuring RoCE over Hyper V do not assign a VLAN ID to the physical interface ...

Page 232: ...t priority to each type of traffic issue the following commands where Priority 4 is tagged for RoCE and Priority 0 is tagged for TCP PS C Users Administrators New NetQosPolicy SMB NetDirectPortMatchCondition 445 PriorityValue8021Action 4 PolicyStore ActiveStore PS C Users Administrators New NetQosPolicy TCP IPProtocolMatchCondition TCP PriorityValue8021Action 0 Policystore ActiveStore PS C Users A...

Page 233: ... class ETS 50 4 Global TCP class ETS 30 0 Global 11 To see the network adapter QoS from the preceding configuration issue the following Windows PowerShell command PS C Users Administrator Get NetAdapterQos Name SLOT 4 Port 1 Enabled True Capabilities Hardware Current MacSecBypass NotSupported NotSupported DcbxSupport None None NumTCs Max ETS PFC 4 4 4 4 4 4 OperationalTrafficClasses TC TSA Bandwid...

Page 234: ...tch and server To configure QoS by enabling DCBX 1 Enable DCBX IEEE CEE or Dynamic 2 Using HII set the RoCE Priority to 0 3 To install the DCB role in the host issue the following Windows PowerShell command PS C Users Administrators Install WindowsFeature Data Center Bridging 4 To set the DCBX Willing mode to True issue the following command PS C Users Administrators set NetQosDcbxSetting Willing ...

Page 235: ...under Property select Quality of Service and then set the value to Enabled b Click OK Figure 13 11 Advanced Properties Enabling QoS 6 Assign the VLAN ID to the interface required for PFC as follows a Open the miniport window and then click the Advanced tab b On the adapter s Advanced Properties page Figure 13 12 under Property select VLAN ID and then set the value ...

Page 236: ...erShell command PS C Users Administrators Get NetAdapterQoS Name Ethernet 5 Enabled True Capabilities Hardware Current MacSecBypass NotSupported NotSupported DcbxSupport CEE CEE NumTCs Max ETS PFC 4 4 4 4 4 4 OperationalTrafficClasses TC TSA Bandwidth Priorities 0 ETS 5 0 3 5 7 1 ETS 95 4 OperationalFlowControl Priority 4 Enabled OperationalClassifications Protocol Port Type Priority ...

Page 237: ...ne Switch Getting the Virtual Machine Switch Capability Creating a VM and Enabling VMMQ on VMNetworkadapters in the VM Default and Maximum VMMQ Virtual NIC Enabling and Disabling VMMQ on a Management NIC Monitoring Traffic Statistics NOTE The preceding example is taken when the adapter port is connected to an Arista 7060X switch In this example the switch PFC is enabled on Priority 4 RoCE App TLVs...

Page 238: ...ect Virtual Switch RSS and then set the value to Enabled 3 Click OK Figure 13 13 Advanced Properties Enabling Virtual Switch RSS Setting the VMMQ Max QPs Default and Non Default VPort To set the VMMQ maximum QPs default and non default VPort 1 Open the miniport window and click the Advanced tab 2 On the Advanced Properties page Figure 13 14 under Property select one of the following VMMQ Max QPs D...

Page 239: ...property Figure 13 14 Advanced Properties Setting VMMQ 4 Click OK Creating a Virtual Machine Switch with or Without SR IOV To create a virtual machine switch with or without SR IOV 1 Launch the Hyper V Manager 2 Select Virtual Switch Manager see Figure 13 15 3 In the Name box type a name for the virtual switch ...

Page 240: ...16 Configuring VMMQ 216 AH0054601 00 B 4 Under Connection type a Click External network b Select the Allow management operating system to share this network adapter check box Figure 13 15 Virtual Switch Manager 5 Click OK ...

Page 241: ... command PS C Users Administrators Set VMSwitch name q1 defaultqueuevmmqenabled true defaultqueuevmmqqueuepairs 4 Getting the Virtual Machine Switch Capability To get the virtual machine switch capability Issue the following Windows PowerShell command PS C Users Administrator Get VMSwitch Name ql fl Figure 13 16 shows example output Figure 13 16 Windows PowerShell Command Get VMSwitch ...

Page 242: ... 3 1 00 0E 1E C4 C0 A4 0 8 PF Activated Adaptive 4 Ethernet 3 2 0 0 0 Activated Unknown 1 Ethernet 3 3 0 0 1 Activated Unknown 1 Ethernet 3 4 0 0 2 Activated Unknown 1 Ethernet 3 5 0 0 3 Activated Unknown 1 Ethernet 3 6 0 0 4 Activated Unknown 1 Ethernet 3 7 0 0 5 Activated Unknown 1 Ethernet 3 8 0 0 6 Activated Unknown 1 Ethernet 3 9 0 0 7 Activated Unknown 1 Ethernet 3 10 0 0 8 Activated Unknown...

Page 243: ...e Queues Ethernet 4 QLogic FastLinQ QL45212 DE 238 False 0 0 16 1 Default and Maximum VMMQ Virtual NIC According to the current implementation a maximum quantity of 4 VMMQs is available per virtual NIC that is up to 16 virtual NICs Four default queues are available as previously set using Windows PowerShell commands The maximum default queue can currently be set to 8 To verify the maximum default ...

Page 244: ...Deploying a Software Defined Network Enabling VXLAN Offload on the Adapter To enable VXLAN offload on the adapter 1 Open the miniport window and then click the Advanced tab 2 On the Advanced Properties page Figure 13 17 under Property select VXLAN Encapsulated Task Offload Figure 13 17 Advanced Properties Enabling VXLAN 3 Set the Value to Enabled 4 Click OK ...

Page 245: ...wing Microsoft TechNet link on Software Defined Networking https technet microsoft com en us windows server docs networking sdn software defined networking sdn Configuring Storage Spaces Direct Windows Server 2016 introduces Storage Spaces Direct which allows you to build highly available and scalable storage systems with local storage For more information refer to the following Microsoft TechnNet...

Page 246: ...r Converged System This section includes instructions to install and configure the components of a Hyper Converged system using the Windows Server 2016 The act of deploying a Hyper Converged system can be divided into the following three high level phases Deploying the Operating System Configuring the Network Configuring Storage Spaces Direct NOTE The disks used in this example are 4 400G NVMe and...

Page 247: ...ed with RDMA enabled host virtual NICs To configure the network on each server 1 Configure the physical network switch as follows a Connect all adapter NICs to the switch port b Enable the switch port and make sure that the switch port supports switch independent teaming mode and is also part of multiple VLAN networks Example Dell switch configuration no ip address mtu 9416 portmode hybrid switchp...

Page 248: ...ds Add VMNetworkAdapter SwitchName SETswitch Name SMB_1 managementOS Add VMNetworkAdapter SwitchName SETswitch Name SMB_2 managementOS d To configure the host virtual NIC to use a VLAN issue the following commands Set VMNetworkAdapterVlan VMNetworkAdapterName SMB_1 VlanId 5 Access ManagementOS Set VMNetworkAdapterVlan VMNetworkAdapterName SMB_2 VlanId 5 Access ManagementOS NOTE Network Quality of ...

Page 249: ... adapter so that the VLAN is active issue the following command Disable NetAdapter vEthernet SMB_1 Enable NetAdapter vEthernet SMB_1 Disable NetAdapter vEthernet SMB_2 Enable NetAdapter vEthernet SMB_2 g To enable RDMA on the host virtual NIC adapters issue the following command Enable NetAdapterRdma SMB1 SMB2 h To verify RDMA capabilities issue the following command Get SmbClientNetworkInterface ...

Page 250: ...e4 Include Storage Spaces Direct Inventory Network System Configuration Step 2 Creating a Cluster Create a cluster with the four nodes which was validated for cluster creation in Step 1 Running Cluster Validation Tool To create a cluster issue the following Windows PowerShell command New Cluster Name ClusterName Node MachineName1 MachineName2 MachineName3 MachineName4 NoStorage The NoStorage param...

Page 251: ...t StoragePool IsReadOnly false ErrorAction SilentlyContinue Get StoragePool IsPrimordial eq false Get VirtualDisk Remove VirtualDisk Confirm false ErrorAction SilentlyContinue Get StoragePool IsPrimordial eq false Remove StoragePool Confirm false ErrorAction SilentlyContinue Get PhysicalDisk Reset PhysicalDisk ErrorAction SilentlyContinue Get Disk Number ne null IsBoot ne true IsSystem ne true Par...

Page 252: ...erShell command creates a virtual disk with both mirror and parity resiliency on the storage pool New Volume StoragePoolFriendlyName S2D FriendlyName VirtualDiskName FileSystem CSVFS_ReFS StorageTierfriendlyNames Capacity Performance StorageTierSizes Size of capacity tier in size units example 800GB Size of Performance tier in size units example 80GB CimSession ClusterName Step 7 Creating or Deplo...

Page 253: ...er V guest drivers for hosting the Nano Server as a virtual machine GuestDrivers Basic drivers for a variety of network adapters and storage controllers This is the same set of drivers included in a Server Core installation of Windows Server 2016 Technical Preview OEMDrivers File Server role and other storage components Storage Windows Defender Antimalware including a default signature file Defend...

Page 254: ... from the NanoServer folder to a folder on your hard drive NanoServerImageGenerator psm1 Convert WindowsImage ps1 4 Start Windows PowerShell as an administrator 5 Change directory to the folder where you pasted the files from Step 3 System Center Virtual Machine Manager Agent Packages Microsoft Windows Server SCVMM Package Packages Microsoft Windows Server SCVMM Compute Package Note Use this packa...

Page 255: ...acenter MediaPath C tmp TP4_iso Bld_10586_iso BasePath Base TargetPath C Nano PhysicalSystem Nano_phy_vhd vhd ComputerName Nano server1 Compute Storage Cluster OEMDrivers DriversPath C Nano Drivers In the preceding example C Nano Drivers is the path for QLogic drivers This command takes about 10 to 15 minutes to create a VHD file A sample output for this command is shown here Windows R Image to Vi...

Page 256: ... server where you want to run the Nano Server VHD 9 To copy the VHD to the physical server and configure it to boot from the new VHD a Go to Computer Management Storage Disk Management b Right click Disk Management and select Attach VHD c Provide the VHD file path d Click OK e Run bcdboot d windows f Right click Disk Management and select Detach VHD 10 Reboot the physical server into the Nano Serv...

Page 257: ...enerator psm1 Verbose 7 Issue the following Windows PowerShell command to create a VHD that sets a computer name and includes the Hyper V guest drivers New NanoServerImage DeploymentType Guest Edition Standard Datacenter MediaPath path to root of media BasePath Base TargetPath NanoServerPhysical NanoServer vhd ComputerName computer name GuestDrivers Example New NanoServerImage DeploymentType Guest...

Page 258: ...been assigned INFO System volume location G INFO Applying image to VHD This could take a while INFO Image was applied successfully INFO Making image bootable INFO Fixing the Device ID in the BCD store on VHD INFO Drive is bootable Cleaning up INFO Closing VHD INFO Deleting pre existing VHD Base vhd INFO Closing Windows image INFO Done Done The log is at C Users ADMINI 1 AppData Local Temp 2 NanoSe...

Page 259: ...rver s administrators 3 Optional Enable CredSSP if applicable Adding the Nano Server to a List of Trusted Hosts At an elevated Windows PowerShell prompt add the Nano Server to the list of trusted hosts by issuing the following command Set Item WSMan localhost Client TrustedHosts IP address of Nano Server Examples Set Item WSMan localhost Client TrustedHosts 172 28 41 152 Set Item WSMan localhost C...

Page 260: ...ment tools and associated documentation available from the QLogic Downloads and Documentation page http driverdownloads qlogic com RoCE Configuration To manage the Nano Server with Windows PowerShell remoting 1 Connect to the Nano Server through Windows PowerShell Remoting from another machine For example PS C Windows system32 1p 172 28 41 152 PS C Windows system32 user 172 28 41 152 Administrator...

Page 261: ...apterRdma 4 To assign an IP address and VLAN ID to all interfaces of the adapter issue the following Windows PowerShell commands 172 28 41 152 PS C Set NetAdapterAdvancedProperty InterfaceAlias slot 1 port 1 RegistryKeyword vlanid RegistryValue 5 172 28 41 152 PS C netsh interface ip set address name SLOT 1 Port 1 static 192 168 10 10 255 255 255 0 5 To create SMBShare on the Nano Server issue the...

Page 262: ...achine issue the following Windows PowerShell command PS C Windows system32 net use z 192 168 10 10 smbshare This command completed successfully 7 To perform read write on SMBShare and check RDMA statistics on the Nano Sever issue the following Windows PowerShell command 172 28 41 152 PS C Get NetAdapterStatistics RdmaStatistics Figure 13 23 shows the command output Figure 13 23 Windows PowerShell...

Page 263: ...nections Verify that the cable connections at the network adapter and the switch are attached properly Verify the adapter installation by reviewing Installing the Adapter on page 10 Ensure that the adapter is properly seated in the slot Check for specific hardware problems such as obvious damage to board components or the PCI edge connector Verify the configuration settings and change them if they...

Page 264: ... drivers are loaded for your Windows Linux or VMware system Verifying Drivers in Windows See the Device Manager to view vital information about the adapter link status and network connectivity Verifying Drivers in Linux To verify that the qed ko driver is loaded properly issue the following command lsmod grep i module name If the driver is loaded the output of this command shows the size of the dr...

Page 265: ... Driver qede x x x x Verifying Drivers in VMware To verify that the VMware ESXi drivers are loaded issue the following command esxcli software vib list Testing Network Connectivity This section provides procedures for testing network connectivity in Windows and Linux environments Testing Network Connectivity for Windows Test network connectivity using the ping command To determine if the network c...

Page 266: ...r the ethtool command ethtool s ethX speed SSSS Microsoft Virtualization with Hyper V Microsoft Virtualization is a hypervisor virtualization system for Windows Server 2012 R2 For more information on Hyper V go to https technet microsoft com en us library Dn282278 aspx Linux specific Issues Problem Errors appear when compiling driver source code Solution Some installations of Linux distributions d...

Page 267: ...hat Windows OS installer reports is shown at the bottom of the Windows Setup dialog box as shown in Figure 14 1 Figure 14 1 Windows Setup Error Message In other cases the error message may indicate a need to ensure that the disk s controller is enabled in the computer s BIOS menu To avoid any of the depicted error messages ensure that there is no USB flash drive attached until the setup asks for t...

Page 268: ...01 00 B Collecting Debug Data Use the commands in Table 14 1 to collect debug data Table 14 1 Collecting Debug Data Commands Debug Data Description demesg T Kernel logs ethtool d Register dump sys_info sh System information available in the driver bundle ...

Page 269: ...ors for the state of the adapter port link and activity Table A 1 Adapter Port Link and Activity LEDs Port LED LED Appearance Network State Link LED Off No link cable disconnected Continuously illuminated Link Activity LED Off No port activity Blinking Port activity ...

Page 270: ...ers support a variety of cables and optical modules that comply with SFF8024 Specific form factor compliance is as follows SFPs SFF8472 for memory map SFF8419 or SFF8431 low speed signals and power Quad small form factor pluggable QSFP SFF8636 for memory map SFF8679 or SFF8436 low speed signals and power Optical modules electrical input output active copper cables ACC and active optical cables AOC...

Page 271: ...inued To view the most current list of supported cables and optical modules view the QLogic FastLinQ 41000 45000 Series Interoperability Matrix located here http www qlogic com Resources Documents LineCards LC_41000 45000_Interoperability_Matrix pdf Table B 1 Tested Cables and Optical Modules Speed Form Factor Manufac turer Part Number Type Cable Lengtha Gauge Cables 10G DAC b Cisco COPQAA4JAA SFP...

Page 272: ...FP28 to SFP28 3 30 HP 844471 B21 SFP28 to SFP28 0 5 26 844474 B21 SFP28 to SFP28 1 26 844477 B21 SFP28 to SFP28 3 26 844480 B21 SFP28 to SFP29 5 26 40G DAC Splitter 4 10G Dell 470 AAVO QSFP40GB to 4XSFP10GB 1 26 470 AAXG QSFP40GB to 4XSFP10GB 3 26 470 AAXH QSFP40GB to 4XSFP10GB 5 26 Table B 1 Tested Cables and Optical Modules Continued Speed Form Factor Manufac turer Part Number Type Cable Lengtha...

Page 273: ...SFP100GB to 4XSFP28GB 3 30 NDAQGJ 0001 QSFP100GB to 4XSFP28GB 1 26 NDAQGF 0002 QSFP100GB to 4XSFP28GB 2 30 Arista CAB Q 4S 100G 3M QSFP100GB to 4XSFP28GB 3 26 Optical Solutions 10G Optical Transceivers Finisar FTLX8571D3BCL QL SFP 10G Optical Transceiver SR N A N A FTLX1471D3BCL QL SFP 10G Optical Transceiver LR N A N A Avago AFBR 703SMZ SFP 10G Optical Transceiver SR N A N A AFBR 701SDZ SFP 10G O...

Page 274: ...cal Transceiver SR N A N A 10G AOC c Dell 470 ABLV SFP 10G AOC 2 N A 470 ABLZ SFP 10G AOC 3 N A 470 ABLT SFP 10G AOC 5 N A 470 ABML SFP 10G AOC 7 N A 470 ABLU SFP 10G AOC 10 N A 470 ABMD SFP 10G AOC 15 N A 470 ABMJ SFP 10G AOC 15 N A 25G AOC InnoLight TF PY020 N00 SFP28 AOC 20 N A TF PY003 N00 SFP28 AOC 3 N A a Cable length is indicated in meters b DAC is direct attach cable c AOC is active optica...

Page 275: ...nge over time as new switches enter the market or are discontinued To view the most current list of supported switches view the QLogic FastLinQ 41000 45000 Series Interoperability Matrix located here http www qlogic com Resources Documents LineCards LC_41000 45000_Interoperability_Matrix pdf Table B 2 Switches Tested for Interoperability Manufacturer Ethernet Switch Model Arista 7060X Cisco Nexus ...

Page 276: ... of the storage protocol on another PF is disallowed by those management tools Because storage personality is disabled by default only the personality that has been configured using HII or QLogic management tools is written in NVRAM configuration When this limitation is removed users can configure additional PFs on the same port for storage in NPAR Mode Concurrent RoCE and iWARP Is Not Supported o...

Page 277: ...MANICModeOnPartition is currently set to disabled and cannot be enabled NIC and SAN Boot to Base Is Supported Only on Select PFs Ethernet and PXE boot are currently supported only on PF0 and PF1 In NPAR configuration other PFs do not support Ethernet and PXE boot When the Virtualization Mode is set to NPAR non offloaded FCoE boot is supported on Partition 2 PF2 and PF3 and iSCSI boot is supported ...

Page 278: ...rt on the adapter board Advanced Configuration and Power Interface See ACPI bandwidth A measure of the volume of data that can be transmitted at a specific transmission rate A 1Gbps or 2Gbps Fibre Channel port can transmit or receive at nominal rates of 1 or 2Gbps depending on the device to which it is connected This corre sponds to actual bandwidth values of 106MB and 212MB respectively BAR Base ...

Page 279: ... system In Fibre Channel a target device DHCP Dynamic host configuration protocol Enables computers on an IP network to extract their configuration from servers that have information about the computer only after it is requested driver The software that interfaces between the file system and a physical data storage device or network media dynamic host configuration protocol See DHCP eCore A layer ...

Page 280: ...col See FTP FTP File transfer protocol A standard network protocol used to transfer files from one host to another host over a TCP based network such as the Internet FTP is required for out of band firmware uploads that will complete faster than in band firmware uploads human interface infrastructure See HII HII Human interface infrastructure A specifi cation part of UEFI 2 1 for managing user inp...

Page 281: ... TCP IP network stack to build a large up to 64KB TCP message before sending it to the adapter The adapter hardware segments the message into smaller data packets frames that can be sent over the wire up to 1 500 bytes for standard Ethernet frames and up to 9 000 bytes for jumbo Ethernet frames The segmentation process frees up the server CPU from having to segment large TCP messages into smaller ...

Page 282: ...s SSDs PCI Peripheral component interface A 32 bit local bus specification introduced by Intel PCI Express PCIe A third generation I O standard that allows enhanced Ethernet network performance beyond that of the older peripheral compo nent interconnect PCI and PCI extended PCI X desktop and server slots QoS Quality of service Refers to the methods used to prevent bottlenecks and ensure business c...

Page 283: ... device to appear as multiple separate physical PCIe devices SR IOV permits isolation of PCIe resources for performance interoperability and manageability target The storage device endpoint of a SCSI session Initiators request data from targets Targets are typically disk drives tape drives or other media devices Typically a SCSI peripheral device is the target but an adapter may in some cases be a...

Page 284: ...rotocol See UDP VF Virtual function VI Virtual interface An initiative for remote direct memory access across Fibre Channel and other communication proto cols Used in clustering and messaging virtual interface See VI virtual logical area network See VLAN virtual machine See VM VLAN Virtual logical area network LAN A group of hosts with a common set of require ments that communicate as if they were...

Page 285: ...3 server VLAN ID 64 Add Counters dialog box 202 adding host VNIC 201 VLAN ID to host VNIC 200 address MAC permanent and virtual 39 PCI 39 ADK downloading Windows 165 Advanced Configuration and Power Interface See ACPI 254 affinity settings IRQ 112 agency certifications xxii agreements license xix allocating bandwidth 51 AOC specifications supported 246 applications for adapter management 3 archite...

Page 286: ...cation KCC Class A xxiii checklist preinstallation 9 troubleshooting 239 chip type 38 Class A certification xxiii class map 59 60 Coalesce Rx Microseconds 19 Coalesce Tx Microseconds 19 collecting debug data 244 compliance product laser safety xxii product safety xxiv configuring adapter UEFI boot mode 158 DHCP server for iSCSI boot 129 FCoE 157 FCoE boot parameters 161 FCoE crash dump 164 FCoE DC...

Page 287: ...onfiguring for FCoE 164 DCBX definition of 255 disabling to configure QoS 205 enabling 59 enabling to configure QoS 210 protocol 63 protocol supported 59 debug data collecting 244 debug parameter 18 28 definitions of terms 254 deploying Hyper Converged system 222 SDN 221 Windows Nano Server 228 230 233 device definition of 255 FCoE verifying in Linux 168 ID 38 name 38 Device Manage verifying Windo...

Page 288: ...bleshooting Linux 240 troubleshooting VMware 241 troubleshooting Windows 240 upgrade VMware 27 VMware 25 VMware downloading from VMware 26 VMware parameter defaults 29 driver installation 11 kmod RPM 16 Linux without RoCE 13 RoCE inbox OFED 17 source RPM 15 TAR file Linux 16 VMware 26 Windows 20 driver removal Linux non RoCE 14 Linux RoCE 14 VMware 30 Windows 20 dynamic host configuration protocol...

Page 289: ...ng VXLAN 29 firewall 84 firmware downloading updates xx properties 40 reset 29 upgrading on Linux 33 upgrading on Windows 33 upgrading on Windows Nano 34 Firmware Upgrade Utility running in Nano Server 24 flow control default Linux 19 default VMware 29 troubleshooting Linux issue 242 FTP definition of 256 network installation 13 functional description of adapter 1 G GID index values VLAN 74 VLAN i...

Page 290: ...onics Engineers See IEEE intended audience of guide xv interfaces RoCE configuring 198 verifying Ethernet 242 Internet Protocol definition of 256 Internet small computer system interface See iSCSI Internet wide area RDMA protocol See iWARP interoperability switches 251 IP checksum offloads support 2 definition of 256 IPv4 standards support 6 IPv6 standards support 6 IQN definition of 256 initiator...

Page 291: ...buntu 16 FCoE devices verifying 168 FCoE offload configuring 166 firmware updating on 33 iSCSI boot from SAN configuring for SLES 12 133 iSER and iWARP on 103 issues troubleshooting 242 iWARP configuring 91 kernel module qed ko 148 166 minimum host OS 8 performance optimizing 111 SR IOV configuring 177 Linux drivers See also Linux downloading from QLogic 13 17 installing 11 installing with RDMA 17...

Page 292: ...98 network adapters VMkernel 84 connectivity 241 installations 13 network connectivity testing Linux 242 Windows 241 Network Direct Functionality enabling 87 Network Direct Kernel Provider Interface RoCE interfaces 198 network I O control 83 network interface card See NIC network state indicators 245 Nexus 6000 Ethernet switch 59 NFS network installation 13 NIC partitioning See NPAR non volatile m...

Page 293: ...oCE traffic 203 performance RoCE 67 peripheral component interface See PCI PFC feature support 2 priority 61 PFs definition of 258 max PFs per 28 physical characteristics adapter 5 physical function See PF pnputil tool adding and installing driver package with 22 deleting oem0 inf package with 23 upgrading or installing drivers with 24 policy map 59 60 port activity indicator 245 port group 83 por...

Page 294: ... Suite See QCS CLI QLogic FastLinQ 41000 45000 Series Interoperability Matrix accessing 247 251 QoS definition of 258 configuring 60 configuring by disabling DCBX 205 configuring by enabling DCBX 210 QPs setting VMMQ max 214 QSFPs supported specifications 246 quality of service See QoS queues Tx Rx 28 30 R RDMA 57 definition of 258 adapters supported 66 applications 58 configuring interfaces 80 co...

Page 295: ...transport_fc ko in Linux FCoE offload 166 scsi_transport_iscsi ko in Linux iSCSI offload 148 SCSI definition of 259 SDN deploying 221 in RoCE over SET 203 SerDes interface for DAC transceiver 2 serializer deserializer See SerDes server message block See SMB SET creating virtual switch with 204 defined 203 procedures 203 RDMA traffic running 205 VLAN ID assigning 204 SFF8024 compliant specification...

Page 296: ...ffloads support 2 definition of 259 TCP segmentation offload See TSO technical support xx contacting xxi knowledge database xxi training from xx warranty xxi terms defined 254 tested cables and optical modules 247 tested switches 251 testing network connectivity Linux 242 Windows 241 TLV definition of 259 iSCSI lossless 59 parameters 62 RoCE defined 213 TPA feature enabling disabling 28 traffic RD...

Page 297: ...0 VI definition of 260 vib file 27 26 virtual extensible LAN See VXLAN virtual function See VF virtual interface definition of 260 virtual LAN See VLAN virtual logical area network See VLAN virtual machine See VM virtual NIC See VNIC virtual switch assigning to VMNetworkadapter 174 creating 199 creating with SR IOV 172 Hyper V creating 199 204 RSS enabling 214 Virtual Switch Manager creating VM sw...

Page 298: ...arranty product xxi what s in this guide xv Windows Assessment and Deployment Kit downloading 165 drivers installing 20 drivers removing 20 drivers verifying current 240 FCoE boot from SAN 163 firmware updating on 33 Hyper Converged system deploying 222 image files injecting 165 iWARP configuring 86 SR IOV configuring 170 Storage Spaces Direct configuring 221 Windows Nano Server deploying and mana...

Page 299: ... All other brand and product names are trademarks or registered trademarks of their respective owners This document is provided for informational purposes only and may contain errors Cavium reserves the right without notice to make changes to this document or in product design or specifications Cavium disclaims any warranty of any kind expressed or implied and does not guarantee that any results o...