Cavium QL41112HFCU-DE, User Manual

Page 1: ...AH0054602 00 A Third party information brought to you courtesy of Dell EMC User s Guide Converged Network Adapters QLogic 41xxx Series ...

Page 2: ...ii AH0054602 00 A User s Guide Converged Network Adapters QLogic 41xxx Series Document Revision History Revision A April 28 2017 Changes Sections Affected Initial release of new guide for Dell ...

Page 3: ...roduct Overview Functional Description 1 Features 1 Adapter Specifications 3 Physical Characteristics 3 Standards Specifications 3 2 Hardware Installation System Requirements 4 Safety Precautions 5 Preinstallation Checklist 5 Installing the Adapter 6 3 Driver Installation Installing Linux Driver Software 8 Installing the Linux Drivers Without RoCE 10 Removing the Linux Drivers 10 Installing Linux ...

Page 4: ...er Software 26 VMware Drivers and Driver Packages 27 Installing the VMware Driver 28 VMware Driver Optional Parameters 29 VMware Driver Parameter Defaults 31 Removing the VMware Driver 31 FCoE Support 32 iSCSI Support 32 4 Upgrading the Firmware Running the DUP by Double Clicking 33 Running the DUP from a Command Line 35 Running the DUP Using the bin File 36 5 Adapter Preboot Configuration Getting...

Page 5: ... 69 7 iSCSI Configuration iSCSI Boot 71 iSCSI Boot Setup 72 Selecting the Preferred iSCSI Boot Mode 72 Configuring the iSCSI Target 73 Configuring iSCSI Boot Parameters 73 Adapter UEFI Boot Mode Configuration 75 Configuring iSCSI Boot 78 Static iSCSI Boot Configuration 79 Dynamic iSCSI Boot Configuration 86 Enabling CHAP Authentication 88 Configuring the DHCP Server to Support iSCSI Boot 89 DHCP i...

Page 6: ...de 111 Windows FCoE Boot from SAN 116 Windows Server 2012 2012 R2 and 2016 FCoE Boot Installation 116 Configuring FCoE 117 FCoE Crash Dump 117 Injecting Slipstreaming Adapter Drivers into Windows Image Files 118 Configuring Linux FCoE Offload 119 Differences Between qedf and bnx2fc 120 Configuring qedf ko 120 Verifying FCoE Devices in Linux 121 Boot from SAN Considerations 122 9 iWARP Configuratio...

Page 7: ...witch Embedded Teaming 162 Creating a Hyper V Virtual Switch with SET and RDMA Virtual NICs 163 Enabling RDMA on SET 163 Assigning a VLAN ID on SET 163 Running RDMA Traffic on SET 164 Configuring QoS for RoCE 164 Configuring QoS by Disabling DCBX on the Adapter 164 Configuring QoS by Enabling DCBX on the Adapter 169 Configuring VMMQ 172 Enabling VMMQ on the Adapter 173 Setting the VMMQ Max QPs Def...

Page 8: ...PowerShell Remoting 192 Adding the Nano Server to a List of Trusted Hosts 192 Starting the Remote Windows PowerShell Session 193 Managing QLogic Adapters on a Windows Nano Server 193 RoCE Configuration 193 13 Troubleshooting Troubleshooting Checklist 197 Verifying that Current Drivers Are Loaded 198 Windows 198 Linux 198 VMware 199 Testing Network Connectivity 199 Testing Network Connectivity for ...

Page 9: ... Setup Main Configuration Page NPAR Mode 40 5 5 Firmware Information Window 41 5 6 Dell System Setup NIC Configuration 43 5 7 Dell System Setup NIC Configuration Boot Mode 45 5 8 Dell System Setup FCoE General Parameters 46 5 9 Dell System Setup FCoE Configuration 46 5 10 Dell System Setup iSCSI General Parameters 48 5 11 Dell System Setup iSCSI Initiator Parameters 49 5 12 Dell System Setup iSCSI...

Page 10: ... Configuration 113 8 4 Dell System Setup FCoE Mode Enabled 114 8 5 Dell System Setup FCoE General Parameters 115 8 6 Dell System Setup FCoE General Parameters 116 9 1 Dell System Setup for iWARP NIC Configuration 124 9 2 PowerShell Command Get NetAdapterRdma 125 9 3 PowerShell Command Get NetOffloadGlobalSetting 125 9 4 Perfmon Add Counters 126 9 5 Perfmon Verifying iWARP Traffic 126 9 6 LIO Targe...

Page 11: ...mand Get NetAdapter 163 12 9 Advanced Properties Enable QoS 165 12 10 Advanced Properties Setting VLAN ID 166 12 11 Advanced Properties Enabling QoS 170 12 12 Advanced Properties Setting VLAN ID 171 12 13 Advanced Properties Enabling Virtual Switch RSS 173 12 14 Advanced Properties Setting VMMQ 174 12 15 Virtual Switch Manager 175 12 16 PowerShell Command Get VMSwitch 176 12 17 Advanced Properties...

Page 12: ...rs 29 3 7 VMware Driver Parameter Defaults 31 3 8 QLogic 41xxx Series Adapter VMware FCoE Driver 32 3 9 QLogic 41xxx Series Adapter iSCSI Driver 32 5 1 Adapter Properties 41 5 2 Device level Parameters 43 6 1 Operating System Support for RoCE and OFED 54 6 2 Advanced Properties for RoCE 58 7 1 Configuration Options 74 7 2 DHCP Option 17 Parameter Definitions 89 7 3 DHCP Option 43 Sub option Defini...

Page 13: ...nverged Network Adapter full height bracket QL41262HFCU DE 10 25Gb Converged Network Adapter low profile bracket QL41262HLCU DE Intended Audience This guide is intended for system administrators and other technical staff members responsible for configuring and managing adapters installed on Dell PowerEdge servers in Windows Linux or VMware environments What Is in This Guide Following this preface ...

Page 14: ...9 iWARP Configuration provides procedures for configuring Internet wide area RDMA protocol iWARP on Windows and Linux systems Chapter 10 SR IOV Configuration provides procedures for configuring single root input output virtualization SR IOV on Windows Linux and VMware systems Chapter 11 iSCSI Extensions for RDMA describes how to configure iSCSI Extensions for RDMA iSER for RHEL and SLES Chapter 12...

Page 15: ...tallation Checklist on page 6 For more information visit www qlogic com Text in bold font indicates user interface elements such as a menu items buttons check boxes or column headings For example Click the Start button point to Programs point to Accessories and then click Command Prompt Under Notification Options select the Warning Alarms check box Text in Courier font indicates a file name direct...

Page 16: ...ackets indicate a variable whose value you must specify For example serial_number square brackets indicate an optional parameter For example file_name means specify a file name or omit it to select the default file name vertical bar indicates mutually exclusive options select one option only For example on off 1 2 3 4 ellipsis indicates that the preceding item may be repeated For example x means o...

Page 17: ...eries Adapters to comply with emission immunity and product safety standards EMI and EMC Requirements FCC Part 15 compliance Class A FCC compliance information statement This device complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 this device may not cause harmful interference and 2 this device must accept any interference received including interferenc...

Page 18: ...6 RF Conducted Susceptibility EN61000 4 8 Power Frequency Magnetic Field EN61000 4 11 Voltage Dips and Interrupt VCCI 2010 04 Class A AS NZS CISPR22 Class A KCC Class A Korea RRA Class A Certified Korean Language Format Class A Product Name Model Converged Network Adapters and Intelligent Ethernet Adapters Certification holder QLogic Corporation Manufactured date Refer to date code listed on produ...

Page 19: ...ty UL60950 1 2nd Edition 2007 03 3 27 UL CSA C22 2 60950 1 07 2nd Edition Use only with listed ITE or equivalent Complies with 21 CFR 1040 10 and 1040 11 2006 95 EC low voltage directive TUV EN60950 1 2006 A11 A1 A12 IEC60950 1 2nd Edition 2005 CB CB Certified to IEC 60950 1 2nd Edition ...

Page 20: ...sing the operating system s teaming feature you can split your network into virtual LANs VLANs as well as group multiple network adapters together into teams to provide network load balancing and fault tolerance For more information about teaming see your operating system documentation Features The 41xxx Series Adapters provide the following features Some features may not be available on all adapt...

Page 21: ...atures Jumbo frames up to 9 600 bytes The OS and the link partner must support jumbo frames Virtual LANs VLAN Flow control IEEE Std 802 3x Logical link control IEEE Std 802 2 High speed on chip reduced instruction set computer RISC processor Integrated 96KB frame buffer memory not applicable to all models 1 024 classification filters not applicable to all models Support for multicast addresses thr...

Page 22: ...for use in a standard PCIe slot Standards Specifications Supported standards specifications include PCI Express Base Specification rev 3 0 PCI Express Card Electromechanical Specification rev 3 0 PCI Bus Power Management Interface Specification rev 1 2 IEEE Specifications 802 3 2012 IEEE Standard for Ethernet flow control 802 1q VLAN 802 1AX Link Aggregation 802 1ad QinQ 802 1p Priority Encoding 1...

Page 23: ... 2 For a complete list of supported operating systems visit the QLogic Downloads and Documentation page driverdownloads qlogic com Table 2 1 Host Hardware Requirements Hardware Requirement Architecture IA 32 or EMT64 that meets operating system requirements PCIe PCIe Gen2 x8 2x25G NIC x16 PCIe Gen3 x8 2x25G NIC x16 Full dual port 25Gb bandwidth is supported on PCIe Gen3 x8 or faster slots Memory 8...

Page 24: ...ers NOTE Table 2 2 denotes minimum host OS requirements For the latest list of supported OSs visit www qlogic com WARNING The adapter is being installed in a system that operates with voltages that can be lethal Before you open the case of your system observe the following precautions to protect yourself and to prevent damage to the system components Remove any metallic objects or jewelry from you...

Page 25: ... the power outlet and that you are following proper electrical grounding procedures 2 Open the system case and select the slot that matches the adapter size which can be PCIe Gen 2 x8 or PCIe Gen 3 x8 A lesser width adapter can be seated into a greater width slot x8 in an x16 but a greater width adapter cannot be seated into a lesser width slot x8 in an x4 If you do not know how to identify a PCIe...

Page 26: ...r with the adapter clip or screw 7 Close the system case and disconnect any personal anti static devices CAUTION Do not use excessive force when seating the card as this may damage the system or the adapter If you have difficulty seating the adapter remove it realign it and try again ...

Page 27: ...ameters Linux Driver Parameter Defaults Linux Driver Messages Statistics The 41xxx Series Adapter Linux drivers and supporting documentation are available on the Dell Support page dell support com Table 3 1 describes the 41xxx Series Adapter Linux drivers Table 3 1 QLogic 41xxx Series Adapters Linux Drivers Linux Driver Description qed The qed interfaces with the qede qedr qedi and qedf drivers Th...

Page 28: ...ede Linux Ethernet driver for the 41xxx Series Adapter This driver directly controls the hard ware and is responsible for sending and receiving Ethernet packets on behalf of the Linux host networking stack This driver also receives and processes device interrupts on behalf of itself for L2 networking The qede driver requires Linux kernel version 2 6 32 or later Testing was concentrated on the x86_...

Page 29: ...Removing the Linux Drivers There are two procedures for removing Linux drivers one for a non RoCE environment and another for an RoCE environment Choose the procedure that matches your environment To remove Linux drivers in a non RoCE environment unload and remove the drivers Follow the procedure that relates to the original installation method and the OS If the Linux drivers were installed using ...

Page 30: ...ede ko qedr ko To remove Linux drivers in a non RoCE environment 1 To get the path to the currently installed drivers issue the following command modinfo driver name 2 Unload and remove the Linux drivers If the Linux drivers were installed using an RPM package issue the following commands modprobe r qede depmod a rpm e qlgc fastlinq kmp default version arch If the Linux drivers were installed usin...

Page 31: ...x drivers modprobe r qedr modprobe r qede modprobe r qed depmod a 3 Remove the driver module files If the drivers were installed using an RPM package issue the following command rpm e qlgc fastlinq kmp default version arch If the drivers were installed using a TAR file issue the following commands for your operating system For RHEL cd lib modules version extra qlgc fastlinq rm rf qed ko qede ko qe...

Page 32: ...paths For SLES lib modules version updates qlgc fastlinq For RHEL lib modules version extra qlgc fastlinq 4 Turn on all ethX interfaces ifconfig ethX up 5 For SLES use YaST to configure the Ethernet interfaces to automatically start at boot by setting a static IP address or enabling DHCP on the interface Installing Linux Drivers Using the kmp kmod RPM Package To install kmod RPM package 1 Issue th...

Page 33: ...RHEL lib modules version extra qlgc fastlinq 3 Test the drivers by loading them unload the existing drivers first if necessary rmmod qede rmmod qed modprobe qed modprobe qede Installing the Linux Drivers with RoCE To install Linux drivers in an inbox OFED environment 1 Download the 41xxx Series Adapter Linux drivers from Dell dell support com 2 Configure RoCE on the adapter as described in Configu...

Page 34: ... Optional Parameters Parameter Description debug Controls driver verbosity level similar to ethtool s dev msglvl int_mode Controls interrupt mode other than MSI X gro_enable Enables the hardware generic receive offload GRO fea ture under development This is similar to the kernel s soft ware GRO but only performed by the device hardware err_flags_override A bitmap for disabling or forcing the actio...

Page 35: ...ues and bits 16 and greater are driver specific Statistics To view detailed statistics and configuration information use the ethtool utility See the ethtool man page for more information Table 3 3 Linux Driver Parameter Defaults Parameter qed Driver Default qede Driver Default Speed Auto negotiation with speed advertised Auto negotiation with speed advertised MSI MSI X Enabled Enabled Flow Control...

Page 36: ...Options Installing the Windows Drivers Install Windows driver software using the Dell Update Package DUP Running the DUP in the GUI DUP Installation Options DUP Installation Examples Running the DUP in the GUI To run the DUP in the GUI 1 Double click the icon representing the Dell Update Package file NOTE Currently Windows supports only 25G 41xxx Series Adapters NOTE The actual file name of the De...

Page 37: ... AH0054602 00 A 2 In the Dell Update Package window Figure 3 1 click Install Figure 3 1 Dell Update Package Window 3 In the QLogic Super Installer InstallShield Wizard s Welcome window Figure 3 2 click Next Figure 3 2 QLogic InstallShield Wizard Welcome Window ...

Page 38: ...nse Agreement b To continue select I accept the terms in the license agreement c Click Next Figure 3 3 QLogic InstallShield Wizard License Agreement Window 5 Complete the wizard s Setup Type window Figure 3 4 as follows a Select one of the following setup types Click Complete to install all program features Click Custom to manually select the features to be installed b To continue click Next ...

Page 39: ...stall By default all features are selected To change a feature s install setting click the icon next to it and then select one of the following options This feature will be installed on the local hard drive Marks the feature for installation without affecting any of its subfeatures This feature and all subfeatures will be installed on the local hard drive Marks the feature and all of its subfeatur...

Page 40: ...ue Figure 3 5 InstallShield Wizard Custom Setup Window 7 In the InstallShield Wizard s Ready To Install window Figure 3 6 click Install The InstallShield Wizard installs the QLogic Adapter drivers and Management Software Installer Figure 3 6 InstallShield Wizard Ready to Install the Program Window ...

Page 41: ...dismiss the installer Figure 3 7 InstallShield Wizard Completed Window 9 In the Dell Update Package window Figure 3 8 Update installer operation was successful indicates completion Optional To open the log file click View Installation Log The log file shows the progress of the DUP installation any previous installed versions any error messages and other information about the installation ...

Page 42: ... components to a directory drivers path To install or update only the driver components driveronly Advanced Use the passthrough option to send all text following passthrough directly to the QLogic installation software of the DUP This mode suppresses any provided GUIs but not necessarily those of the QLogic software passthrough Advanced To return a coded description of this DUP s supported feature...

Page 43: ...e exe s driveronly To change from the default log location to C my path with spaces log txt DUP_file_name exe l C my path with spaces log txt Removing the Windows Drivers To remove the Windows drivers 1 In the Control Panel click Programs and then click Programs and Features 2 In the list of programs select QLogic FastLinQ Driver Installer and then click Uninstall 3 Follow the instructions to remo...

Page 44: ...tion Installing Windows Driver Software 25 AH0054602 00 A 3 On the Advanced page Figure 3 9 select an item under Property and then change the Value for that item as needed Figure 3 9 Setting Advanced Adapter Properties ...

Page 45: ...nto hibernation To have the controller remain on at all times do not select the Allow the computer to turn off the device to save power check box Figure 3 10 Figure 3 10 Power Management Options Installing VMware Driver Software This section describes the qedentv VMware ESXi driver for the 41xxx Series Adapters VMware Drivers and Driver Packages Installing the VMware Driver VMware Driver Optional ...

Page 46: ...ling the VMware Driver Procedures in the individual driver Read Me files Table 3 4 VMware Drivers VMware Drivers Description qedentv Native networking driver qedrntv Native RDMA Offload RoCEv1 and RoCEv2 driver a a The certified RoCE driver is not included in this release The uncertified driver may be available as an early preview qedf Native FCoE Offload driver qedil Legacy iSCSI Offload driver T...

Page 47: ...nto an account with administrator authority 3 Unzip the driver ZIP file and then extract the vib file 4 To use the Linux scp utility to copy a vib file from a local system into the tmp directory on an ESX server with IP address 10 10 10 10 issue the following command scp qedentv 1 0 3 11 1OEM 550 0 0 1331820 x86_64 vib root 10 10 10 10 tmp You can place the file anywhere that is accessible to the ...

Page 48: ... can be supplied as command line arguments to the esxcfg module command Table 3 6 VMware Driver Optional Parameters Parameter Description hw_vlan Globally enables 1 or disables 0 hardware VLAN insertion and removal Disable this parameter when the upper layer needs to send or receive fully formed packets hw_vlan 1 is the default num_queues Specifies the number of TX RX queue pairs num_queues can be...

Page 49: ...rmation level includes the Notice level 0x3FFFFFFF indicates Verbose level for all driver submodules includes the Information and Notice levels auto_fw_reset Enables 1 or disables 0 the driver automatic firmware recovery capability When this parameter is enabled the driver attempts to recover from events such as transmit timeouts firmware asserts and adapter parity errors The default is auto_fw_re...

Page 50: ...Driver Parameter Defaults Parameter Default Speed Autonegotiation with all speeds advertised The speed parameter must be the same on all ports If auto negotiation is enabled on the device all of the device ports will use autonegotiation Flow Control Autonegotiation with RX and TX advertised MTU 1 500 range 46 9 600 Rx Ring Size 8 192 range 128 8 192 Tx Ring Size 8 192 range 128 8 192 MSI X Enabled...

Page 51: ...e FCoE Driver Driver Description qedf The QLogic VMware FCoE driver is a kernel mode driver that pro vides a translation layer between the VMware SCSI stack and the QLogic FCoE firmware and hardware Table 3 9 QLogic 41xxx Series Adapter iSCSI Driver Driver Description qedil The qedil driver is the QLogic VMware iSCSI HBA driver Similar to qedf qedil is a kernel mode driver that provides a translat...

Page 52: ...n run the firmware DUP from the command line with several supported command line options Running the DUP by Double Clicking Running the DUP from a Command Line Running the DUP Using the bin File Linux only Running the DUP by Double Clicking To run the firmware DUP by double clicking the executable file 1 Double click the icon representing the firmware Dell Update Package file The Dell Update Packa...

Page 53: ... dialog box click Yes to continue the installation The installer indicates that it is loading the new firmware as shown in Figure 4 2 Figure 4 2 Dell Update Package Loading New Firmware When complete the installer indicates the result of the installation as shown in Figure 4 3 Figure 4 3 Dell Update Package Installation Results ...

Page 54: ... 4 4 Dell Update Package Finish Installation Running the DUP from a Command Line Running the firmware DUP from the command line with no options specified results in the same behavior as double clicking the DUP icon Note that the actual file name of the DUP will vary To run the firmware DUP from a command line Issue the following command C Network_Firmware_2T12N_WN32_ version _X16 EXE ...

Page 55: ... the bin File The following procedure is supported only on Linux OS To update the DUP using the bin file 1 Copy the Network_Firmware_NJCX1_LN_X Y Z BIN file to the system under test SUT 2 Change the file type into an executable file as follows chmod 777 Network_Firmware_NJCX1_LN_X Y Z BIN 3 To start the update process issue the following command Network_Firmware_NJCX1_LN_X Y Z BIN 4 After the firm...

Page 56: ...ion of this Update Package is the same as the currently installed version Software application name BCM57810 10 Gigabit Ethernet rev 10 p2p2 Package version 08 07 26 Installed version 08 07 26 Continue Y N Y Y entered update was forced by user Executing update WARNING DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE UPDATE IS IN PROGRESS THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME U...

Page 57: ...el Parameters Configuring Port level Parameters Configuring FCoE Boot Configuring iSCSI Boot Configuring Partitions Getting Started To start the HII application 1 Open the System Setup window for your platform For information about launching the System Setup consult the user guide for your platform 2 In the System Setup window Figure 5 1 select Device Settings and then press ENTER NOTE The HII scr...

Page 58: ...Settings window Figure 5 2 select the 41xxx Series Adapter port that you want to configure and then press ENTER Figure 5 2 Dell System Setup Device Settings 4 The Main Configuration Page presents the adapter management options If you are not using NPAR set Partitioning Mode to Default as shown in Figure 5 3 ...

Page 59: ...00 A Figure 5 3 Dell System Setup Main Configuration Page Default Mode Setting Partitioning Mode to NPAR adds the Partitions Configuration option to the Main Configuration Page as shown in Figure 5 4 Figure 5 4 Dell System Setup Main Configuration Page NPAR Mode ...

Page 60: ...image version which comprises several firmware images MFW Version is the management firmware version UEFI Driver Version is the unified extensible firmware interface UEFI or extensible firmware interface EFI driver version Figure 5 5 Firmware Information Window Table 5 1 Adapter Properties Adapter Property Description Partitioning Mode Default or NPAR Device Name Factory assigned device name Chip ...

Page 61: ...protocol RoCE priority MFW Crash Dump Feature Virtualization Mode NPAREP Mode To configure device level parameters 1 On the Main Configuration Page select NIC Configuration see Figure 5 3 on page 40 and then press ENTER 2 On the NIC Configuration page select values for the device level parameters as shown in Figure 5 6 NOTE The iSCSI physical function PF is enumerated when the iSCSI Offload featur...

Page 62: ...ard error correction FEC must match that of the connected switch or device port Auto Negotiated default specifies that dual port 25Gb adapters auto negotiate available speeds and link parameters FEC Pause over DAC connections a FEC is negotiated in Auto Negotiated mode 10GBASE T is also auto negotiated per standards 1Gbps sets the port link speed to 1Gbps Select this option only when a 1G trans ce...

Page 63: ...g Boot Mode options as needed Set Boot Mode to FCoE to enable FCoE offload Set Boot Mode to iSCSI HW to enable iSCSI offload FEC Mode FEC options are available when the Link Speed is set explicitly to 25Gbps on 25Gb adapters None Disables FEC Fire Code Enables FEC BaseR FEC on 25Gb adapters Boot Mode Enables UNDI UEFI PXE boot FCoE boot from SAN iSCSI boot or None SR IOV Enables Enabled or disable...

Page 64: ...ring FCoE Boot FCoE general parameters include the following FIP VLAN ID Fabric Login Retry Count Target Login Retry Count To configure the FCoE boot configuration parameters 1 On the Main Configuration Page select FCoE Boot Configuration Menu and then select one of the following options FCoE General Parameters Figure 5 8 FCoE Configuration Figure 5 9 2 Press ENTER 3 Choose values for the FCoE Gen...

Page 65: ...5 Adapter Preboot Configuration Configuring FCoE Boot 46 AH0054602 00 A Figure 5 8 Dell System Setup FCoE General Parameters Figure 5 9 Dell System Setup FCoE Configuration 4 Click Back ...

Page 66: ...get Configuration iSCSI Second Target Configuration 2 Press ENTER 3 Choose values for the appropriate iSCSI configuration parameters iSCSI General Parameters Figure 5 10 on page 48 TCP IP Parameters Via DHCP iSCSI Parameters Via DHCP CHAP Authentication IP Version ARP Redirect DHCP Request Timeout Target Login Timeout DHCP Vendor ID iSCSI Initiator Parameters Figure 5 11 on page 49 IPv4 Address IP...

Page 67: ...A iSCSI Second Target Parameters Figure 5 13 on page 50 IPv4 Address TCP Port Boot LUN iSCSI Name CHAP ID CHAP Secret 4 Click Back 5 When prompted click Yes to save the changes Changes take effect after a system reset Figure 5 10 Dell System Setup iSCSI General Parameters ...

Page 68: ...5 Adapter Preboot Configuration Configuring iSCSI Boot 49 AH0054602 00 A Figure 5 11 Dell System Setup iSCSI Initiator Parameters Figure 5 12 Dell System Setup iSCSI First Target Parameters ...

Page 69: ...5 Adapter Preboot Configuration Configuring iSCSI Boot 50 AH0054602 00 A Figure 5 13 Dell System Setup iSCSI Second Target Parameters ...

Page 70: ...allocations 1 On the Main Configuration Page select Partitions Configuration and then press ENTER 2 On the Partitions Configuration page Figure 5 14 select Global Bandwidth Allocation Figure 5 14 Partitions Configuration Page 3 On the Global Bandwidth Allocation page Figure 5 15 click each partition minimum and maximum TX bandwidth field for which you want to allocate bandwidth There are eight par...

Page 71: ... value The total of the minimum TX bandwidth values of all partitions on a single port must equal 100 or be all zeros Partition n Maximum TX Bandwidth is the maximum transmit bandwidth of the selected partition expressed as a percentage of the maximum physical port link speed Values can be 1 100 The per partition maximum TX bandwidth value applies regardless of the DCBX ETS mode setting If the max...

Page 72: ...Configuration page Figure 5 14 on page 51 select Partition n Configuration For example selecting Partition 1 Configuration opens the Partition 1 Configuration page Figure 5 16 which shows the Partition 1 parameters Personality PCI Device ID PCI bus Address Permanent MAC Address and Virtual MAC Address Figure 5 16 Partition 1 Configuration ...

Page 73: ...E on the Adapter for Linux Supported Operating Systems and OFED Table 6 1 shows the operating system support for RoCE v1 v2 and OFED NOTE Some RoCE features may not be fully enabled in the current release For details refer to Appendix D Feature Constraints Table 6 1 Operating System Support for RoCE and OFED Operating System RoCE v1 RoCE v2 Inbox OFED Out of Box OFED Windows Server 2012 Yes Yes No...

Page 74: ...llowing command ib_send_bw version Building libqedr in inbox OFED requires installing libibverbs devel Running user space applications in inbox OFED requires installing the InfiniBand Support group by yum groupinstall InfiniBand Support that contains libibcm libibverbs and more OFED and RDMA applications that depend on libibverbs also require the QLogic RDMA user space library libqedr Install libq...

Page 75: ...ult lossy traffic class 3 is used for the FCoE traffic class and 4 is used for lossless iSCSI TLV over DCB traffic class 4 Click Back 5 When prompted click Yes to save the changes Changes will not take effect until after a system reset For Windows you can configure DCBX using the HII or QoS method The configuration shown in this section is through HII For QoS refer to Configuring QoS for RoCE on p...

Page 76: ...h In this example use a value of 50 percent switch config policy map type queuing roce switch config class type queuing class roce switch config bandwidth percent 50 7 Configure network QoS policy maps to set priority flow control for no drop traffic class switch config policy map type network qos roce switch config class type network qos class roce switch config pause no drop 8 Apply the new poli...

Page 77: ...RoCE on the adapter a Open the Windows Device Manager and then open the 41xxx Series Adapters NDIS Miniport Properties b On the QLogic FastLinQ Adapter Properties click the Advanced tab 2 On the Advanced page configure the properties listed in Table 6 2 by selecting each item under Property and choosing an appropriate Value for that item Then click OK Table 6 2 Advanced Properties for RoCE Propert...

Page 78: ...E is enabled on the adapter using Windows PowerShell The Get NetAdapterRdma command lists the adapters that support RDMA both ports are enabled PS C Users Administrator Get NetAdapterRdma Name InterfaceDescription Enabled SLOT 4 3 Port 1 QLogic FastLinQ QL45212 True SLOT 4 3 Port 2 QLogic FastLinQ QL45212 True NOTE If you are configuring RoCE over Hyper V do not assign a VLAN ID to the physical in...

Page 79: ...rect establishes two RDMA connections per port which provides good performance including line rate at a higher block size for example 64KB To optimize performance you can change the quantity of RDMA connections per RDMA interface to four or greater To increase the quantity of RDMA connections to four or more issue the following command in PowerShell PS C Users Administrator Set ItemProperty Path H...

Page 80: ...fault RoCE GID Index and address issue the ibv_devinfo command and compare it with the sys or class parameters For example ibv_devinfo d qedr0 v grep GID GID 0 fe80 0000 0000 0000 020e 1eff fec4 1b20 GID 1 fe80 0000 0000 0000 020e 1eff fec4 1b20 GID 2 0000 0000 0000 0000 0000 ffff 1e01 010a GID 3 0000 0000 0000 0000 0000 ffff 1e01 010a GID 4 3ffe ffff 0000 0f21 0000 0000 0000 0004 GID 5 3ffe ffff ...

Page 81: ...00 020e 1eff fec4 1b20 v1 p4p1 qedr0 1 1 fe80 0000 0000 0000 020e 1eff fec4 1b20 v2 p4p1 qedr0 1 2 0000 0000 0000 0000 0000 ffff 1e01 010a 30 1 1 10 v1 p4p1 qedr0 1 3 0000 0000 0000 0000 0000 ffff 1e01 010a 30 1 1 10 v2 p4p1 qedr0 1 4 3ffe ffff 0000 0f21 0000 0000 0000 0004 v1 p4p1 qedr0 1 5 3ffe ffff 0000 0f21 0000 0000 0000 0004 v2 p4p1 qedr0 1 6 0000 0000 0000 0000 0000 ffff c0a8 6403 192 168 1...

Page 82: ...F x 1 Client ib_send_bw d qedr0 F x 1 192 168 100 3 Verifying RoCE v2 Through Different Subnets To verify RoCE v2 through different subnets 1 Set the route configuration for the server and client using DCBX PFC configuration System Settings Server VLAN IP 192 168 100 3 and Gateway 192 168 100 1 Client VLAN IP 192 168 101 3 and Gateway 192 168 101 1 Server Configuration sbin ip link add link p4p1 n...

Page 83: ... following procedure Use any flow control method Pause DCBX CEE or DCBX IEEE and enable IP routing for RoCE v2 See Preparing the Ethernet Switch on page 56 for RoCE v2 configuration or refer to the vendor switch documents If you are using PFC configuration and L3 routing run RoCE v2 traffic over the VLAN using a different subnet and use RoCE v2 VLAN GID index Server ib_send_bw d qedr0 F x 5 Client...

Page 84: ...s for RDMA_CM Applications Use the following scripts from the FastLinQ source package to configure RoCE show_rdma_cm_roce_ver sh qedr0 is configured to IB RoCE v1 qedr1 is configured to IB RoCE v1 config_rdma_cm_roce_ver sh v2 configured rdma_cm for qedr0 to RoCE v2 configured rdma_cm for qedr1 to RoCE v2 Server Settings Figure 6 4 Configuring RDMA_CM Applications Server ...

Page 85: ... the following RPMs from the RHEL ISO image libibverbs devel x x x x86_64 rpm required for libqedr library perftest x x x x86_64 rpm required for InfiniBand bandwidth and latency applications or using Yum install the inbox OFED yum groupinstall Infiniband Support yum install perftest yum install tcl tcl devel tk zlib devel libibverbs libibverbs devel 3 Install the new Linux drivers as described in...

Page 86: ...n Installing the Linux Drivers with RoCE on page 14 Verifying the RoCE Configuration on Linux After installing OFED installing the Linux driver and loading the RoCE drivers verify that the RoCE devices were detected on all Linux operating systems To verify RoCE configuration on Linux 1 Stop firewall tables using service systemctl commands 2 For RHEL only verify if the RDMA service has started Use ...

Page 87: ... IP address and enable the port using a configuration method such as ifconfig ifconfig ethX 192 168 10 10 24 up 6 Issue the ibv_devinfo command For each PCI function you should see a separate hca_id as shown in the following example root captain ibv_devinfo hca_id qedr0 transport InfiniBand 0 fw_ver 8 3 9 0 node_guid 020e 1eff fe50 c7c0 sys_image_guid 020e 1eff fe50 c7c0 vendor_id 0x1077 vendor_pa...

Page 88: ...QPN 0xff0000 PSN 0xb3e07e GID fe80 20e 1eff fe50 c7c0 8192000 bytes in 0 02 seconds 4211 28 Mbit sec 1000 iters in 0 02 seconds 15 56 usec iter To display RoCE statistics issue the following commands where X is the device number mount t debugfs nodev sys kernel debug cat sys kernel debug qedr qedrX stats VLAN Interfaces and GID Index Values If you are using VLAN interfaces on both the server and t...

Page 89: ... AH0054602 00 A NOTE The default GID value is zero 0 for back to back or Pause settings For server switch configurations you must identify the proper GID value If you are using a switch refer to the corresponding switch configuration documents for the proper settings ...

Page 90: ...oot to enable network boot of operating systems to diskless systems iSCSI boot allows a Windows Linux or VMware operating system boot from an iSCSI target machine located remotely over a standard IP network For both Windows and Linux operating systems iSCSI boot can be configured to boot with two distinctive paths iSCSI SW also known as non offload path with Microsoft Open iSCSI initiator iSCSI HW...

Page 91: ... Parameters Selecting the Preferred iSCSI Boot Mode Boot mode option is listed under iSCSI Configuration Figure 7 1 of the adapter and the setting is port specific Refer to the OEM user manual for direction on accessing the device level configuration menu under UEFI HII Figure 7 1 Dell System Setup NIC Configuration NOTE Boot from SAN boot is supported only in UEFI and not in legacy BIOS ...

Page 92: ...ssociate an iSCSI initiator with the iSCSI target Record the following information iSCSI target name TCP port number iSCSI Logical Unit Number LUN initiator iSCSI qualified name IQN CHAP authentication details 5 After configuring the iSCSI target obtain the following Target IQN Target IP address Target TCP port number Target LUN Initiator IQN CHAP ID and secret Configuring iSCSI Boot Parameters Co...

Page 93: ...iSCSI target If CHAP Authentication is enabled configure the CHAP ID and CHAP Secret on the iSCSI Initiator Parameters Configuration page IP Version This option is specific to IPv6 Toggles between IPv4 and IPv6 All IP settings are lost if you switch from one protocol version to another DHCP Request Timeout Allows you to specify a maximum wait time in seconds for a DHCP request and response to comp...

Page 94: ...EFI Boot Mode Configuration To configure the boot mode 1 Restart the system 2 Access the System Utilities menu Figure 7 2 Dell System Setup Boot Settings NOTE SAN boot is supported in UEFI environment only Make sure the system boot option is UEFI and not legacy ...

Page 95: ...ot 76 AH0054602 00 A 3 In System Setup Device Settings select the QLogic device Figure 7 3 Refer to the OEM user guide on accessing the PCI device configuration menu Figure 7 3 Dell System Setup Device Settings Configuration Utility ...

Page 96: ... the Main Configuration Page select NIC Configuration Figure 7 4 and then press ENTER Figure 7 4 Selecting NIC Configuration 5 On the NIC Configuration page Figure 7 5 select Boot Mode and press ENTER to select one of the following iSCSI boot modes iSCSI SW ...

Page 97: ...ot iSCSI boot configuration options include Static iSCSI Boot Configuration Dynamic iSCSI Boot Configuration Enabling CHAP Authentication NOTE The iSCSI HW option is not listed if the iSCSI Offload feature is disabled at port level If the preferred boot mode is iSCSI HW make sure the iSCSI offload feature is enabled Not all adapter versions support iSCSI offload and iSCSI HW offloaded boot Additio...

Page 98: ...s System s initiator IQN Target parameters obtained in Configuring the iSCSI Target on page 73 For information on configuration options see Table 7 1 on page 74 To configure the iSCSI boot parameters using static configuration 1 In the Device HII Main Configuration Page select iSCSI Configuration Figure 7 6 and then press ENTER Figure 7 6 Dell System Setup iSCSI Configuration ...

Page 99: ...ure 7 8 press the UP ARROW and DOWN ARROW keys to select a parameter and then press the ENTER key to select or input the following values TCP IP Parameters via DHCP Disabled iSCSI Parameters via DHCP Disabled CHAP Authentication As required IP Version As required IPv4 or IPv6 CHAP Mutual Authentication As required DHCP Vendor ID Not applicable for static configuration HBA Boot Mode Enabled Virtual...

Page 100: ...4602 00 A Figure 7 8 Dell System Setup iSCSI General Parameters 4 Return to the iSCSI Configuration page and then press the ESC key 5 Select iSCSI Initiator Parameters Figure 7 9 and then press ENTER Figure 7 9 Dell System Setup iSCSI Initiator Parameters ...

Page 101: ... Initiator Configurations page select the following parameters and then type a value for each IPv4 Address Subnet Mask IPv4 Default Gateway IPv4 Primary DNS IPv4 Secondary DNS iSCSI Name Corresponds to the iSCSI initiator name to be used by the client system CHAP ID CHAP Secret ...

Page 102: ...ge and then press ESC NOTE Note the following for the preceding items with asterisks The label will change to IPv6 or IPv4 default based on IP Version set on the iSCSI General Parameters page Figure 7 10 Carefully enter the IP address There is a no error checking performed against the IP address to check for duplicates incorrect segment or network assignment ...

Page 103: ...SCSI First Target Parameters page set the Connect option to Enabled to the iSCSI target 10 Type values for the following parameters for the iSCSI target and then press ENTER IPv4 Address TCP Port Boot LUN iSCSI Name CHAP ID CHAP Secret NOTE For the preceding parameters with an asterisk the label will change to IPv6 or IPv4 default based on IP version set on the iSCSI General Parameters page as sho...

Page 104: ...eters 11 Return to the iSCSI Boot Configuration page and then press ESC 12 If you want configure a second iSCSI target device select iSCSI Second Target Parameters Figure 7 13 and enter the parameter values as you did in Step 10 Otherwise proceed to Step 13 Figure 7 13 Dell System Setup iSCSI Second Target Parameters ...

Page 105: ...tem to apply the changes to the adapter s running configuration Dynamic iSCSI Boot Configuration In a dynamic configuration ensure that the system s IP address and target or initiator information are provided by a DHCP server see IPv4 and IPv6 configurations in Configuring the DHCP Server to Support iSCSI Boot on page 89 Any settings on the following parameters are ignored and do not need to be cl...

Page 106: ... in Figure 7 15 TCP IP Parameters via DHCP Enabled iSCSI Parameters via DHCP Enabled CHAP Authentication As required IP Version As required IPv4 or IPv6 CHAP Mutual Authentication As required DHCP Vendor ID As required HBA Boot Mode Disabled Virtual LAN ID As required Virtual LAN Boot Mode Enabled NOTE When using a DHCP server the DNS server entries are overwritten by the values provided by the DH...

Page 107: ...n the Initiator Parameters window type values for the following CHAP ID up to 255 characters CHAP Secret if authentication is required must be 12 to 16 characters in length 4 Press ESC to return to the iSCSI Boot configuration page 5 On the iSCSI Boot configuration page select iSCSI First Target Parameters 6 In the iSCSI First Target Parameters window type values used when configuring the iSCSI ta...

Page 108: ... options that provide configuration information to the DHCP client For iSCSI boot QLogic adapters support the following DHCP configurations DHCP Option 17 Root Path DHCP Option 43 Vendor specific Information DHCP Option 17 Root Path Option 17 is used to pass the iSCSI target information to the iSCSI client The format of the root path as defined in IETC RFC 4173 is iscsi servername protocol port LU...

Page 109: ...ons QLogic recommends that customers use DHCP Option 43 when performing dynamic iSCSI boot configuration LUN The logical unit number to use on the iSCSI target The value of the LUN must be represented in hexadecimal format A LUN with an ID OF 64 would have to be configured as 40 within the option 17 parameter on the DHCP server targetname The target name in either IQN or EUI format Refer to RFC 37...

Page 110: ...alue is QLGC ISAN as shown in the General Parameters of the iSCSI Boot Configuration menu The content of Option 16 should be 2 byte length DHCP Vendor ID DHCPv6 Option 17 Vendor Specific Information DHCPv6 Option 17 vendor specific information provides more configuration options to the iSCSI client In this configuration three additional sub options are provided that assign the initiator IQN to the...

Page 111: ... the case make the iSCSI interface on the adapter a member of that VLAN To configure VLAN for iSCSI boot 1 Go to the iSCSI Configuration Page for the port 2 Select iSCSI General Parameters Table 7 4 DHCP Option 17 Sub option Definitions Sub option Definition 201 First iSCSI target information in the standard root path format iscsi servername protocol port LUN targetname 202 Second iSCSI target inf...

Page 112: ...imize server processor use This section covers how to configure the Windows iSCSI offload feature for the QLogic FastLinQ 41xxx Series Adapters With the proper iSCSI offload licensing you can configure your iSCSI capable FastLinQ 41xxx Series Adapter to offload iSCSI processing from the host processor The following sections describe how to enable the system to take advantage of QLogic s iSCSI offl...

Page 113: ...ft iSCSI initiator applet At the first launch the system prompts for an automatic service start Confirm the selection for the applet to launch Configuring Microsoft Initiator to Use QLogic s iSCSI Offload After the IP address is configured for the iSCSI adapter you must use Microsoft Initiator to configure and add a connection to the iSCSI target using the QLogic iSCSI adapter For more details on ...

Page 114: ...1 Open Microsoft Initiator 2 To configure the initiator IQN name according to your setup follow these steps a On the iSCSI Initiator Properties click the Configuration tab b On the Configuration page Figure 7 17 click Change to modify the initiator name Figure 7 17 iSCSI Initiator Properties Configuration Page ...

Page 115: ...Windows Server 96 AH0054602 00 A c In the iSCSI Initiator Name dialog box type the new initiator IQN name and then click OK Figure 7 18 Figure 7 18 iSCSI Initiator Node Name Change 3 On the iSCSI Initiator Properties click the Discovery tab ...

Page 116: ...7 iSCSI Configuration iSCSI Offload in Windows Server 97 AH0054602 00 A 4 On the Discovery page Figure 7 19 under Target portals click Discover Portal Figure 7 19 iSCSI Initiator Discover Target Portal ...

Page 117: ... the IP address or DNS name box type the IP address of the target b Click Advanced Figure 7 20 Target Portal IP Address 6 In the Advanced Settings dialog box Figure 7 21 complete the following under Connect using a For Local adapter select the QLogic name or model Adapter b For Initiator IP select the adapter IP address c Click OK ...

Page 118: ...7 iSCSI Configuration iSCSI Offload in Windows Server 99 AH0054602 00 A Figure 7 21 Selecting the Initiator IP Address 7 On the iSCSI Initiator Properties Discovery page click OK ...

Page 119: ...7 iSCSI Configuration iSCSI Offload in Windows Server 100 AH0054602 00 A 8 Click the Targets tab and then on the Targets page Figure 7 22 click Connect Figure 7 22 Connecting to the iSCSI Target ...

Page 120: ...ns page in QConvergeConsole GUI Question What tools should be used to create the connection to the target Answer Use Microsoft iSCSI Software Initiator version 2 08 or later Question How do I know that the connection is offloaded Answer Use Microsoft iSCSI Software Initiator From a command line type oiscsicli sessionlist From Initiator Name an iSCSI offloaded connection will display an entry begin...

Page 121: ...d the latest QLogic iSCSI boot images into the NVRAM of the adapter 4 Configure the iSCSI target to allow a connection from the remote device Ensure that the target has sufficient disk space to hold the new OS installation 5 Configure the UEFI HII to set the iSCSI boot type offload or non offload correct initiator and target parameters for iSCSI boot 6 Save the settings and reboot the system The r...

Page 122: ...csi_boot_sysfs ko is the Linux iSCSI sysfs interface that provides helpers to export iSCSI boot information uio ko is the Linux Userspace I O interface used for light L2 memory mapping for iscsiuio These modules must be loaded before qedi can be functional Otherwise you might encounter an unresolved symbol error If the qedi module is installed in the distribution update path the requisite will be ...

Page 123: ...es were detected correctly To verify iSCSI devices in Linux 1 To verify that the qedi and associated kernel modules were loaded issue the following command lsmod grep qedi qedi 114578 2 qed 697989 1 qedi uio 19259 4 cnic qedi libiscsi 57233 2 qedi bnx2i scsi_transport_iscsi 99909 5 qedi bnx2i libiscsi iscsi_boot_sysfs 16000 1 qedi 2 To verify that the iSCSI devices were detected properly issue the...

Page 124: ...aded loaded usr lib systemd system iscsiuio service disabled vendor preset disabled Active active running since Fri 2017 01 27 16 33 58 IST 6 days ago Docs man iscsiuio 8 Process 3745 ExecStart usr sbin iscsiuio code exited status 0 SUCCESS Main PID 3747 iscsiuio CGroup system slice iscsiuio service 3747 usr sbin iscsiuio Jan 27 16 33 58 localhost localdomain systemd 1 Starting iSCSI UserSpace I O...

Page 125: ...he iscsiadm command iscsiadm m session P3 Attached SCSI devices Host Number 59 State running scsi59 Channel 00 Id 0 Lun 0 Attached scsi disk sdb State running scsi59 Channel 00 Id 0 Lun 1 Attached scsi disk sdc State running scsi59 Channel 00 Id 0 Lun 2 Attached scsi disk sdd State running scsi59 Channel 00 Id 0 Lun 3 Attached scsi disk sde State running scsi59 Channel 00 Id 0 Lun 4 Attached scsi ...

Page 126: ...following boot parameter with the DUD option For RHEL 7 x and SLES 12 0 rd iscsi ibft No separate options are required for older distributions of RHEL and SLES For FastLinQ DUD package for example on RHEL 7 fastlinq 8 18 10 0 dd rhel7u3 3 10 0_514 el7 x86_64 iso 4 Install the OS on the target LUN To migrate from a non offload interface to an offload interface 1 Upgrade qedi transport supported Ope...

Page 127: ...bft if present b Remove any ip value boot option if present c For SLES 12 x and RHEL 7 x replace iscsi_firmware with rd iscsi firmware d If the iscsi_firmware or rd iscsi firmare boot option is not present complete one of the following For RHEL 7 x and SLES 12 x add rd scsi firmware For earlier versions of RHEL and SLES add iscsi_firmware 5 Create a backup of the original grub cfg file which is in...

Page 128: ...w grub cfg file 9 Create a new initramfs image by issuing the following command dracut force 10 On the adapter s preboot iSCSI Boot Configuration Menu change the value of the iSCSI offload a On the iSCSI Boot Configuration Menu set iSCSI Offload to Enable b Set HBA Mode to Enable NOTE The OS can now boot through the offload interface ...

Page 129: ...Considerations FCoE Boot from SAN This section describes the installation and boot procedures for the Windows Linux and ESXi operating systems including Preparing System BIOS for FCoE Build and Boot Windows FCoE Boot from SAN NOTE Some FCoE features may not be fully enabled in the current release For details refer to Appendix D Feature Constraints NOTE FCoE Boot from SAN is not supported on ESXi 5...

Page 130: ...ode only Set the platform in boot mode protocol using the system BIOS configuration to UEFI Configuring Adapter UEFI Boot Mode To configure the boot mode to FCOE 1 Restart the system 2 Press the OEM hot key to enter System Setup Figure 8 1 This is also known as UEFI HII Figure 8 1 Dell System Setup Selecting Device Settings NOTE FCoE BFS is not supported in legacy BIOS mode NOTE SAN boot is suppor...

Page 131: ...8 FCoE Configuration FCoE Boot from SAN 112 AH0054602 00 A 3 On the Device Settings page select the QLogic device Figure 8 2 Figure 8 2 Dell System Setup Device Settings Port Selection ...

Page 132: ...2 00 A 4 On the Main Configuration Page select NIC Configuration Figure 8 3 and then press ENTER Figure 8 3 Dell System Setup NIC Configuration 5 On the NIC Configuration page select Boot Mode and then press ENTER to select FCoE as a preferred boot mode ...

Page 133: ...ameters and then press ENTER 3 On the FCoE General Parameters page Figure 8 5 press the UP ARROW and DOWN ARROW keys to select a parameter and then press ENTER to select and input the following values Fabric Discovery Retry Count Default value or as required NOTE FCoE is not listed as a boot option if the FCoE Mode feature is disabled at the port level If the Boot Mode preferred is FCoE make sure ...

Page 134: ... Configuration page 5 Press ESC and then select FCoE Target Parameters 6 Press ENTER 7 In the FCoE Target Parameters Menu enable Connect to the preferred FCoE target 8 Type values for the following parameters Figure 8 6 for the iSCSI target and then press ENTER World Wide Port Name Target n Boot LUN n Where the value of n is between 1 and 8 enabling you to configure 8 FCoE targets ...

Page 135: ...Windows Server 2012 2012 R2 and 2016 FCoE Boot Installation Configuring FCoE FCoE Crash Dump Windows Server 2012 2012 R2 and 2016 FCoE Boot Installation For Windows Server 2012 2012R2 2016 boot from SAN installation QLogic requires the use of a slipstream DVD or ISO image with the latest QLogic drivers injected See Injecting Slipstreaming Adapter Drivers into Windows Image Files on page 118 ...

Page 136: ...he DVD ROM device 7 Boot from DVD and begin installation 8 Follow the on screen instructions 9 On the screen that shows the list of disks available for the installation the FCoE target disk should be visible This target is a disk connected through the FCoE boot protocol located in the remote FCoE target 10 To proceed with Windows Server 2012 2012R2 2016 installation select Next and follow the on s...

Page 137: ...er to which to extract the driver package For example type c temp d Follow the driver installer instructions to install the drivers in the specified folder In this example the driver files are installed here c temp Program File 64 QLogic Corporation QDrivers 3 Download the Windows Assessment and Deployment Kit ADK version 10 from Microsoft https developer microsoft com en us windows hardware windo...

Page 138: ... FIP fabric login FLOGI libfc ko is the Linux FC kernel library needed for several functions including Name server login and registration rport session management scsi_transport_fc ko is the Linux FC SCSI transport library used for remote port and SCSI target management These modules must be loaded before qedf can be functional otherwise errors such as unresolved symbol can result If the qedf modu...

Page 139: ...reated by fipvlan for fcoemon qedf does not sit on top of the net_device qedf is not dependent on network drivers such as bnx2x and cnic qedf will automatically initiate FCoE discovery on link up because it is not dependent on fipvlan or fcoemon for FCoE interface creation Configuring qedf ko No explicit configuration is required for qedf ko The driver automatically binds to the exposed FCoE funct...

Page 140: ...df scsi_dh_alua scsi_dh_rdac dm_multipath scsi_transport_fc scsi_transport_sas libfc scsi_transport_iscsi scsi_dh_emc libata sd_mod sr_mod 2 Check dmesg to verify that the FCoE devices were detected properly In this example the two detected FCoE CNA devices are SCSI host numbers 4 and 5 dmesg grep qedf 235 321185 0000 00 00 0 qedf_init 3728 QLogic FCoE Offload Driver v8 18 8 0 235 322253 0000 21 0...

Page 141: ...sk SANBlaze VLUN P2T1L7 V7 3 fc sdj 5 0 0 8 disk SANBlaze VLUN P2T1L8 V7 3 fc sdk 5 0 0 9 disk SANBlaze VLUN P2T1L9 V7 3 fc Configuration information for the host is located in sys class fc_host hostX where X is the number of the SCSI host In the preceding example X could 4 or 5 The hostX file contains attributes for the FCoE function such as worldwide port name and fabric ID Boot from SAN Conside...

Page 142: ...ection provides procedures for configuring iWARP through HII enabling iWARP and verifying RDMA and verifying iWARP traffic To configure iWARP through HII 1 Access the Dell System Setup and then click Device Settings 2 On the Device Settings page select a port for the 25G 41xxx Series Adapter 3 On the Main Configuration Page for the selected adapter click NIC Configuration 4 On the NIC Configuratio...

Page 143: ...Success Saving Changes message box click OK 8 Repeat Step 2 through Step 7 to configure the NIC and DCB for the second port 9 To complete adapter preparation of both ports a On the Device Settings page click Finish b On the main menu click Finish c Exit to reboot the system To enable iWARP and verify RDMA 1 To enable iWARP on the adapter a Open the Device Manager b Open the QLogic FastLinQ Adapter...

Page 144: ...dma command Figure 9 2 shows the Get NetAdapterRdma command output listing supported adapters Figure 9 2 PowerShell Command Get NetAdapterRdma 3 To verify that RDMA is enabled in the OS launch PowerShell and then issue the Get NetOffloadGlobalSetting command Figure 9 3 shows the Get NetOffloadGlobalSetting command output showing NetworkDirect as enabled Figure 9 3 PowerShell Command Get NetOffload...

Page 145: ...re 9 4 Perfmon Add Counters If iWARP traffic is running counters appear as shown in the Figure 9 5 example Figure 9 5 Perfmon Verifying iWARP Traffic 4 To verify the SMB connection a At a command prompt issue the net use command as follows C Users Administrator net use New connections will be remembered ...

Page 146: ...68 11 20 15903 192 168 11 10 445 0 Kernel 56 Connection 192 168 11 20 16159 192 168 11 10 445 0 Kernel 56 Connection 192 168 11 20 15903 192 168 11 10 445 0 Kernel 60 Listener fe80 e11d 9ab5 a47d 4f0a 56 445 NA 0 Kernel 60 Listener 192 168 11 20 445 NA 0 Kernel 60 Listener fe80 71ea bdd2 ae41 b95f 60 445 NA 0 Kernel 60 Listener 192 168 11 20 16159 192 168 11 10 445 0 Configuring iWARP on Linux QLo...

Page 147: ... RPM 4 When driver installation is complete load NIC drivers into iWARP mode 5 Compile and install NIC drivers RDMA drivers and RDMA user libraries as follows cd fastlinq x x x make install make libqedr_install 6 Load the NIC and RDMA drivers into iWARP mode by issuing the following command modprobe qedr Detecting the Device To detect the device 1 To verify whether iWARP devices are detected view ...

Page 148: ...pplications Linux supported RDMA applications for iWARP include the following ibv_devinfo ib_devices ib_send_bw lat ib_write_bw lat ib_read_bw lat ib_atomic_bw lat For iWARP all applications must use the RDMA communication manager rdma_cm using the R option rdma_server rdma_client rdma_xserver rdma_xclient rping NFS over RDMA NFSoRDMA iSER Running Perftest for iWARP All perftest tools are supporte...

Page 149: ...00 00 00 00 00 00 remote address LID 0000 QPN 0x0098 PSN 0x46fffc GID 00 14 30 196 195 62 00 00 00 00 00 00 00 00 00 00 bytes iterations BW peak MB sec BW average MB sec MsgRate Mpps 65536 1000 2250 38 2250 36 0 036006 Using iSER with iWARP Configure the iSER initiator and target similar to RoCE to work with iWARP You can use different methods to create a Linux IO Target LIO one is listed in this ...

Page 150: ...kstores ramdisk Ramdisk1 1 iscsi iqn 2017 04 com org iserport1 target1 tpg1 portals create 192 168 21 4 ip_port 3261 iscsi iqn 2017 04 com org iserport1 target1 tpg1 portals 192 168 21 4 3261 enable_iser boolean true iscsi iqn 2017 04 com org iserport1 target1 tpg1 set attribute authentication 0 demo_mode_write_protect 0 generate_node_acls 1 cache_dynamic_acls 1 saveconfig Figure 9 6 shows the tar...

Page 151: ...L VOLUME 3 56 dev sda 6 0 0 0 cd dvd hp DVD ROM DUD0N UMD0 dev sr0 7 0 0 0 disk LIO ORG Ramdisk1 1 4 0 dev sdb Configuring NFS RDMA NFS RDMA for iWARP includes both server and client configuration steps To configure the NFS server 1 In the etc exports file for the directories that you must export using NFS RDMA on the server make the following entry tmp nfs server fsid 0 async insecure no_root_squ...

Page 152: ... the NFS file system as appropriate for your version For NFS Version 3 mount o rdma port 20049 192 168 2 4 tmp nfs server tmp nfs client For NFS Version 4 mount t nfs4 o rdma port 20049 192 168 2 4 tmp nfs client 3 Verify that the file system is mounted by issuing the mount command Ensure that the RDMA port and file system versions are correct mount grep rdma ...

Page 153: ...ance interoperability and manageability This chapter provides instructions for Configuring SR IOV on Windows Configuring SR IOV on Linux Configuring SR IOV on VMware Configuring SR IOV on Windows To configure SR IOV on Windows 1 Access the Dell System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices NOTE Some SR IOV features may not be fully ena...

Page 154: ...re 10 1 Dell System Setup for SR IOV Integrated Devices 4 On the Main Configuration Page for the selected adapter click Device Level Configuration 5 On the Main Configuration Page Device Level Configuration Figure 10 2 a Set the Virtualization Mode to SR IOV b Click Back Figure 10 2 Dell System Setup for SR IOV Device Level Configuration 6 On the Main Configuration Page click Finish ...

Page 155: ...r b Open the miniport adapter properties and then click the Advanced tab c On the Advanced properties page Figure 10 3 under Property select SR IOV and then set the value to Enabled d Click OK Figure 10 3 Adapter Properties Advanced Enabling SR IOV 10 To create a Virtual Machine Switch with SR IOV Figure 10 4 a Launch the Hyper V Manager b Select Virtual Switch Manager c In the Name box type a nam...

Page 156: ...Apply Figure 10 4 Virtual Switch Manager Enabling SR IOV f The Apply Networking Changes message box advises you that Pending changes may disrupt network connectivity To save your changes and continue click Yes 11 To get the virtual machine switch capability issue the following PowerShell command PS C Users Administrator Get VMSwitch Name SR IOV_vSwitch fl ...

Page 157: ...d will include the following SR IOV capabilities IovVirtualFunctionCount 96 IovVirtualFunctionsInUse 1 12 To create a virtual machine VM and export the virtual function VF in the VM a Create a virtual machine b Add the VMNetworkadapter to the virtual machine c Assign a virtual switch to the VMNetworkadapter ...

Page 158: ... 00 A d In the Settings for VM VM_Name dialog box Figure 10 5 Hardware Acceleration page under Single root I O virtualization select the Enable SR IOV check box and then click OK Figure 10 5 Settings for VM Enabling SR IOV 13 Install the QLogic drivers for VF in the VM ...

Page 159: ...ing PowerShell command PS C Users Administrator Get NetadapterSriovVf Figure 10 7 shows example output Figure 10 7 PowerShell Command Get NetadapterSriovVf Configuring SR IOV on Linux To configure SR IOV on Linux 1 Access the Dell System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices 3 On the System Integrated Devices page see Figure 10 1 on p...

Page 160: ...ck Processor Settings 5 On the Processor Settings Figure 10 8 page a Set the Virtualization Technology option to Enabled b Click Back Figure 10 8 Dell System Setup Processor Settings for SR IOV 6 On the System Setup page select Device Settings 7 On the Device Settings page select Port 1 for the QLogic adapter ...

Page 161: ...02 00 A 8 On the Device Level Configuration page Figure 10 9 a Set the Virtualization Mode to SR IOV b Click Back Figure 10 9 Dell System Setup for SR IOV Integrated Devices 9 On the Main Configuration Page click Finish save your settings and then reboot the system ...

Page 162: ...10 Figure 10 10 Editing the grub conf File for SR IOV b Save the grub conf file and then reboot the system c To verify that the changes are in effect issue the following command dmesg grep I iommu A successful input output memory management unit IOMMU command output should show for example Intel IOMMU enabled d To view VF details number of VFs and total VFs issue the find sys grep I sriov command ...

Page 163: ...sys devices pci0000 00 0000 00 02 0 0000 04 00 0 sriov_numvfs b Review the command output Figure 10 11 to confirm that actual VFs were created Figure 10 11 Command Output for sriov_numvfs 12 To view a list of all PF and VF interfaces issue the following command Ip link show ifconfig a will Figure 10 12 shows example output Figure 10 12 Command Output for ip link show Command ...

Page 164: ...address b Ensure that the VF interface is up and running with the assigned MAC address 14 Power off the VM and attach the VF Some OSs support hot plugging of VFs to the VM a In the Virtual Machine dialog box Figure 10 13 click Add Hardware Figure 10 13 RHEL68 Virtual Machine b In the left pane of the Add New Virtual Hardware dialog box Figure 10 14 click PCI Host Device c In the right pane select ...

Page 165: ...VMware 1 Access the Dell System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices 3 On the Integrated Devices page see Figure 10 1 on page 135 a Set the SR IOV Global Enable option to Enabled b Click Back 4 On the Dell System Setup window click Device Settings 5 On the Device Settings page select a port for the 25G 41xxx Series Adapter 6 On the D...

Page 166: ... Corp QLogic FastLinQ QL41xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_0 0000 05 02 1 Network controller QLogic Corp QLogic FastLinQ QL41xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_1 0000 05 02 2 Network controller QLogic Corp QLogic FastLinQ QL41xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_2 0000 05 02 3 Network controller QLogic Corp QLogic FastLinQ QL41xxx Series 10 ...

Page 167: ...0591 1 true 005 02 1 60591 2 false 005 02 2 3 false 005 02 3 4 false 005 02 4 5 false 005 02 5 6 false 005 02 6 7 false 005 02 7 8 false 005 03 0 9 false 005 03 1 10 false 005 03 2 11 false 005 03 3 12 false 005 03 4 13 false 005 03 5 14 false 005 03 6 15 false 005 03 7 14 Attach VFs to the VM as follows a Power off the VM and attach the VF Some OSs support hot plugging of VFs to the VM b Add a ho...

Page 168: ...figuration changes and close this dialog box click OK Figure 10 15 VMware Host Edit Settings 16 Power on the VM and then issue the ifconfig a command to verify that the added network interface is listed 17 If no inbox driver is available install the driver 18 As needed add more VFs in the VM ...

Page 169: ...system While running I O performing interface down up tests or performing cable pull tests can cause driver or iSER module errors that may crash the system If this happens reboot the system Configuring iSER for RHEL To configure iSER for RHEL 1 Install inbox OFED as described in RoCE Configuration for RHEL on page 66 Out of box OFEDs are not supported for iSER because the ib_isert module is not av...

Page 170: ...p qed and lsmod grep iser commands 6 Verify that there are separate hca_id instances by issuing the ibv_devinfo command as shown in Step 6 on page 68 7 Check the RDMA connection on the initiator device and the target device a On the initiator device issue the following command rping s C 10 v b On the target device issue the following command rping c a 192 168 100 99 C 10 v Figure 11 1 shows an exa...

Page 171: ...l Instances 9 Install Linux iSCSI Initiator Utilities using the yum install iscsi initiator utils commands a To discover the iSER target issue the iscsiadm command For example iscsiadm m discovery t st p 192 168 100 99 3260 b To change the transport mode to iSER issue the iscsiadm command For example iscsiadm m node T iqn 2015 06 test target1 o update n iface transport_name v iser c To connect to ...

Page 172: ...e Iface Transport is iser in the target connection as shown Figure 11 3 Issue the iscsiadm command for example iscsiadm m session P2 Figure 11 3 Iface Transport Confirmed e To check for a new iSCSI device as shown Figure 11 4 issue the lsscsi command Figure 11 4 Checking for New iSCSI Device ...

Page 173: ...g 2 0 1 4 10 noarch rpm python netifaces 0 8 6 55 x86_64 rpm python rtslib 2 2 6 6 noarch rpm python urwid 1 1 1 6 144 x86_64 rpm targetcli 2 1 3 8 x86_64 rpm 2 Before starting the targetcli load all RoCE device drivers and iSER modules as follows modprobe qed modprobe qede modprobe qedr modprobe ib_iser Initiator modprobe ib_isert Target 3 Before configuring iSER targets configure NIC interfaces ...

Page 174: ...do f CPUFREQ continue echo n performance CPUFREQ done Verify that all CPU cores are set to maximum performance mode by issuing the following command cat sys devices system cpu cpu cpufreq scaling_governor Configuring Kernel sysctl Settings Set the kernel sysctl settings as follows sysctl w net ipv4 tcp_mem 4194304 4194304 4194304 sysctl w net ipv4 tcp_wmem 4096 65536 4194304 sysctl w net ipv4 tcp_...

Page 175: ...e systemctl stop irqbalance cat proc interrupts grep qedr Shows IRQ assigned to each port queue echo 1 proc irq XX smp_affinity_list echo 2 proc irq YY smp_affinity_list echo 4 proc irq ZZ smp_affinity_list echo 8 proc irq XYZ smp_affinity_list Configuring Block Device Staging Set the block device staging settings for each iSCSI device or target as follows echo noop sys block sdd queue scheduler e...

Page 176: ... V NDKPI Mode 2 In Windows Server 2016 Hyper V host virtual network adapters host virtual NICs support RDMA RoCE configuration procedures in this section include Creating a Hyper V Virtual Switch with an RDMA Virtual NIC Adding a VLAN ID to Host Virtual NIC Verifying If RoCE is Enabled Adding Host Virtual NICs Virtual Ports Mapping the SMB Drive and Running RoCE Traffic NOTE DCBX is required for R...

Page 177: ...r V Manager 2 Click Virtual Switch Manager see Figure 12 1 Figure 12 1 Enabling RDMA in Host Virtual NIC 3 Create a virtual switch 4 Select the Allow management operating system to share this network adapter check box In Windows Server 2016 a new parameter Network Direct RDMA is added in the Host virtual NIC To enable RDMA in a host virtual NIC 1 Open the Hyper V Virtual Ethernet Adapter Propertie...

Page 178: ...e NetAdapterRdma vEthernet New Virtual Switch PS C Users Administrator Adding a VLAN ID to Host Virtual NIC To add VLAN ID to a host virtual NIC 1 To find the host virtual NIC name issue the following PowerShell command PS C Users Administrator Get VMNetworkAdapter ManagementOS Figure 12 3 shows the command output Figure 12 3 PowerShell Command Get VMNetworkAdapter 2 To set the VLAN ID to the host...

Page 179: ...ssue the following command Add VMNetworkAdapter SwitchName New Virtual Switch Name SMB ManagementOS 2 Enable RDMA on host virtual NICs as shown in To enable RDMA in a host virtual NIC on page 158 3 To assign a VLAN ID to the virtual port issue the following command Set VMNetworkAdapterVlan VMNetworkAdapterName SMB VlanId 5 Access ManagementOS NOTE Note the following about adding a VLAN ID to a hos...

Page 180: ... Running RoCE Traffic To map the SMB drive and run the RoCE traffic 1 Launch the Performance Monitor Perfmon 2 Complete the Add Counters dialog box Figure 12 5 as follows a Under Available counters select RDMA Activity b Under Instances of selected object select the adapter c Click Add Figure 12 5 Add Counters Dialog Box ...

Page 181: ...2016 Technical Preview SET integrates limited NIC Teaming functionality into the Hyper V Virtual Switch Use SET to group between one and eight physical Ethernet network adapters into one or more software based virtual network adapters These adapters provide fast performance and fault tolerance if a network adapter failure occurs To be placed on a team SET member network adapters must all be instal...

Page 182: ...rShell Command New VMSwitch Enabling RDMA on SET To enable RDMA on SET 1 To view the SET on the adapter issue the following PowerShell command PS C Users Administrator Get NetAdapter vEthernet SET Figure 12 8 shows command output Figure 12 8 PowerShell Command Get NetAdapter 2 To enable RDMA on SET issue the following PowerShell command PS C Users Administrator Enable NetAdapterRdma vEthernet SET ...

Page 183: ...and server To configure QoS by disabling DCBX 1 Disable DCBX on the adapter 2 Using HII set the RoCE Priority to 0 3 To install the DCB role in the host issue the following PowerShell command PS C Users Administrators Install WindowsFeature Data Center Bridging 4 To set the DCBX Willing mode to False issue the following PowerShell command PS C Users Administrators set NetQosDcbxSetting Willing 0 5...

Page 184: ...es Enable QoS 6 Assign the VLAN ID to the interface as follows a Open the miniport window and then click the Advanced tab b On the adapter s Advanced Properties page Figure 12 10 under Property select VLAN ID and then set the value c Click OK NOTE The preceding step is required for priority flow control PFC ...

Page 185: ... Advanced Properties Setting VLAN ID 7 To enable priority flow control for RoCE on a specific priority issue the following command PS C Users Administrators Enable NetQoSFlowControl Priority 4 NOTE If configuring RoCE over Hyper V do not assign a VLAN ID to the physical interface ...

Page 186: ...nd assign relevant priority to each type of traffic issue the following commands where Priority 4 is tagged for RoCE and Priority 0 is tagged for TCP PS C Users Administrators New NetQosPolicy SMB NetDirectPortMatchCondition 445 PriorityValue8021Action 4 PolicyStore ActiveStore PS C Users Administrators New NetQosPolicy TCP IPProtocolMatchCondition TCP PriorityValue8021Action 0 Policystore ActiveS...

Page 187: ...icClass Name Algorithm Bandwidth Priority PolicySet IfIndex IfAlias Default ETS 20 2 3 5 7 Global RDMA class ETS 50 4 Global TCP class ETS 30 0 Global 11 To see the network adapter QoS from the preceding configuration issue the following PowerShell command PS C Users Administrator Get NetAdapterQos Name SLOT 4 Port 1 Enabled True Capabilities Hardware Current MacSecBypass NotSupported NotSupported...

Page 188: ...tems in use The PFC ETS and traffic classes configuration must be the same on the switch and server To configure QoS by enabling DCBX 1 Enable DCBX IEEE CEE or Dynamic 2 Using HII set the RoCE Priority to 0 3 To install the DCB role in the host issue the following PowerShell command PS C Users Administrators Install WindowsFeature Data Center Bridging 4 To set the DCBX Willing mode to True issue t...

Page 189: ...Property select Quality of Service and then set the value to Enabled b Click OK Figure 12 11 Advanced Properties Enabling QoS 6 Assign the VLAN ID to the interface required for PFC as follows a Open the miniport window and then click the Advanced tab b On the adapter s Advanced Properties page Figure 12 12 under Property select VLAN ID and then set the value c Click OK ...

Page 190: ...mmand PS C Users Administrators Get NetAdapterQoS Name Ethernet 5 Enabled True Capabilities Hardware Current MacSecBypass NotSupported NotSupported DcbxSupport CEE CEE NumTCs Max ETS PFC 4 4 4 4 4 4 OperationalTrafficClasses TC TSA Bandwidth Priorities 0 ETS 5 0 3 5 7 1 ETS 95 4 OperationalFlowControl Priority 4 Enabled OperationalClassifications Protocol Port Type Priority ...

Page 191: ...ty Creating a Virtual Machine and Enabling VMMQ on VMNetworkadapters in the Virtual Machine Default and Maximum VMMQ Virtual NIC Enabling and Disabling VMMQ on a Management NIC Monitoring Traffic Statistics NOTE The preceding example is taken when the adapter port is connected to an Arista 7060X switch In this example the switch PFC is enabled on Priority 4 RoCE App TLVs are defined The two traffi...

Page 192: ...the value to Enabled 3 Click OK Figure 12 13 Advanced Properties Enabling Virtual Switch RSS Setting the VMMQ Max QPs Default and Non Default VPort To set the VMMQ maximum QPs default and non default VPort 1 Open the miniport window and click the Advanced tab 2 On the Advanced Properties page Figure 12 14 under Property select one of the following VMMQ Max QPs Default VPort VMMQ Max QPs Non Defaul...

Page 193: ...ch with or Without SRIOV To create a virtual machine switch with or without SRIOV 1 Launch the Hyper V Manager 2 Select Virtual Switch Manager see Figure 12 15 3 In the Name box type a name for the virtual switch 4 Under Connection type a Select External network b Select the Allow management operating system to share this network adapter check box ...

Page 194: ...K Figure 12 15 Virtual Switch Manager Enabling VMMQ on the Virtual Machine Switch To enable VMMQ on the virtual machine switch Issue the following PowerShell command PS C Users Administrators Set VMSwitch name q1 defaultqueuevmmqenabled true defaultqueuevmmqqueuepairs 4 ...

Page 195: ...tor Get VMSwitch Name ql fl Figure 12 16 shows example output Figure 12 16 PowerShell Command Get VMSwitch Creating a Virtual Machine and Enabling VMMQ on VMNetworkadapters in the Virtual Machine To create a virtual machine and enable VMMQ on VMNetworksadapters in the virtual machine 1 Create a virtual machine 2 Add the VMNetworkadapter to the virtual machine 3 Assign a virtual switch to the VMNet...

Page 196: ...ctivated Unknown 1 Ethernet 3 65 0 0 63 Activated Unknown 1 Ethernet 3 66 00 15 5D 36 0A 04 0 16 PF Activated Adaptive 4 Ethernet 3 67 00 15 5D 36 0A 05 1 0 PF Activated Adaptive 4 Ethernet 3 68 00 15 5D 36 0A 06 0 0 PF Activated Adaptive 4 Ethernet 3 69 00 15 5D 36 0A 07 0 8 PF Activated Adaptive 4 Ethernet 3 70 00 15 5D 36 0A 08 0 16 PF Activated Adaptive 4 Ethernet 3 71 00 15 5D 36 0A 09 1 0 PF...

Page 197: ...maximum default queue use the VMswitch capability Enabling and Disabling VMMQ on a Management NIC To enable or disable VMMW on a management NIC To enable VMMQ on a management NIC issue the following command PS C Users Administrator Set VMNetworkAdapter ManagementOS vmmqEnabled true The MOS VNIC will also have four VMMQs To disable VMMQ on a management NIC issue the following command PS C Users Adm...

Page 198: ...Offload Figure 12 17 Advanced Properties Enabling VXLAN 3 Set the Value to Enabled 4 Click OK Deploying a Software Defined Network To take advantage of VXLAN encapsulation task offload on virtual machines you must deploy a software defined network SDN that utilizes a Microsoft Network Controller For more details refer to the following Microsoft TechNet link on Software Defined Networking https tec...

Page 199: ...storage systems with local storage For more information refer to the following Microsoft TechnNet link https technet microsoft com en us windows server docs storage storage spaces storage spaces direct windows server 2016 Configuring the Hardware Figure 12 18 shows an example of hardware configuration Figure 12 18 Example Hardware Configuration NOTE The disks used in this example are 4 400G NVMe a...

Page 200: ...he Windows server roles Hyper V 3 Install the following features Failover Cluster Data center bridging DCB 4 Connect the nodes to domain and adding domain accounts Configuring the Network To deploy Storage Spaces Direct the Hyper V switch must be deployed with RDMA enabled host virtual NICs To configure the network on each server 1 Configure the physical network switch as follows a Connect all ada...

Page 201: ... switch embedded teaming issue the following command New VMSwitch Name SETswitch NetAdapterName port1 port2 port3 port4 EnableEmbeddedTeaming true c To add host virtual NICs to the virtual switch issue the following commands Add VMNetworkAdapter SwitchName SETswitch Name SMB_1 managementOS Add VMNetworkAdapter SwitchName SETswitch Name SMB_2 managementOS NOTE Network Quality of Service is used to ...

Page 202: ...le NetAdapter vEthernet SMB_1 Disable NetAdapter vEthernet SMB_2 Enable NetAdapter vEthernet SMB_2 g To enable RDMA on the host virtual NIC adapters issue the following command Enable NetAdapterRdma SMB1 SMB2 h To verify RDMA capabilities issue the following command Get SmbClientNetworkInterface where RdmaCapable EQ true Configuring Storage Spaces Direct Configuring Storage Spaces Direct in Window...

Page 203: ...em before enabling Storage Spaces Direct Otherwise they will not be included in the Storage Spaces Direct storage pool Step 3 Configuring a Cluster Witness QLogic recommends that you configure a witness for the cluster so that this four node system can withstand two nodes failing or being offline With these systems you can configure file share witness or cloud witness For more information go to ht...

Page 204: ...ue IsSystem ne true PartitionStyle eq RAW Group NoElement Property FriendlyName Sort Property PsComputerName Count Step 5 Enabling Storage Spaces Direct After creating the cluster use the Enable ClusterStorageSpacesDirect PowerShell cmdlet The cmdlet places the storage system into the Storage Spaces Direct mode and automatically does the following Creates a single large pool that has a name such a...

Page 205: ...r private clouds and data centers It is similar to Windows Server in Server Core mode but is significantly smaller has no local logon capability and supports only 64 bit applications tools and agents The Nano Server takes less disk space sets up faster and requires fewer updates and restarts than Windows Server When it does restart it restarts much faster Roles and Features Table 12 1 shows the ro...

Page 206: ...s Microsoft NanoServer DNS Package Desired State Configuration DSC Packages Microsoft NanoServer DSC Package Internet Information Server IIS Packages Microsoft NanoServer IIS Package Host Support for Windows Containers Containers System Center Virtual Machine Manager Agent Packages Microsoft Windows Server SCVMM Package Packages Microsoft Windows Server SCVMM Compute Package Note Use this package ...

Page 207: ...ServerImageGenerator script by issuing the following command Import Module NanoServerImageGenerator psm1 Verbose 7 To create a VHD that sets a computer name and includes the OEM drivers and Hyper V issue the following PowerShell command New NanoServerImage DeploymentType Host Edition Standard Datacenter MediaPath path to root of media BasePath Base TargetPath NanoServerPhysical NanoServer vhd Comp...

Page 208: ...disk INFO Creating single partition INFO Formatting windows volume INFO Windows path I has been assigned INFO System volume location I INFO Applying image to VHD This could take a while INFO Image was applied successfully INFO Making image bootable INFO Fixing the Device ID in the BCD store on VHD INFO Drive is bootable Cleaning up INFO Dismounting VHD INFO Closing Windows image INFO Done Done The...

Page 209: ...age 2 Go to the NanoServer folder from the downloaded file in Step 1 3 Copy the following files from the NanoServer folder to a folder on your hard drive NanoServerImageGenerator psm1 Convert WindowsImage ps1 4 Start Windows PowerShell as an administrator 5 Change directory to the folder where you pasted the files from Step 3 6 Import the NanoServerImageGenerator script by issuing the following co...

Page 210: ... cmdlet New NanoServerImage at command pipeline position 1 Supply values for the following parameters Windows R Image to Virtual Hard Disk Converter for Windows R 10 Copyright C Microsoft Corporation All rights reserved Version 10 0 14300 1000 amd64fre rs1_release_svc 160324 1723 INFO Looking for the requested Windows image in the WIM file INFO Image 1 selected ServerTuva INFO Creating sparse disk...

Page 211: ...Windows Management Instrumentation WMI Windows Remote Management and Emergency Management Services EMS This section describes how to access Nano Server using Windows PowerShell remoting Managing a Nano Server with Windows PowerShell Remoting To manage Nano Server with Windows PowerShell remoting 1 Add the IP address of the Nano Server to your management computer s list of trusted hosts 2 Add the a...

Page 212: ...rver go to https technet microsoft com en us library mt126167 aspx Managing QLogic Adapters on a Windows Nano Server To manage QLogic adapters in Nano Server environments refer to the Windows QConvergeConsole GUI and Windows QLogic Control Suite CLI management tools and associated documentation available from the QLogic Downloads and Documentation page http driverdownloads qlogic com RoCE Configur...

Page 213: ...ue the following PowerShell command 172 28 41 152 PS C Users Administrator Documents Get NetAdapterRdma Figure 12 20 shows example output Figure 12 20 PowerShell Command Get NetAdapterRdma 4 To assign an IP address and VLAN ID to all interfaces of the adapter issue the following PowerShell commands 172 28 41 152 PS C Set NetAdapterAdvancedProperty InterfaceAlias slot 1 port 1 RegistryKeyword vlani...

Page 214: ...2 22 PowerShell Command New SMBShare 6 To map the SMBShare as a network drive in the client machine issue the following PowerShell command PS C Windows system32 net use z 192 168 10 10 smbshare This command completed successfully 7 To perform read write on SMBShare and check RDMA statistics on the Nano Sever issue the following PowerShell command 172 28 41 152 PS C Get NetAdapterStatistics RdmaSta...

Page 215: ...12 Windows Server 2016 Deploying and Managing a Nano Server 196 AH0054602 00 A Figure 12 23 shows the command output Figure 12 23 PowerShell Command Get NetAdapterStatistics ...

Page 216: ...rify that the cable connections at the network adapter and the switch are attached properly Verify the adapter installation by reviewing Installing the Adapter on page 6 Ensure that the adapter is properly seated in the slot Check for specific hardware problems such as obvious damage to board components or the PCI edge connector Verify the configuration settings and change them if they are in conf...

Page 217: ...sure that the current drivers are loaded for your Windows Linux or VMware system Windows See the Device Manager to view vital information about the adapter link status and network connectivity Linux To verify that the qed ko driver is loaded properly issue the following command lsmod grep i module name If the driver is loaded the output of this command shows the size of the driver in bytes The fol...

Page 218: ...xx Ethernet Driver qede x x x x VMware To verify that the VMware ESXi drivers are loaded issue the following command esxcli software vib list Testing Network Connectivity This section provides procedures for testing network connectivity in Windows and Linux environments Testing Network Connectivity for Windows Test network connectivity using the ping command To determine if the network connection ...

Page 219: ...tualization with Hyper V Microsoft Virtualization is a hypervisor virtualization system for Windows Server 2012 R2 For more information on Hyper V go to https technet microsoft com en us library Dn282278 aspx Linux specific Issues Problem 41xxx Series Adapter devices with SFP Flow Control default to Off rather than Rx Tx Enable Solution The Flow Control default setting for revision 1 6 x and newer...

Page 220: ...cellaneous Issues Problem The 41xxx Series Adapter has shut down and an error message appears indicating that the fan on the adapter has failed Solution The 41xxx Series Adapter shut down to prevent permanent damage Contact QLogic Technical Support for assistance ...

Page 221: ...ors for the state of the adapter port link and activity Table A 1 Adapter Port Link and Activity LEDs Port LED LED Appearance Network State Link LED Off No link cable disconnected Continuously illuminated Link Activity LED Off No port activity Blinking Port activity ...

Page 222: ...ort a variety of cables and optical modules that comply with SFF8024 Specific form factor compliance is as follows SFPs SFF8472 for memory map SFF8419 or SFF8431 low speed signals and power Quad small form factor pluggable QSFPs SFF8636 for memory map SFF8679 or SFF8436 low speed signals and power Optical modules electrical input output active copper cables ACC and active optical cables AOC 10G SF...

Page 223: ...roduct release and is subject to change over time as new components enter the market or are discontinued Table B 1 Tested Cables and Optical Modules Speed or Form Factor Manufacturer Description and Part Number 10G DAC Cisco 10G Twin axial Cable 3 0M Dell 10G Twin axial Cable with Cisco SFP CN 0J564N 52204 51F ACU1 A00 3 0M 25G DAC Amphenol NDCCGF 00011 0m NDCCGF 0005 3 0m NDAQGF 0001 1m Splitter ...

Page 224: ...ssion and then log in to the switch as follows Login admin Password admin 3 Enable configuration of the switch port Dell enable Password xxxxxx Dell config 4 Identify the module and port to be configured The following example uses module 1 port 5 Dell conf stack unit 1 port 5 portmode Set portmode for a module Dell conf stack unit 1 port 5 portmode dual Enable dual mode quad Enable quad mode singl...

Page 225: ...ort 5 interface 1 and confirm the auto negotiation status Dell conf interface tw 1 5 1 Dell conf if tf 1 5 1 intf type cr4 autoneg Enable autoneg b Disable auto negotiation Dell conf if tf 1 5 1 no intf type cr4 autoneg c Verify that auto negotiation is disabled Dell conf if tf 1 5 1 do show run interface tw 1 5 1 interface twentyFiveGigE 1 5 1 no ip address mtu 9416 switchport flowcontrol rx on t...

Page 226: ...sonality that has been configured using HII or QLogic management tools is written in NVRAM configuration When this limitation is removed users can configure additional PFs on the same port for storage Concurrent RoCE and iWARP Is Not Supported on the Same Port RoCE and iWARP are not supported on the same port HII and QLogic management tools do not allow users to configure both concurrently NPAR Co...

Page 227: ...disabled However RDMANICModeOnPartition is currently set to disabled and cannot be enabled NIC and SAN Boot to Base Is Supported Only on Select PFs Ethernet and PXE boot are currently supported only on PF0 and PF1 In NPAR configuration other PFs do not support Ethernet and PXE boot When the Virtualization Mode is set to NPAR non offloaded FCoE boot is supported on Partition 2 PF2 and PF3 and iSCSI...

Page 228: ...rt on the adapter board advanced configuration and power interface See ACPI bandwidth A measure of the volume of data that can be transmitted at a specific transmission rate A 1Gbps or 2Gbps Fibre Channel port can transmit or receive at nominal rates of 1 or 2Gbps depending on the device to which it is connected This corre sponds to actual bandwidth values of 106MB and 212MB respectively BAR Base ...

Page 229: ...Channel a target device DHCP Dynamic host configuration protocol Enables computers on an IP network to extract their configuration from servers that have information about the computer only after it is requested driver The software that interfaces between the file system and a physical data storage device or network media dynamic host configuration protocol See DHCP eCore A layer between the OS an...

Page 230: ...le transfer protocol A standard network protocol used to transfer files from one host to another host over a TCP based network such as the Internet FTP is required for out of band firmware uploads that will complete faster than in band firmware uploads human interface infrastructure See HII HII Human interface infrastructure A specifi cation part of UEFI 2 1 for managing user input localized strin...

Page 231: ...ack to build a large up to 64KB TCP message before sending it to the adapter The adapter hardware segments the message into smaller data packets frames that can be sent over the wire up to 1 500 bytes for standard Ethernet frames and up to 9 000 bytes for jumbo Ethernet frames The segmentation process frees up the server CPU from having to segment large TCP messages into smaller packets that will ...

Page 232: ...at of the older peripheral compo nent interconnect PCI and PCI extended PCI X desktop and server slots QoS Quality of service Refers to the methods used to prevent bottlenecks and ensure business continuity when transmitting data over virtual ports by setting priorities and allocating bandwidth quality of service See QoS PF Physical function RDMA Remote direct memory access The ability for one nod...

Page 233: ... SCSI session Initiators request data from targets Targets are typically disk drives tape drives or other media devices Typically a SCSI peripheral device is the target but an adapter may in some cases be a target A target can contain many LUNs A target is a device that responds to a requested by an initiator the host system Peripherals are targets but for some commands for example a SCSI COPY com...

Page 234: ...initiative for remote direct memory access across Fibre Channel and other communication proto cols Used in clustering and messaging virtual interface See VI virtual logical area network See VLAN virtual machine See VM VLAN Virtual logical area network LAN A group of hosts with a common set of require ments that communicate as if they were attached to the same wire regardless of their physical loca...

Page 235: ... All other brand and product names are trademarks or registered trademarks of their respective owners This document is provided for informational purposes only and may contain errors Cavium reserves the right without notice to make changes to this document or in product design or specifications Cavium disclaims any warranty of any kind expressed or implied and does not guarantee that any results o...