Dell QLogic QL45212, User Manual

Page 1: ...BC0154503 00 B Third party information brought to you courtesy of Dell User s Guide 25Gb Intelligent Ethernet Adapter QL45212 ...

Page 2: ...lassification filters from 512 to 1 024 Removed the Microsoft Windows Server only qualification from the virtualization feature Features on page 1 In Table 2 1 updated the list of supported direct attach copper cables In Table 2 2 updated the supported Windows and Linux operating system versions added VMware System Requirements on page 5 Updated the section titled Installing the Add In NIC to Inst...

Page 3: ...r Preboot Adapter Configuration Preboot Adapter Configuration on page 36 Added the chapter Configuring RoCE Configuring RoCE on page 44 Added the chapter iSCSI Extensions for RDMA iSCSI Extensions for RDMA on page 62 Updated the firmware versions listed in the exam ple to be generic Linux on page 70 Updated the connectivity testing procedures for Windows and Linux to include information about how ...

Page 4: ...ication xiv EMI and EMC Requirements xiv KCC Class A xv Product Safety Compliance xv 1 Product Overview Functional Description 1 Features 1 Adaptive Interrupt Frequency 3 ASIC with Embedded RISC Processor 3 Adapter Specifications 3 Physical Characteristics 3 Standards Specifications 3 2 Installing the Hardware System Requirements 5 Safety Precautions 6 Preinstallation Checklist 6 Installing the Ad...

Page 5: ...Messages 17 Statistics 17 Windows Driver Software 17 Installing the Windows Drivers 17 Running the Dell Update Package in the GUI 17 Running the Dell Update Package from the Command Line 23 Installation Options 24 Examples 25 Removing the Windows Drivers 25 Managing Adapter Properties 26 Setting Power Management Options 26 VMware Driver Software 27 Installing VMware Drivers 27 VMware Driver Option...

Page 6: ...iguring In box OFED for SLES Linux 57 Installing Out of box OFED for SLES Linux 57 Verifying the RoCE Configuration on Linux 58 VLAN Interfaces and GID Index Values 61 7 iSCSI Extensions for RDMA Configuring iSER for RHEL 62 Configuring iSER for SLES12 Linux 66 Optimizing Linux Performance 66 Set CPUs to Maximum Performance Mode 67 Set Kernel sysctl Settings 67 IRQ Affinity Settings 67 Block Devic...

Page 7: ...User s Guide 25Gb Intelligent Ethernet Adapter QL45212 vii BC0154503 00 B A Adapter LEDS B Dell Z9100 Switch Configuration Glossary Index ...

Page 8: ...ustom Setup Window 21 3 6 InstallShield Wizard Ready to Install the Program Window 22 3 7 InstallShield Wizard Completed Window 22 3 8 Dell Update Package Window 23 3 9 Power Management 26 4 1 Dell Update Package Splash Screen 31 4 2 Continue Dell Update Package Installation 32 4 3 Loading New Firmware 33 4 4 Result of Installation 34 4 5 Finish Installation 34 4 6 Command Line Options 35 5 1 Main...

Page 9: ...rements 5 2 2 Host Operating System Requirements 5 3 1 QLogic QL45212 Linux Drivers 8 3 2 qede Driver Optional Parameters 16 3 3 Linux Driver Parameter Defaults 16 3 4 VMware Driver Optional Parameters 28 3 5 VMware Driver Parameter Defaults 30 5 1 Adapter Properties 38 5 2 Link Speed Options 41 A 1 Port LED Indications 74 ...

Page 10: ...oduct Overview provides a product functional description a list of features a list of supported operating systems and the adapter specifications Chapter 2 Installing the Hardware describes how to install the adapter including the list of system requirements and a preinstallation checklist Chapter 3 Installing Drivers describes the installation of the adapter drivers Chapter 4 Upgrading Firmware de...

Page 11: ...ury or death Text in blue font indicates a hyperlink jump to a figure table or section in this guide and links to Web sites are shown in underlined blue For example Table 9 2 lists problems related to the user interface and remote agent See Installation Checklist on page 6 For more information visit www qlogic com Text in bold font indicates user interface elements such as a menu items buttons che...

Page 12: ...se Agreements Refer to the QLogic Software End User License Agreement for a complete listing of all license agreements affecting this product Technical Support Dell provides several online and telephone based support and service options Availability varies by country and product and some services may not be available in your area To contact Dell for sales technical support or customer service issu...

Page 13: ...classe 1 Produkt der Laser Klasse 1 Luokan 1 Laserlaite Caution Class 1 laser radiation when open Do not view directly with optical instruments Attention Radiation laser de classe 1 Ne pas regarder directement avec des instruments optiques Vorsicht Laserstrahlung der Klasse 1 bei geöffneter Abdeckung Direktes Ansehen mit optischen Instrumenten vermeiden Varoitus Luokan 1 lasersäteilyä kun laite on...

Page 14: ...received including interference that may cause undesired operation ICES 003 Compliance Class A This Class A digital apparatus complies with Canadian ICES 003 Cet appareil numériqué de la classe A est conformé à la norme NMB 003 du Canada CE Mark 2004 108 EC EMC Directive Compliance EN55022 2010 Class A1 2007 CISPR22 2006 Class A EN55024 2010 EN61000 3 2 Harmonic Current Emission EN61000 3 3 Voltag...

Page 15: ...UL60950 1 2nd Edition 2007 03 3 27 UL CSA C22 2 60950 1 07 2nd Edition Use only with listed ITE or equivalent Complies with 21 CFR 1040 10 and 1040 11 2006 95 EC low voltage directive TUV EN60950 1 2006 A11 A1 A12 IEC60950 1 2nd Edition 2005 CB CB Certified to IEC 60950 1 2nd Edition A class equipment Business purpose info telecom munications equipment As this equipment has undergone EMC registrat...

Page 16: ...er includes a 25Gb Ethernet MAC with full duplex capability Features The QL45212 Adapter provides the following features NIC partitioning NPAR Remote direct memory access over converged Ethernet RoCE Single chip solution 25Gb MAC SerDes interface for direct attach copper DAC transceiver connection PCI Express Gen3 x8 Zero copy capable hardware Performance features TCP IP UDP checksum offloads TCP ...

Page 17: ...nced network features Jumbo frames up to 9600 bytes The OS and the link partner must support jumbo frames Virtual LANs VLANs Flow control IEEE Std 802 3x Logical Link Control IEEE Std 802 2 High speed on chip reduced instruction set computer RISC processor Integrated 96 KB frame buffer memory 1 024 classification filters Support for multicast addresses through 128 bit hashing hardware function Ser...

Page 18: ...RISC processor which provides the flexibility to add new features to the card and adapts it to future network requirements through software downloads The RISC processor also enables the adapter drivers to exploit the built in host offload functions on the adapter as host operating systems are enhanced to take advantage of these functions Adapter Specifications Physical Characteristics The QL45212 ...

Page 19: ...4 BC0154503 00 B 2 Installing the Hardware System Requirements Safety Precautions Preinstallation Checklist Installing the Adapter ...

Page 20: ...IA 32 or EMT64 processor that meets operating system requirements PCIe PCIe Gen2 x8 PCIe Gen3 x8 Full dual port 25Gb bandwidth is supported on PCIe Gen3 x8 or faster slots Memory 8GB RAM minimum Direct Attach Cables DAC Amphenol NDDCCGF 0001 Amphenol NDAQGF 0003 Amphenol NDCCGF 0005 Leoni ParaLink LA0SF064 SD R Table 2 2 Host Operating System Requirements Operating System Requirement Windows Serve...

Page 21: ...a damaged adapter WARNING The adapter is being installed in a system that operates with voltages that can be lethal Before you open the case of your system observe the following precautions to protect yourself and to prevent damage to the system components Remove any metallic objects or jewelry from your hands and wrists Make sure to use only insulated or nonconducting tools Verify that the system...

Page 22: ...eater width slot x8 in a x16 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 If you do not know how to identify a PCIe slot refer to your system documentation 3 Remove the blank cover plate from the slot that you selected 4 Align the adapter connector edge with the PCIe connector slot in the system 5 Applying even pressure at both corners of the card push the adapt...

Page 23: ...inux Driver Optional Parameters Linux Driver Parameter Defaults Linux Driver Messages Statistics The QL45212 Adapter Linux drivers and supporting documentation are available at dell support com Table 3 1 describes the QL45212 Adapter Linux drivers Table 3 1 QLogic QL45212 Linux Drivers Linux Driver Description qed The Linux core module manages all PCI device resources registers host interface queu...

Page 24: ...page 13 qede Linux Ethernet driver for the QL45212 Adapter This driver directly controls the hardware and is responsible for sending and receiving Ethernet pack ets on behalf of the Linux host networking stack This driver also receives and processes device interrupts on behalf of itself for L2 networking The qede driver requires Linux kernel version 2 6 32 or later Testing was concentrated on the ...

Page 25: ...nvironment To remove Linux drivers in a non RoCE environment 1 Unload and remove the Linux drivers If the Linux drivers were installed using an RPM package type the following commands rmmod qede rmmod qed depmod a rpm e kmod qlgc fastlinq version arch If the Linux drivers were installed using a TAR file type the following commands rmmod qede rmmod qed depmod a 2 Delete the qed ko qede ko and qedr ...

Page 26: ...obe r qed depmod a 2 Remove driver module files If the drivers were installed using an RPM package type the following command rpm e kmod qlgc fastlinq version arch If the drivers were installed using a TAR file type the following commands for your operating system For RHEL cd lib modules version extra qlgc fastlinq rm rf qed ko qede ko qedr ko For SLES cd lib modules version updates qlgc fastlinq ...

Page 27: ...LES cd usr src packages rpmbuild dd SPECS fastlinq version spec 3 Install the newly compiled RPM rpm ivh RPMS arch kmod qlgc fastlinq version arch rpm The drivers will be installed in the following paths For SLES lib modules version updates qlgc fastlinq For RHEL lib modules version extra qlgc fastlinq 4 Turn on all ethX interfaces ifconfig ethX up 5 For SLES use YaST to configure your Ethernet in...

Page 28: ... drivers using the TAR file 1 Create a directory and extract the TAR files to the directory tar xjvf fastlinq minor version tar bz2 2 Change to the recently created directory and then install the drivers cd fastlinq minor version make clean make install The qed and qede drivers will be installed in the following paths For SLES lib modules version updates qlgc fastlinq For RHEL lib modules version ...

Page 29: ... Download the QL45212 Adapter Linux drivers from dell support com 2 Configure RoCE on the adapter as described in Configuring RoCE on the Adapter for Linux on page 55 3 Remove existing Linux drivers as described in Removing the Linux Drivers on page 10 4 Install the new Linux drivers using one of the following methods Installing Linux Drivers Using the src RPM Package on page 12 Installing Linux D...

Page 30: ...4 Install the new Linux drivers as described in Installing Linux Drivers Using the TAR File on page 13 5 Build and install the libqedr user space driver using one of the following procedures Procedure 1 cd fastlinq minor version make libqedr_install Procedure 2 cd fastlinq minor version libqedr version configure prefix usr libdir exec_prefix lib64 sysconfdir etc make install 6 Test the drivers by ...

Page 31: ...NTx mode 1 or MSI mode 2 gro_disable Enables 0 or disables 1 the HW GRO feature The default is 0 no_tracing Enables 0 or disables 1 the transmission of all driver logs to the ftrace trace buffer Table 3 3 Linux Driver Parameter Defaults Parameter qed Driver Default qede Driver Default Speed Auto negotiation with speed advertised Auto negotiation with speed advertised MSI MSI X Enabled Enabled Flow...

Page 32: ...ity See the ethtool man page for more information Windows Driver Software This section describes the following topics Installing the Windows Drivers Removing the Windows Drivers Managing Adapter Properties Setting Power Management Options Installing the Windows Drivers You can run a software or driver Dell update package in two ways Running the Dell Update Package in the GUI Running the Dell Updat...

Page 33: ...4503 00 B 2 In the Dell Update Package window Figure 3 1 click Install Figure 3 1 Dell Update Package Window 3 In the QLogic Super Installer InstallShield Wizard s Welcome window Figure 3 2 click Next Figure 3 2 QLogic InstallShield Wizard Welcome Window ...

Page 34: ...503 00 B 4 In the wizard s License Agreement window Step 3 3 a Read the QLogic End User Software License Agreement b To continue select I accept the terms in the license agreement c Click Next Figure 3 3 QLogic InstallShield Wizard License Agreement Window ...

Page 35: ...dow Figure 3 4 as follows a Select one of the following setup types Click Complete to install all program features Click Custom to manually select the features to be installed b To continue click Next If you clicked Complete proceed directly to Step 6b Figure 3 4 InstallShield Wizard Setup Type Window ...

Page 36: ... select one of the following options This feature will be installed on the local hard drive This setting marks the feature for installation without affecting any of its subfeatures This feature and all subfeatures will be installed on the local hard drive This setting marks the feature and all of its subfeatures for installation This feature will not be available This setting prevents the feature ...

Page 37: ...The InstallShield Wizard installs the QLogic Adapter drivers and Management Software Installer Figure 3 6 InstallShield Wizard Ready to Install the Program Window 8 When the installation is complete the InstallShield Wizard Completed window appears Figure 3 7 Click Finish to dismiss the installer Figure 3 7 InstallShield Wizard Completed Window ...

Page 38: ...e Package Window Running the Dell Update Package from the Command Line Running the Dell update package from the command line with no options specified results in the same behavior as double clicking the icon representing the Dell update package To run the Dell update package from the command line C DUP_file_name EXE The following shows the syntax for specifying options to customize the Dell update...

Page 39: ...o extract update contents to a directory e path To extract only the driver components to a directory drivers path To install or update only the driver components driveronly To define a specific path for the Dell update package s log file l path NOTE This command requires the s option NOTE This command requires the s option NOTE This command requires the s option NOTE This option cannot be used in ...

Page 40: ...exe s To extract the update contents to the C mydir directory DUP_file_name exe s e C mydir To extract the driver components to the C mydir directory DUP_file_name exe s drivers C mydir To install only the driver components DUP_file_name exe s driveronly To change from the default log location to C my path with spaces log txt DUP_file_name exe l C my path with spaces log txt Removing the Windows D...

Page 41: ...omputer If the device is busy doing something servicing a call for example the operating system will not shut down the device The operating system attempts to shut down every possible device only when the computer attempts to go into hibernation To have the controller stay on at all times do not click the Allow the computer to turn off the device to save power check box Figure 3 9 Figure 3 9 Power...

Page 42: ... and then extract the vib file 4 Copy the vib file to the ESX server You can place the file anywhere that is accessible to the ESX console shell Type the following command to use the Linux scp utility to copy a vib file from a local system into the tmp directory on an ESX server with IP address 10 10 10 10 scp qedentv 1 0 3 11 1OEM 550 0 0 1331820 x86_64 vib root 10 10 10 10 tmp 5 Place the host i...

Page 43: ... or receive fully formed packets hw_vlan 1 is the default num_queues The num_queues parameter specifies the number of Tx Rx queue pairs num_queues can be 1 11 or one of the follow ing 1 Allow the driver to determine the optimal number of queue pairs default 0 Use the default queue You can specify multiple values delimited by commas for mul tiport or multifunction configurations multi_rx_filters Th...

Page 44: ...t The auto_fw_reset parameter enables 1 or disables 0 the driver automatic firmware recovery capability When this parameter is enabled the driver attempts to recover from events such as transmit timeouts firmware asserts and adapter parity errors The default is auto_fw_reset 1 vxlan_filter_en The vxlan_filter_en parameter enables 1 or disables 0 the VxLAN filtering based on the outer MAC the inner...

Page 45: ...e the following command vmkload_mod u qedentv Table 3 5 VMware Driver Parameter Defaults Parameter Default Speed Auto negotiation with speed advertised Flow Control Auto negotiation with Rx and Tx advertised MTU 1 500 range 46 9 600 Rx Ring Size 8 192 range 128 8 192 Tx Ring Size 8 192 range 128 8 192 MSI X Enabled Transmit Send Offload TSO Enabled Large Receive Offload LRO Enabled RSS Enabled fou...

Page 46: ...mware Dell update package can be run from the command line and supports a number of command line options Run by Double Clicking To run the firmware Dell update package by double clicking the executable file follow these instructions 1 Double click the icon representing the firmware Dell update package file The Dell update package splash screen appears as shown in Figure 4 1 Click Install to contin...

Page 47: ...rading Firmware Run by Double Clicking 32 BC0154503 00 B 2 Follow the on screen instructions Click Yes to continue the installation as shown in Figure 4 2 Figure 4 2 Continue Dell Update Package Installation ...

Page 48: ...4 Upgrading Firmware Run by Double Clicking 33 BC0154503 00 B 3 The installer indicates that it is loading the new firmware as shown in Figure 4 3 Figure 4 3 Loading New Firmware ...

Page 49: ...503 00 B 4 When complete the installer indicates the result of the installation as shown in Figure 4 4 Click Yes to reboot Figure 4 4 Result of Installation 5 Click Finish to complete the installation as shown in Figure 4 5 Figure 4 5 Finish Installation ...

Page 50: ... line with no options specified results in the same behavior as double clicking the Dell update package icon Note that the actual file name of the Dell update package will vary C Network_Firmware_2T12N_WN32_ version _X16 EXE The options shown in Figure 4 6 can be used to customize the Dell update package installation Figure 4 6 Command Line Options ...

Page 51: ...t tasks using the Human Infrastructure Interface HII application These tasks include the following Displaying Firmware Image Properties Device Level Configuration NIC Configuration Data Center Bridging Configuration NIC Partitioning Configuration NOTE The HII screens in this chapter are representative and may not match the screens that you see on your system ...

Page 52: ...1 Open the System Setup page for your platform For information about launching the System Setup consult the user guide for your platform 2 In the System Setup page select Device Settings and then click Finish 3 In the Device Settings page select the QL45212 adapter port that you want to configure ...

Page 53: ... adapter properties shown in Table 5 1 Table 5 1 Adapter Properties Adapter Property Description Device Name Factory assigned device name Chip Type ASIC version PCI Device Id Unique vendor specific PCI device ID PCI Address PCI device address in bus device function format Blink LEDs User defined blink count for the port LED Link Status External link status MAC Address Manufacturer assigned permane...

Page 54: ...l firmware images Controller BIOS Version management firmware version EFI Version Unified extensible firmware interface UEFI extensible firmware interface EFI driver version Device Level Configuration Device level configuration comprises enabling single root I O virtualization SR IOV and or NIC partitioning and enabling or disabling NPAReP To perform device level configuration select Device Level ...

Page 55: ...with SR IOV None Disables all virtualization 3 If NPAR is enabled with or without SRIOV and the system is capable of alternate routing ID interpretation ARI choose to enable or disable NPAReP mode Click Disabled to specify four partitions per port Click Enabled to specify eight partitions per port 4 Click Back 5 When prompted click Yes to save the changes Changes will take effect after a system re...

Page 56: ...orts The link speed and the forward error correction FEC must match that of the connected switch or device port 3 Click Back 4 When prompted click Yes to save the changes Changes will take effect after a system reset Table 5 2 Link Speed Options Option Description Auto Negotiated Automatically negotiates the link parameters with the con nected switch or device default Forward error correction FEC ...

Page 57: ...choose from the following options Disabled Disables DCB IEEE IEEE protocol CEE Converged Enhanced Ethernet CEE protocol Dynamic Dynamically applies CEE or IEEE to match the attached link partner 3 In the Data Center Bridging DCB Settings window click the RoCE Priority field and then a type a value from 0 7 4 Click Back 5 When prompted click Yes to save the changes Changes will take effect after a ...

Page 58: ... maximum physical port link speed Values can be 0 100 When DCBX ETS mode is enabled the per traffic class DCBX ETS minimum bandwidth value overrides the per partition minimum TX bandwidth value The total of the minimum TX bandwidth values of all partitions on a single port must equal 100 or be all zeros Partition N Maximum TX Bandwidth is the maximum transmit bandwidth of the selected partition ex...

Page 59: ...aring the Adapter Preparing the Ethernet Switch Configuring RoCE on the Adapter for Windows Server Configuring RoCE on the Adapter for Linux Supported Operating Systems and OFED RoCE is supported on the following operating systems Windows 2012 R2 RHEL 6 6 6 7 7 1 7 2 SLES 11 SP4 12 SP1 RoCE is supported on OFED as follows In box OFED all operating systems Out of box OFED 3 12 1 GA RHEL 6 6 Out of ...

Page 60: ...ftest OFED version type the following command ib_send_bw version RHEL 7 2 does not support the current out of box OFED3 18 1 version qedr cannot operate with RDMA connection manager CM when I O memory management unit IOMMU is enabled Applications that do not support RDMA work with IOMMU For any RDMA application problem that you encounter compare your results with those reported for other brands of...

Page 61: ...y value This value should match the corresponding value on the DCB switch In this example type 4 4 Click Back 5 When prompted click Yes to save the changes Changes will not take effect unit after a system reset Preparing the Ethernet Switch This section describes how to configure a Cisco Nexus 6000 Ethernet Switch and a Dell Z9100 Ethernet switch for RoCE Cisco Nexus 6000 Ethernet Switch Dell Z910...

Page 62: ...e switch config class type queuing class roce switch config bandwidth percent 50 7 Configure network QoS policy maps to set priority flow control for no drop traffic class switch config policy map type network qos roce switch config class type network qos class roce switch config pause no drop 8 Apply the new policy at the system level switch config system qos switch config service policy type qos...

Page 63: ... roce Dell conf dcbmap roce 2 Configure two ETS traffic classes in the DCB map with 50 percent bandwidth assigned for RoCE group 1 Dell conf dcbmap roce priority group 0 bandwidth 50 pfc off Dell conf dcbmap roce priority group 1 bandwidth 50 pfc on 3 Configure the PFC priority to match the adapter priority 4 Dell conf dcbmap roce priority pgid 0 0 0 0 1 0 0 0 4 Verify the DCB map configuration Pr...

Page 64: ...K TSA Rate Mbps Burst KB Rate Mbps Burst KB 0 0 1 2 3 5 6 7 40 ETS 1 4 60 ETS 2 3 Dell Conf do show interfaces twentyFiveGigE 1 5 1 pfc detail Interface twentyFiveGigE 1 5 1 Admin mode is on Admin is enabled Priority list is 4 Remote is enabled Priority list is 4 Remote Willing Status is enabled Local is enabled Priority list is 4 Oper status is init PFC DCBX Oper status is Up State Machine Type i...

Page 65: ... Rx pkts 66 Input Appln Priority TLV pkts 99 Output Appln Priority TLV pkts 0 Error Appln Priority TLV Pkts 7 Configure the DCBX protocol CEE in this example Dell conf interface twentyFiveGigE 1 8 1 Dell conf if tf 1 8 1 protocol lldp Dell conf if tf 1 8 1 lldp dcbx version cee 8 Assign a VLAN ID to the switch port to match the VLAN ID assigned to the adapter 5 Dell conf interface vlan 5 Dell conf...

Page 66: ...r for Windows Server comprises enabling RoCE on the adapter and verifying the Network Direct MTU size To configure RoCE on a Windows Server host 1 Enable RoCE on the adapter a Open the Windows Device Manager and then open the QL45212 NDIS Miniport Properties b Select Network Direct Functionality from the Property menu and then select Enabled in the Value field Click OK ...

Page 67: ...RDMA both ports are enabled PS C Users Administrator Get NetAdapterRdma Name InterfaceDescription Enabled SLOT 4 3 Port 1 QLogic FastLinQ QL45212 True SLOT 4 3 Port 2 QLogic FastLinQ QL45212 True 4 Verify that RoCE is enabled on the host operating system using PowerShell The Get NetOffloadGlobalSetting command shows NetworkDirect is enabled PS C Users Administrators Get NetOffloadGlobalSetting Rec...

Page 68: ... drive run RoCE traffic and verify the results a Map an SMB drive and then run RoCE traffic b Launch Performance Monitor c Select Local computer under Select counters from computer Select Add RDMA Activity and then select All instances under Instances of selected object The following figure shows the presence of RDMA traffic ...

Page 69: ...nnection 192 168 10 20 16159 192 168 10 10 445 0 Kernel 60 Connection 192 168 10 20 15903 192 168 10 10 445 0 Kernel 56 Listener fe80 e11d 9ab5 a47d 4f0a 56 445 NA 0 Kernel 56 Listener 192 168 11 20 445 NA 0 Kernel 60 Listener fe80 71ea bdd2 ae41 b95f 60 445 NA 0 Kernel 60 Connection 192 168 10 20 445 NA 0 6 By default Microsoft s SMB Direct establishes two RDMA connections per port which provides...

Page 70: ...box OFED for RHEL Configuring In box OFED for RHEL To prepare in box OFED for RHEL 1 While installing or upgrading the operating system select the Infiniband and OFED support packages 2 Install the following RPMs from RHEL ISO image libibverbs devel x x x x86_64 rpm required for libqedr library perftest x x x x86_64 rpm required for IB bandwidth and latency applications or using Yum install the in...

Page 71: ...pted for a few dependency packages that you will need to copy from the operating system ISO image 4 When the installation is complete reboot the sever 5 Compile and install the FastLinQ drivers from the TAR file as described in Installing Linux Drivers for RoCE in an Out of box OFED Environment on page 15 The FastLinQ package contains all relevant driver modules and libqedr qed driver core qede Et...

Page 72: ...rpm required for libqedr installation perftest x x x x86_64 rpm required for bandwidth and latency applications 3 Install the Linux drivers as described in Installing Linux Drivers for RoCE in an In box OFED Environment on page 14 Installing Out of box OFED for SLES Linux To install out of box OFED for SLES Linux 1 Download latest out of box OFED from the following link http downloads openfabrics ...

Page 73: ... that the RoCE devices were detected on all Linux operating systems 1 Stop firewall tables using service systemctl commands 2 Verify that the RDMA services has started using the service rdma status command If RDMA has not started type the following command service rdma start If RDMA does not start type either of the following alternative commands etc init d rdma start or systemctl start rdma servi...

Page 74: ... 020e 1eff fe50 c7c0 sys_image_guid 020e 1eff fe50 c7c0 vendor_id 0x1077 vendor_part_id 5684 hw_ver 0x0 phys_port_cnt 1 port 1 state PORT_ACTIVE 1 max_mtu 4096 5 active_mtu 1024 3 sm_lid 0 port_lid 0 port_lmc 0x00 link_layer Ethernet 7 Verify the L2 and RoCE connectivity between all servers one server acts as a server another acts as a client Verify the L2 connection using a simple ping command Ve...

Page 75: ...c570 8192000 bytes in 0 05 seconds 1436 97 Mbit sec 1000 iters in 0 05 seconds 45 61 usec iter Client root lambodar ibv_rc_pingpong d qedr0 g 0 192 168 10 165 local address LID 0x0000 QPN 0xff0000 PSN 0x934d28 GID fe80 20e 1eff fe50 c570 remote address LID 0x0000 QPN 0xff0000 PSN 0xb3e07e GID fe80 20e 1eff fe50 c7c0 8192000 bytes in 0 02 seconds 4211 28 Mbit sec 1000 iters in 0 02 seconds 15 56 us...

Page 76: ... be used for any perftest applications ibv_devinfo d qedr0 v grep GID GID 0 fe80 0000 0000 0000 020e 1eff fe50 c5b0 GID 1 0000 0000 0000 0000 0000 ffff c0a8 0103 GID 2 2001 0db1 0000 0000 020e 1eff fe50 c5b0 GID 3 2001 0db2 0000 0000 020e 1eff fe50 c5b0 GID 4 0000 0000 0000 0000 0000 ffff c0a8 0b03 IP address for VLAN interface GID 5 fe80 0000 0000 0000 020e 1e00 0350 c5b0 VLAN ID 3 NOTE The defau...

Page 77: ...orming interface down up tests or performing cable pull tests can cause driver or iSER module errors that may crash the system If this happens reboot the system Configuring iSER for RHEL To configure iSER for RHEL 1 Configure in box OFED as described Configuring In box OFED for RHEL on page 55 Out of box OFEDs are not supported for iSER because the ib_isert module is not available in the out of bo...

Page 78: ...loaded on the initiator and target devices using the lsmod grep qed and lsmod grep iser commands 6 Verify that there are separate hca_id instances by typing the ibv_devinfo command as described in Step 6 on page 59 7 Check the RDMA connection on the initiator device and the target device Initiator device rping s C 10 v Target device rping c a 192 168 100 99 C 10 v The following figure shows an exa...

Page 79: ...igure 9 Install Linux iSCSI Initiator Utilities using the yum install iscsi initiator utils commands a Discover the iSER target by typing command such as the following iscsiadm m discovery t st p 192 168 100 99 3260 b Change the transport mode to iSER by typing command such as the following iscsiadm m node T iqn 2015 06 test target1 o update n iface transport_name v iser c Connect to or login to t...

Page 80: ...0 B d Confirm that the Iface Transport is iser in the target connection as shown in the following figure Type a command such as the following iscsiadm m session P2 e Check for a new iSCSI device as shown in the following figure Type command such as the following lsscsi ...

Page 81: ...6_64 rpm python rtslib 2 2 6 6 noarch rpm python urwid 1 1 1 6 144 x86_64 rpm targetcli 2 1 3 8 x86_64 rpm 2 Before starting the TargetCLI load all RoCE device drivers and iSER modules modprobe qed modprobe qede modprobe qedr modprobe ib_iser Initiator modprobe ib_isert Target 3 Configure NIC interfaces and run L2 and RoCE traffic before configuring iSER targets as described in Step 7 on page 59 4...

Page 82: ...sysctl w net ipv4 tcp_rmem 4096 87380 4194304 sysctl w net core wmem_max 4194304 sysctl w net core rmem_max 4194304 sysctl w net core wmem_default 4194304 sysctl w net core rmem_default 4194304 sysctl w net core netdev_max_backlog 250000 sysctl w net ipv4 tcp_timestamps 0 sysctl w net ipv4 tcp_sack 1 sysctl w net ipv4 tcp_low_latency 1 sysctl w net ipv4 tcp_adv_win_scale 1 echo 0 proc sys vm nr_hu...

Page 83: ...68 BC0154503 00 B Block Device Staging Set the block device staging settings for each iSCSI device target echo noop sys block sdd queue scheduler echo 2 sys block sdd queue nomerges echo 0 sys block sdd queue add_random echo 1 sys block sdd queue rq_affinity ...

Page 84: ...cable connections at the network adapter and the switch are attached properly Verify the adapter installation by reviewing Installing the Adapter on page 7 Verify that the adapter is properly seated in the slot Check for specific hardware problems such as obvious damage to board components or the PCI edge connector Verify the configuration settings and change them if they are in conflict with anot...

Page 85: ...ontention Verifying that Current Drivers are Loaded Windows See the Device Manager to view vital information about the adapter link status and network connectivity Linux To verify that the qed ko driver is loaded properly run lsmod grep i module name If the driver is loaded the output of this command shows the size of the driver in bytes The following example shows the drivers loaded for the qed m...

Page 86: ...de x x x x Testing Network Connectivity Network Connectivity Testing for Windows Network connectivity can be tested using the ping command to determine if the network connection is working 1 Click Start and then click Run 2 Type cmd in the Open box and then click OK 3 Type ipconfig all to view the network connection to be tested 4 Type ping ip_address and then press ENTER The ping statistics that ...

Page 87: ...ction is working The adapter link speed can be forced to 10Gbps or 25Gbps using the operating systems GUI tool or the ethtool command ethtool s ethX speed SSSS Microsoft Virtualization with Hyper V Microsoft Virtualization is a hypervisor virtualization system for Windows Server 2012 R2 For more information on Hyper V see https technet microsoft com en us library Dn282278 aspx Linux Problem QL4521...

Page 88: ... Miscellaneous Problem The QL45212 Adapter has shut down and an error message appears indicating that the fan on the adapter has failed Solution The QL45212 Adapter shut down to prevent permanent damage Contact QLogic Technical Support for assistance ...

Page 89: ...ink and Activity LEDs located adjacent to the port connector as described in Table A 1 Table A 1 Port LED Indications Port LED LED Appearance Network State Link LED Off No link cable discon nected Continuously illuminated Link Activity LED Off No network activity Blinking Network activity ...

Page 90: ...st be explicitly configured to connect to the adapter at 25Gbps To configure a Dell Z9100 switch port to connect to the QL45212 Adapter at 25Gbps 1 Establish a serial port connection between your management workstation and the switch 2 Open a command line session and then log in to the switch Login admin Password admin 3 Enable configuration of the switch port Dell enable Password xxxxxx Dell conf...

Page 91: ...h 10G speed 25G Quad port mode with 25G speed Dell conf stack unit 1 port 5 portmode quad speed 25G For information about changing the adapter link speed see Testing Network Connectivity on page 71 5 Verify that the port is operating at 25Gbps Dell Dell show running config grep port 5 stack unit 1 port 5 portmode quad speed 25G 6 To disable auto negotiation on switch port 5 do the following a Iden...

Page 92: ...conf if tf 1 5 1 do show run interface tw 1 5 1 interface twentyFiveGigE 1 5 1 no ip address mtu 9416 switchport flowcontrol rx on tx on no shutdown no intf type cr4 autoneg For more information about configuring the DZ9100 switch refer to the Dell Z9100 Switch Configuration Guide at support dell com ...

Page 93: ...ons to satisfy the require ments of protocols and applications in the data center Because existing high perfor mance data centers typically comprise multiple application specific networks that run on different link layer technologies Fibre Channel for storage and Ethernet for network management and LAN connectivity DCB enables 802 1 bridges to be used for the deployment of a converged network wher...

Page 94: ...oads human interface infrastructure HII A specification part of UEFI 2 1 for managing user input localized strings fonts and forms that allows OEMs to develop graphical interfaces for preboot configuration Institute of Electrical and Electronics Engineers IEEE An international nonprofit organization for the advancement of technology related to electricity maximum transmission unit MTU Refers to th...

Page 95: ... the memory of another with address and size semantics over a network This capability is an important feature of VI networks serializer deserializer SerDes A pair of functional blocks commonly used in high speed communications to compen sate for limited input output These blocks convert data between serial data and parallel interfaces in each direction unified extensible firmware interface UEFI A ...

Page 96: ...class map 46 47 Coalesce Rx Microseconds 16 Coalesce Tx Microseconds 16 connection manager 45 converged enhanced Ethernet See CEE D DAC cable 5 data center bridging exchange See DCBX DCB map 48 51 DCBX 42 enable 46 protocol 50 debug parameter 16 29 Dell update package command line 35 GUI 31 device ID 38 name 38 device level configuration 39 Device Manage 70 direct attach copper See DAC disable_tpa...

Page 97: ...48 Extensible Firmware Interface See EFI F fan failure 73 features 1 filtering VxLAN 29 firmware properties 39 reset 29 upgrade 31 flow control 16 30 frequency adaptive interrupt 3 G GID 61 global ID See GID gro_disable parameter 16 H hardware installation 5 HII 36 46 human interface infrastructure See HII hw_vlan parameter 28 30 Hyper V 72 I IEEE protocol 42 information level 29 int_mode paramete...

Page 98: ...8 30 O OFED 9 in box RHEL 55 in box SLES 57 limitations 45 out of box RHEL 56 out of box SLES 57 open fabrics enterprise distribution See OFED P packet jumbo parameters Linux 16 partitions per port 40 PCIe 5 Performance Monitor 53 performance RoCE 54 PF 28 PFC 48 physical characteristics 3 physical function See PF policy map 46 47 port speed 41 power management 26 PowerShell Windows 52 priority PF...

Page 99: ...3 specifications 3 speed 41 src RPM package 12 SR IOV 40 standards 3 statistics driver 17 switch configuration 46 Nexus 6000 Ethernet 46 Z9100 Ethernet 48 75 system requirements 5 T TAR file 13 TCP segmentation offload See TSO TPA 28 transmit send offload See TSO troubleshooting 69 TSO 16 29 30 Tx Ring Size 16 30 U UEFI 39 Unified Extensible Firmware Interface See UEFI V verbose level 29 VF maximu...

Page 100: ......

Page 101: ...gistered trademarks of Dell Inc Microsoft Windows and Windows PowerShell are registered trademarks of Microsoft Corporation Linux is a registered trademark of Linus Torvalds VMware is a registered trademark of VMware Inc PCIe and PCI Express are registered trademarks of PCI SIG Corporation Cisco and Cisco Nexus are registered trademarks of Cisco Systems Inc Amphenol is a registered trademark of Am...