Mellanox Technologies ConnectX-4 Lx, User Manual

Page 1: ...www mellanox com ConnectX 4 VPI Single and Dual QSFP28 Adapter Card User Manual for Dell PowerEdge P N 0068F2 06W1HY 0NNJ2M 0JJN39 Rev 1 3...

Page 2: ...OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLI...

Page 3: ...2 Ethernet QSFP28 Interface 14 2 3 PCI Express Interface 14 2 4 LED Interface 14 Chapter 3 Hardware Installation 15 3 1 System Requirements 15 3 1 1 Hardware 15 3 2 Operating Systems Distributions 15...

Page 4: ...4 1 7 1 Updating the Device Online 32 4 1 7 2 Updating the Device Manually 32 4 1 7 3 Updating the Device Firmware Automatically upon System Boot 32 4 1 8 UEFI Secure Boot 33 4 1 8 1 Enrolling Mellano...

Page 5: ...of Tables Table 1 Revision History Table 7 Table 2 Documents List 8 Table 3 Single and Dual port EDR Adapter Cards 11 Table 4 Features 12 Table 5 Hardware and Software Requirements 22 Table 6 06W1HY 0...

Page 6: ...ng of the Dual port x16 Adapter Card 41 Figure 4 Single port Low Profile Bracket 42 Figure 5 Single port Full Height Bracket 42 Figure 6 Dual port Low Profile Bracket 43 Figure 7 Dual port Full Height...

Page 7: ...pecifications on page 38 Updated 0068F2 0NNJ2M Specifications on page 39 Updated Airflow Specifications on page 40 Updated Adapter Card LED Operations on page 37 Added Mechanical Drawing of the Single...

Page 8: ...ement tools for a single node See http www mellanox com Products Software Firm ware Tools Mellanox Firmware Utility mlxup User Manual and Release Notes Mellanox firmware update and query utility used...

Page 9: ...M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance...

Page 10: ...uit device These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing an...

Page 11: ...able 3 Single and Dual port EDR Adapter Cards Ordering Part Number OPN Single port Cards 06W1HY tall bracket 0JJN39 short bracket Dual port Cards 0068F2 tall bracket 0NNJ2M short bracket Data Transmis...

Page 12: ...802 3by 25 Gigabit Ethernet IEEE 802 3ae 10 Gigabit Ethernet IEEE 802 3az Energy Efficient Ethernet IEEE 802 3ap based auto negotiation and KRstartup IEEE 802 3ad 802 1AX Link Aggregation IEEE 802 1Q...

Page 13: ...s for example from GPU to CPU and therefore significantly reduces application run time ConnectX 4 advanced acceleration technology enables higher cluster effi ciency and scalability to tens of thousan...

Page 14: ...e The network ports of the ConnectX 4 adapter card are compliant with the IEEE 802 3 Ethernet standards listed in Table 4 Features on page 12 Ethernet traffic is transmitted through the cards QSFP28 c...

Page 15: ...2 Shut down your system if active 3 After shutting down the system turn off power and unplug the cord 4 Remove the card from its package Please note that the card must be placed on an antistatic surfa...

Page 16: ...9 lbf in torque on the screws 3 6 Card Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper...

Page 17: ...aligned with the slot opening and the adapter faceplate is visible against the system chassis 6 Secure the adapter with the adapter clips or screws 7 Close the system case 3 7 Cables and Modules To ob...

Page 18: ...g port See Section 7 4 Adapter Card LED Operations on page 40 4 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When data is being transferred...

Page 19: ...chanisms on the bracket clips or screws 4 Holding the adapter card from its center gently pull the adapter card from the PCI Express slot 5 When the port connectors reach the top of the chassis window...

Page 20: ...lue display box check the fields VEN and DEV fields are separated by In the display example above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox T...

Page 21: ...ectX 4 VPI Single and Dual QSFP28 Port Adapter Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 21 lspci grep i Mellanox Network controller Mellanox Technologies MT4115 Family Connect...

Page 22: ...arch iso You can download it from http www mellanox com Products Soft ware InfiniBand Drivers Table 5 Hardware and Software Requirements Requirements Description Platforms A server platform with an a...

Page 23: ...er nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre...

Page 24: ...on machine as root Step 2 Mount the ISO image on your machine Step 3 Run the installation script MLNX_OFED_LINUX x x x rhel6 3 x86_64 mlnx_add_kernel_support sh m tmp MLNX_OFED_ LINUX x x x rhel6 3 x8...

Page 25: ...such as prefix kernel version and installation parameters can be retrieved by running the command etc infiniband info Most of the Mellanox OFED components can be configured or reconfigured after the i...

Page 26: ...r the opt directory openshmem bupc fca and ibutils The kernel modules are installed under lib modules uname r updates on SLES and Fedora Distributions lib modules uname r extra mlnx ofa_kernel on RHEL...

Page 27: ...stallation machine as root Step 2 Mount the ISO image on your machine and copy its content to a shared location in your net work You can download it from http www mellanox com Products Software Infini...

Page 28: ...llanox 100 1 354 K s in 0s 2014 04 20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 sudo rpm import RPM GPG KEY Mellanox warning rpmts_HdrFromFdno Header V3 DSA SHA1 Signature key ID 6224c050...

Page 29: ...vma vpi installer package with KMP support mlnx ofed all Installs all available packages in MLNX_OFED mlnx ofed basic Installs basic packages required for running Mellanox cards mlnx ofed guest Insta...

Page 30: ...ype of installation is applicable to Debian and Ubuntu Operating Systems 4 1 6 1 Setting up MLNX_OFED apt get Repository Step 1 Log into the installation machine as root Step 2 Extract the MLNX_OFED p...

Page 31: ...024g 09FCC269 2013 08 11 sudo apt get update apt cache search mlnx ofed mlnx ofed vma eth MLNX_OFED vma eth installer package with DKMS support mlnx ofed hpc MLNX_OFED hpc installer package with DKMS...

Page 32: ...e Step 4 Reboot your machine after the firmware burning is completed 4 1 7 3 Updating the Device Firmware Automatically upon System Boot As of MLNX_OFED v3 1 x x x firmware can be automatically update...

Page 33: ...ded onto the system key ring by the kernel Follow these steps below to add the Mellanox s x 509 public key to your system Step 1 Download the x 509 public key Step 2 Add the public key to the MOK list...

Page 34: ...or each boot for the first module that either has no signature or whose key is not in the kernel key ring So it s much easier to miss this mes sage You won t see it on repeated tests where you unload...

Page 35: ...w mellanox com Products Software Firmware Tools 5 1 Firmware Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX4 Part Number 06W1HY Description ConnectX 4 VPI adapter...

Page 36: ...he adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working...

Page 37: ...m ibdev2netdev Card Detection lspci grep i Mellanox Mellanox Firmware Tool MFT Download and install MFT http www mellanox com content pages php pg management_tools menu_section 34 Refer to the User Ma...

Page 38: ...tiates when connected to another Mellanox InfiniBand product Ethernet 100GBASE CR4 100GBASE SR4 50G R2 EC 40GBASE CR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 25GBASE CR CR S 25GBASE SR 25GBASE LR 25G R EC...

Page 39: ...n connected to another Mellanox InfiniBand product Ethernet 100GBASE CR4 100GBASE SR4 50G R2 EC 40GBASE CR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 25GBASE CR CR S 25GBASE SR 25GBASE LR 25G R EC 10GBASE S...

Page 40: ...provide additional cooling For information on how to adjust the fan speed on Dell PowerEdge servers please refer to the Dell iDRAC User s Guide for that server Please contact your Dell account team fo...

Page 41: ...Mellanox Technologies 41 7 5 Board Mechanical Drawing and Dimensions Figure 2 Mechanical Drawing of the Single port x16 Adapter Card Figure 3 Mechanical Drawing of the Dual port x16 Adapter Card All d...

Page 42: ...er Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 42 7 6 Bracket Mechanical Drawings Figure 4 Single port Low Profile Bracket Figure 5 Single port Full Height Bracket Link and Activ...

Page 43: ...l PowerEdge Rev 1 3 Mellanox Technologies 43 Figure 6 Dual port Low Profile Bracket Figure 7 Dual port Full Height Bracket Port 1 Link and Activity Bi color LED Port 2 Link and Activity Bi color LED P...

Page 44: ...28 Connector Pinout Figure 8 Connector and Cage Views Table 9 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2...

Page 45: ...ccording to the PCI Express 3 0 specification 16 GND GND 17 RXP_1 Rx1p 18 RXN_1 Rx1n 19 GND GND 20 GND GND 21 RXN_2 Rx2n 22 RXP_2 Rx2p 23 GND GND 24 RXN_4 Rx4n 25 RXP_4 Rx4p 26 GND GND 27 ModPrsl_Port...

Page 46: ...ConnectX 4 VPI Single and Dual QSFP28 Port Adapter Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 46 Figure 9 PCIe x16 Connector Pinout...

Page 47: ...h Mellanox adapter card has a different identifier printed on the label serial number the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID an...

Page 48: ...ConnectX 4 VPI Single and Dual QSFP28 Port Adapter Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 48 Figure 11 0068F2 Board Labels Example Figure 12 0JJN39 Board Labels Example...

Page 49: ...ConnectX 4 VPI Single and Dual QSFP28 Port Adapter Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 49 Figure 13 0NNJ2M Board Labels Example...

Page 50: ...tee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings During periods of lightning activity do not work on the equipment or connect or dis connect cables Some co...

Page 51: ...l QSFP28 Port Adapter Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 51 CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and...

Page 52: ...d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation...

Page 53: ...ies 53 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposi...

Page 54: ...temperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blitzs...

Page 55: ...1 3 Mellanox Technologies 55 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f...

Page 56: ...e aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad d...

Page 57: ...nectX 4 VPI Single and Dual QSFP28 Port Adapter Card User Manual for Dell PowerEdge Rev 1 3 Mellanox Technologies 57 PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 6082...