Mellanox Technologies ConnectX MHQH19-XSC, User Manual

Page 1: ...www mellanox com ConnectX IB HCA Adapter Card with QSFP Connectors User Manual P N MHRH19 XTC MHRH19 XSC MHRH29 XSC MHRH29 XTC MHQH19 XTC MHQH19 XSC MHQH29 XSC MHQH29 XTC Rev 1 5...

Page 2: ...AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY...

Page 3: ...3 2 FlexBoot 23 3 3 NVIDIA GPUDirect Support 23 3 4 Updating HCA Adapter Card Firmware 24 Chapter 4 HCA InfiniBand Card Installation 26 4 1 Hardware and Software Requirements 26 4 2 Installation Instr...

Page 4: ...Installing the New Bracket 45 Appendix D Avertissements de s curit d installation Warnings in French 47 Appendix E Sicherheitshinweise Warnings in German 48 Appendix F Advertencias de seguridad para l...

Page 5: ...Connector 17 Figure 6 Flash Jumper 18 Figure 7 Support Download Assistant 25 Figure 8 Hardware Devices 28 Figure 9 PCI Device 29 Figure 10 Schematic of the Dual Port Adapter Card MH RQ H29B X ST R 38...

Page 6: ...HRH 12 9 X ST C 19 Table 8 VPD Layout for MHQH 12 9 X ST C 21 Table 9 Hardware and Software Requirements 26 Table 10 Specifications for MHRH19 X ST C 34 Table 11 Specifications for MHQH19 X ST C 35 Ta...

Page 7: ...9 1 4 Added MH QR H 19 Short cards Added gaskets to bracket installation moved Bracket installation from Appendix to Chapter 5 December 2008 1 3 Fixed links to website Added single port cards August 2...

Page 8: ...Web page http www mellanox com Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in me...

Page 9: ...ox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database informa tion fo...

Page 10: ...on MT25408 ConnectX IBthe ConnectXintegrated circuit device The cards described in this manual have the following main features QSFP ports for connecting InfiniBand traffic at 10Gb s SDR 20Gb s DDR 40...

Page 11: ...this manual Table 3 HCA Cards List Ordering Part Number OPN IB Port Speed On board Memory Size Short Tall Brack et HCA Card with Brackets MHRH19 XTC 20Gb s DDR Single port No Mem Tall MHRH19 XSC 20Gb...

Page 12: ...Gb s DDR Dual port No Mem Tall MHRH29 XSC Short MHQH29 XTC 40Gb s QDR Dual port No Mem Tall MHQH29 XSC Short Table 3 HCA Cards List Ordering Part Number OPN IB Port Speed On board Memory Size Short Ta...

Page 13: ...icate two ports field I 9 to indicate PCI Express 2 0 x8 running at 5 0GT s field X X to indicate no on board memory field B S to indicate a short bracket and field R C to indicate RoHS R5 w Exemption...

Page 14: ...Installation Instructions 2 Over temperature 3 During Lightning Electrical Hazard 4 Copper Cable Connecting Disconnecting Read all installation instructions before connecting the equipment to the pow...

Page 15: ...nal Electrical Codes This equipment should be installed replaced or serviced only by trained and qualified personnel Disposal of this equipment should be in accordance to all national laws and regula...

Page 16: ...Numbering 2 1 1 InfiniBand Interface The ConnectX device is compliant with the InfiniBand Architecture Specification Release 1 2 1 It has compliant 4X InfiniBandports with four Tx Rx pairs of SerDes...

Page 17: ...er the physical link or the logical link or both connections have not been established Figure 4 Physical and Logical Link Indications 2 1 4 I2C Compatible Interface A three pin header on the HCA Adapt...

Page 18: ...e schematic in Appendix A Specifications on page 34 for the jumper location 2 3 3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C The EEPROM is used for storing the Vital P...

Page 19: ...neering Change Level of the card rev 45 Length 0x2 46 Revision A1 STR PCB revision 48 VPD Keyword SN STR Serial Number 50 Length 0x18 51 SerialNumber STR_SPC 00 00XXXX XX 75 VPD Keyword V0 STR Misc In...

Page 20: ...Adapter Card Interfaces Rev 1 5 Mellanox Technologies 20 255 Small Resource Type END Tag 0x11 0x78 Table 7 VPD Layout for MHRH 12 9 X ST C Offset Decimal Item Value Format Description...

Page 21: ...neering Change Level of the card rev 45 Length 0x2 46 Revision A1 STR PCB revision 48 VPD Keyword SN STR Serial Number 50 Length 0x18 51 SerialNumber STR_SPC 00 00XXXX XX 75 VPD Keyword V0 STR Misc In...

Page 22: ...Adapter Card Interfaces Rev 1 5 Mellanox Technologies 22 255 Small Resource Type END Tag 0x11 0x78 Table 8 VPD Layout for MHQH 12 9 X ST C Offset Decimal Item Value Format Description...

Page 23: ...er InfiniBand BoIB Boot over Ethernet BoE or Boot over iSCSI Bo iSCSI This technology is based on the Pre boot Execution Environment PXE standard specification and FlexBoot software is based on the op...

Page 24: ...un cat sys module ib_core parameters gpu_direct_enable 1 The number of shared pages by GPUDirect is reported under cat sys module ib_core parameters gpu_direct_shares 3 3 2 3 NVIDIA Driver Installatio...

Page 25: ...w mellanox com Support Download Center A firmware binaries table lists a binary file HCA Adapter per adaptercard The file name of each such binary is composed by combining the firmware name the firmwa...

Page 26: ...e for connection If the card is installed in a PCI slot with less lanes than the card requires then the adapter card will not provide the optimum data transfer 4 3 Set Up Table 9 Hardware and Software...

Page 27: ...r card using specific tools see http www mellanox com content pages php pg firmware_HCA_FW_identification Otherwise use the instructions below relevant to your particular card and OS 4 3 1 1 Windows 1...

Page 28: ...ardware Devices 3 Select a PCI Device InfiniBand Controller entry 4 Right click 5 Select Properties to display the PCI Device Properties window 6 Click the Details tab and select Device Instance Id Wi...

Page 29: ...VEN_15B3 DEV_6368 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 0x6368 this is a valid Mellanox Technologies PCI Device ID 8 If the PCI device does not have...

Page 30: ...h1 Supported ports TP Supported link modes 10baseT Half 10baseT Full 100baseT Half 100baseT Full 1000baseT Half 1000baseT Full Supports auto negotiation Yes Advertised link modes 10baseT Half 10baseT...

Page 31: ...p the downloaded file and run flint mstflint command flint d device i fw file guid GUID guids 4 GUIDS mac MAC macs 2 MACs burn Check VPI setup Ethernet InfiniBand configuration For Example flint d dev...

Page 32: ...e param_name value Ethernet Network Tuning To improve network perfomance by tuning your network see the Mellanox Performance Tuning Guide located at http www mellanox com related docs prod_software Pe...

Page 33: ...rt the connector up side down This may damage the adapter card 3 Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down...

Page 34: ...tic modules 0 C to 55 C Protocol Support Regulatory InfiniBand QoS RDMA Support Data Rate PCI Express IBTA v1 2 Auto Negotiationb 20Gb s 5Gb s or 10Gb s 2 5Gb s 4 InfiniBand Virtual Lanes for each por...

Page 35: ...t Yes All Ports QDR 2 0 SERDES 5 0 GT s b The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architec ture Specification Release 1 2 EMC Safety Env...

Page 36: ...N 55022 1998 A1 2000 A2 2003 class A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A2 2005 EN 55024 1998 A1 2001 A2 2003 standards harmonized under EMC Directive 2004 108 EC Article 6 2 AS NZS 3548 IEC E...

Page 37: ...W for active optic modules 12 8W for passive cables only 19 8W for active optic modules 0 C to 55 C Protocol Support Regulatory InfiniBand QoS RDMA Support Data Rate PCI Express IBTA v1 2 Auto Negotia...

Page 38: ...echanical tolerances are 0 1mm Table 14 HCA Cards Certification Status HCA Card P N FCC Class USA EN Class Europe ICES Class Canada VCCI Japan KCC BCC Korea C Tick MHRH19 XSC YES YES YES YES MHRH19 XT...

Page 39: ...YES YES YES YES MHRH29 XTC YES YES YES YES MHQH29 XSC YES YES YES YES YES MHQH29 XTC YES YES YES YES YES YES Table 14 HCA Cards Certification Status HCA Card P N FCC Class USA EN Class Europe ICES Cl...

Page 40: ...e limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate radio freq...

Page 41: ...ironment radio interference may occur in which case the user may be required to take corrective actions A 6 5 KCC Certification Korea Device User s information A CLASS A device commercial broadcasting...

Page 42: ...le Connector Plug and Pinout B 2 PCI Express x8 Connector Pinout These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 2 0 specif...

Page 43: ...ccRx 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 TX2p GND TX4n TX4p GND 1 1 1 1 1 1 1 1 1 1 20 GND GND GND RX3n RX1p RX1n 30 1 2 3 4 5 6 7 8 9 2 2 2 2 2 2 2 2 2 1 2 3 4 5 6 7 8 3 3 3 3 3 3 3 3 RX4p GND GND...

Page 44: ..._2 Rx2p 23 GND GND 24 RXN_4 Rx4n 25 RXP_4 Rx4p 26 GND GND 27 ModPrsl_Port0 Mod PrsL 28 IntL IntL 29 VccTx 30 Vcc1 31 LPMode_Port0 LPMode 32 GND GND 33 TXP_3 Tx3p 34 TXN_3 Tx3n 35 GND GND 36 TXP_1 Tx1p...

Page 45: ...m the Adapter Card Figure 13 Remove the Bracket 1 Remove the two screws holding the bracket in place 2 The bracket comes loose from the card 3 Save the two screws and the two fiber washers C 3 Install...

Page 46: ...from the procedure above step 1 Figure 15 Placing the Bracket on the Card 6 Make sure that the LEDs are aligned onto the bracket holes 7 Use a torque driver to apply up to 2 lbs in torque on the scre...

Page 47: ...maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne faut pas utiliser le...

Page 48: ...gstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blit...

Page 49: ...Adem s para garantizar una circu laci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pul gadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o descone...

Page 50: ...Rev 1 5 Mellanox Technologies 50 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio nales...