Mellanox Technologies MHEH28-XSC, User Manual

Page 1: ...Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Card User s Manual P N MHEH28 XSC MHEH28 XTC MHGH28 XSC MHGH28 XTC MHGH29 XSC MHGH29 XTC MHJH29 XSC MHJH29 XTC Rev 1 2 ...

Page 2: ...ogies Inc All other marks and names mentioned herein may be trademarks of their respective companies ConnectX IB Dual Port InfiniBand Document Number 2802 Mellanox Technologies Inc 2900 Stender Way Santa Clara CA 95054 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam 20692 Israel Tel 972 4 909 7200 Fax 972 4 959 3245 ...

Page 3: ...rt of a Cluster Firmware Update 13 Chapter 4 Adapter Card Interfaces 14 4 1 I O Interfaces 14 4 2 I2C Compatible Interface 15 4 3 Power 15 4 4 Memory 16 4 5 VPDs 16 AppendixA Specifications 26 A 1 Board Mechanical Drawing and Dimensions 26 A 2 EMC Certification Statements 27 A 3 MHEH28 XSC and MHEH28 XTC Specifications 29 A 4 MHGH28 XSC and MHGH28 XTC Specifications 29 A 5 MHGH29 XSC and MHGH29 XT...

Page 4: ...ayout for MHGH28 XSC 19 Table 11 VPD Layout for MHGH28 XTC 20 Table 12 VPD Layout for MHGH29 XSC 21 Table 13 VPD Layout for MHGH29 XTC 22 Table 14 VPD Layout for MHJH29 XSC 23 Table 15 VPD Layout for MHJH29 XTC 24 Table 16 HCA Cards Certification Status 27 Table 17 Specifications for MHEH28 XSC XTC 29 Table 18 Specifications for MHGH28 XSC XTC 29 Table 19 Specifications for MHGH29 XSC XTC 30 Table...

Page 5: ...ut 33 Figure 9 Tall Bracket of a Dual IB Port HCA Card 35 Figure 10 Connector Retention Clip 35 Figure 11 Extracting Connector Clip 36 Figure 12 Bracket Screws 36 Figure 13 Rotate the Bracket to Detach it From the Card 37 Figure 14 Hold Kapton Label With Pincers 37 Figure 15 Place Label on Print Side With Label s and Card s Holes Aligned 38 Figure 16 Ensure That Label is Well attached 38 Figure 17...

Page 6: ...tory Table Date Rev Comments Changes August 2008 1 2 Removed watermark May 2008 0 31 Added CTUVUS and CB report status to Cert Table in Appendix A April 2008 0 30 Preliminary release User s Manual for ConnectX IB Adapter Cards Added C tick a nd Mic for relevant cards Changed OFED link to Mellanox OFED Removed Confidential ...

Page 7: ... a customer login account Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits Mellanox Firmware Tools MFT User s Manual Document no 2204UG User s Manual describing the set of MFT firmware management tools for a single InfiniBand node See http www mellanox com under Firmware d...

Page 8: ...Gb s SDR or 20Gb s DDR or 40Gb s DDR Two bracket heights short or tall 1 1 Adapter Cards Table 3 lists the InfiniBand HCA cards described in this manual Table 3 HCA Cards Ordering Part Number OPN PCI Express SERDES Speed IB SDR DDR Short Tall Bracket RoHS Complian ce HCA IC Part Number HCA Card Photo 1 1 The HCA cards have a similar form and fit The main visible difference is in the bracket height...

Page 9: ...rts field I 8 to indicate PCI Express 2 0 x8 running at 2 5GT s field X X to indicate no on board memory field B S to indicate a short bracket and field R C to indicate RoHS R5 w Exemptions compliance Table 4 Mellanox HCA Cards Part Numbering Key HCA Card OPN MHTS I XBR Field Decoder M Mellanox Technologies H Adapter Type H InfiniBand Host Channel Adapter N Ethernet Network Interface Card S Expres...

Page 10: ... 2 Mellanox Technologies 1 3 Finding the GUID and Serial Number on the Adapter Cards All mellanox HCA adapter cards have a label on the printed side of the adapter card that has the card serial number and the card GUID ...

Page 11: ...ts Software Operating Systems Distributions For Windows see WinIB ReadMe at https docs mellanox com dm WinIB ReadMe html For Linux see Mellanox OpenFabrics Enterprise Distribution OFED software package available via the Mellanox OpenFabrics Web site http www mellanox com products ofed php Read all installation instructions before connecting the equipment to the power source This equipment should n...

Page 12: ...CI Express 8 cards each with a standard x8 edge connec tor Please consult the host machine documentation for instructions on how to install a PCI Express card This equipment should be installed replaced or serviced only by trained and qualified personnel Disposal of this equipment should be in accordance to all national laws and regulations This equipment should be installed in compliance with loc...

Page 13: ... Mellanox Firmware Tools MFT package This package is available for download along with its user s manual from the single HCA card firmware update page See http www mellanox com under Firmware downloads A firmware binaries table lists a binary file per HCA card The file name of each such binary is composed by combin ing the firmware name the firmware release version and the card part number Please ...

Page 14: ...finiBand ports 1 and 2 each having four Tx Rx pairs of SerDes Each of the HCA cards listed in Table 3 on page 8 based on this device provides access to these ports by means of two InfiniBand connectors These cards are also compliant with the IBTA specification 1 2 Connector 1 connects to port 1 of the device while connector 2 connects to port 2 Figure 2 Port Numbering Each of the HCA cards is embe...

Page 15: ...sical link or the logical link or both connections have not been established Figure 3 Physical and Logical Link Indications Note The short bracket has the same port and LED footprint as the tall bracket 4 2 I2 C Compatible Interface A three pin header on the HCA card is provided as the I2 C compatible interface See Figure 6 on page 26 for the loca tion on the board Figure 4 I2C Connector 4 3 Power...

Page 16: ...umper on each adapter card that indicates to the device whether an on board Flash device exists or is to be used Table 7 provides information on this jumper See Figure 6 on page 26 for the jumper location 4 4 3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C compatible interface The EEPROM is used for storing the Vital Product Data VPD The VPD format adheres to the PCI ...

Page 17: ...ngth 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHEH28 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC ...

Page 18: ...C STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword V0 STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CS0 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB ...

Page 19: ...7 Length 0x15 18 PN MHGH28 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword V0 STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CS0 94 Large Resource Type VPD W Tag...

Page 20: ...Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword V0 STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CS0 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB 0x9E 96 Length 15 8 MSB 0x00 97 VPD Keyword V1 STR ...

Page 21: ...n Card Part Number 17 Length 0x15 18 PN PN STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 SerialNumber STR_SPC 00 00XXXX XX 71 VPD Keyword V0 STR Misc Information 73 Length 0x10 74 Data PCIe Gen2 x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CS0 94 Large Resourc...

Page 22: ... Part Number 17 Length 0x15 18 PN MHGH29 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword V0 STR Misc Information 73 Length 0x10 74 Data PCIe Gen2 x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CS0 94 Large Reso...

Page 23: ...4 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN PN STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 SerialNumber STR_SPC 00 00XXXX XX 71 VPD Keyword V0 STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92...

Page 24: ...7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle QDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN PN STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision A1 STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 SerialNumber S...

Page 25: ...ength 15 8 MSB 0x00 97 VPD Keyword V1 STR EFI Driver version 99 Length 0x6 100 Data N A STR_SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR_SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length 0x6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Table 15 VPD Layout for MHJH29 XTC ...

Page 26: ...cards covered in this User s Manual have the same mechanical drawing and share the same dimen sions as depicted in Figure 6 Note All dimensions are in millimeters Figure 6 Schematic of the ConnectX IB HCA Card with CX4 Connectors 15 00 33 35 43 17 57 15 3 65 1 90 96 3 57 16 64 40 136 47 J5 I2C Connector J6 Flash Jumper ...

Page 27: ...nt NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equip ment generates uses and can radiate radio frequency energy and if not installed and used in accor Tab...

Page 28: ...atus complies with Canadian ICES 003 Cet appareil numérique de la classe A est conforme à la norme NMB 003 du Canada A 2 5 VCCI Statements Japan Class A Statement Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in wh...

Page 29: ...Gb s 2 5Gb s 8 InfiniBand Virtual Lanes for each port Yes All Ports SDR 2 0 SERDES 2 5 GT s EMC Safety Environmental RoHS FCC 47 CFR part 15 2006 subpart B class A ICES 003 2004 Issue 4 class A VCCI V 3 2007 04 class A EN 55022 1998 A1 2000 A2 2003 class A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A2 2005 EN 55024 1998 A1 2001 A2 2003 stan dards harmonized under EMC Directive 2004 108 EC Article ...

Page 30: ... A1 2001 A2 2003 standards harmonized under EMC Direc tive 89 336 EEC AS NZS 3548 IEC EN 60950 1 2001 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS R5 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specifica tion Release 1 2 Table 19 Specifications for MHGH29 XSC XTC Physical Power and Environmental Size Air Flow 4X 20Gb s Con...

Page 31: ... 003 2004 Issue 4 class A VCCI V 3 2007 04 class A AS NZSCISPR 22 2006 class A EN 55022 1998 A1 2000 A2 2003 class A EN 61000 3 2 2006 EN61000 3 3 1995 A1 2001 A2 2005 EN 55024 1998 A1 2001 A2 2003 standards harmonized under EMC Direc tive 2004 108 EC Article 6 2 AS NZS 3548 XTC only MIC BCC class A XTC only IEC EN 60950 1 2001 IEC 60068 2 64 29 32 RoHS R5 1 The auto negotiation protocol is propri...

Page 32: ...ess IBTA v1 2 Auto Negotiation1 40Gb s 5Gb s or 20Gb s 5Gb s or 10Gb s 2 5Gb s 8 InfiniBand Virtual Lanes for each port Yes All Ports QDR 2 0 SERDES 5 0 GT s 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specifica tion Release 1 2 EMC Safety Environmental RoHS XTC only ICES 003 2004 Issue 4 class A VCCI V 3 2007 04 class A AS...

Page 33: ...or Pinout Figure 8 InfiniBand CX4 Connector Pinout 5 1 2 3 4 5 1 2 3 4 Table 21 I2C compatible Connector Pinout Connector Pin Number HCA Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC IBtxIp 0 IBtxIn 0 IBtxIp 1 IBtxIn 1 IBtxIp 2 IBtxIn 2 IBtxIp 3 IBtxIn 3 IBtxOn 0 IBtxOp 0 IBtxOp 1 IBtxOn 1 IBtxOn 2 IBtxOp 2 IBtxOn 3 IBtxOp 3 G1 G2 G3 G4 G5 G6 G7 G8 G9 GND GND GND GND GND GND GND GND GND S1 S2 S3 S4 ...

Page 34: ...Name IB Port A Signal Name S1 IBtxIp 0 Rx_A1 S2 IBtxIn 0 Rx_A0 S3 IBtxIp 1 Rx_A3 S4 IBtxIn 1 Rx_A2 S5 IBtxIp 2 Rx_A5 S6 IBtxIn 2 Rx_A4 S7 IBtxIp 3 Rx_A7 S8 IBtxIn 3 Rx_A6 S9 IBtxOn 3 Tx_A6 S10 IBtxOp 3 Tx_A7 S11 IBtxOn 2 Tx_A4 S12 IBtxOp 2 Tx_A5 S13 IBtxOn 1 Tx_A2 S14 IBtxOp 1 Tx_A3 S15 IBtxOn 0 Tx_A0 S16 IBtxOp 0 Tx_A1 G1 G6 G9 H1 H2 Signal Ground GND G71 1 The Sense 3 3V signal is used to enable...

Page 35: ...yimide Label Assembling Short Bracket Figure 10 shows the bracket side view of an HCA card C 1 Removing Tall Bracket Figure 10 shows a connector retention clip and the designated names of its sections 1Using a small flat head screwdriver gently push up one hook of a connector s clip toward the connector s top side as shown in Figure 11 on page 36 a 2 Then push the other hook With both hooks unlatc...

Page 36: ...acing your thumb on the LED component 6 In a rotating move toward the component side of the card slide the bracket out of the connector Figure 13 b 7 Gently hold your thumb on the LED component 8 At the same time extract the bracket as shown in Figure 13 b Make sure to protect the LED while extracting the bracket a Gently Push One Hook of Clip b Gently Push Other Hook of Clip c Pull Clip Away Thes...

Page 37: ...s this label may have been installed at the factory Note The label can be provided by Mellanox Technologies P N MEC000821 The following steps are instructions for placing the polyimide label 1 Make sure your working area is ESD protected 2 Hold the label with light pincers See Figure 14 Figure 13 Rotate the Bracket to Detach it From the Card c Bracket Separated From Clips and Screws a Grip the Car...

Page 38: ... thumb on the label to assure the label is well attached onto the board See Figure 16 See also Figure 21 Print Side View After Short Bracket Assembly With Kapton Label on page 40 Now your card is ready for a short bracket assembly C 3 Assembling Short Bracket The short bracket can now be assembled onto the HCA card Figure 15 Place Label on Print Side With Label s and Card s Holes Aligned Figure 16...

Page 39: ... on the bracket should be pointing in the same direction as the PCI connector Make sure the LEDs are aligned into their intended bracket holes 2 Insert a screw along with a washer into each of the two holes on the card intended for holding the bracket 3 Use a torque screwdriver to apply up to 2 lbs in torque on each screw Figure 17 Place Short Bracket onto Card LED Holes Figure 18 Attach Bracket o...

Page 40: ...river to gently slide the clip hooks towards the connector s base side as shown in Figure 19 6 Repeat this step for the second clip See Figure 20 for the assembled short bracket side view and Figure 21 for the Print Side view showing the Kapton label Figure 19 Sliding Connector Clip Evenly Figure 20 Assembled Short Bracket Figure 21 Print Side View After Short Bracket Assembly With Kapton Label ...

Page 41: ...ssant le maximum recommandé de 55 C 131 F En outre pour garantir un bon écoulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne faut pas utiliser le matériel et il ne faut pas brancher ou débrancher les câbles Les câbles InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire très attention en les branchant...

Page 42: ...42 Rev 1 2 Mellanox Technologies 7 Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux ...