Cray CS-Storm 500GT 3U, Hardware Manual

Page 1: ...CS Storm 500GT 3U Server Hardware Guide Rev C H 6150...

Page 2: ...Fan Control Utility 25 Management Daughter Card MDC 35 MDC Control Panel 35 MDC DIP Switch Configuration 36 PCIe Bifurcation of the 4 PCIe Switch Board 39 Environmental Specifications 40 S2600BP Moth...

Page 3: ...mance computing HPC equipment Installation and maintenance tasks should be performed by experienced technicians in accordance with the service agreement Related Publications CS Storm 500GT Hardware Re...

Page 4: ...e Memory Express NVMe A logical device interface specification for accessing non volatile storage media attached through a PCI Express PCIe bus NVMe is commonly flash memory that comes in the form of...

Page 5: ...USA INFORMATION TO THE USER This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to part 15 of the FCC Rules These limits are designed to provide r...

Page 6: ...Cet appareil numerique de la classe A est conforme a la norme NMB 003 du Canada VCCI Marking Class A Japan VCCI A C Tick Marking Class A Australia Replaceable Lithium battery Warning Information UL S...

Page 7: ...TER CONNECT ClusterStor CRAYDOC CRAYPAT CRAYPORT DATAWARP ECOPHLEX LIBSCI NODEKARE The following system family marks and associated model number marks are trademarks of Cray Inc CS CX XC XE XK XMT and...

Page 8: ...and indicators GPU accellerator card status LEDs Front cover Rear cover Model 7201 19in 3U chassis Model 7201A 19in 4U chassis Cover latches Front grill air intake Air vents exhaust Cover latches Ser...

Page 9: ...N 1 power supply redundancy Not ASHRAE compliant Balanced PCIe configuration Motherboard Intel S2600BP Processors Two Intel Xeon Scalable family processors up to 165W TDP Memory Capacity Up to 12 of...

Page 10: ...four 2200W AC power supplies 200 277VAC gold level efficiency 2 2 redundancy with 10 300W accelerators 3 1 redundancy with 10 400W accelerators 3 1 PSUs required Server supports multiple PCIe topologi...

Page 11: ...plane PSU blank The number of drives and PCIe cards can vary depending on server PCIe and GPU configuration Note Fans and fan cage There are 7 pluggable fans each with a 4 pin connector on the bottom...

Page 12: ...e drives The backplanes provide power for the drives and include separate cable connectors for SATA and NVMe drives The disk backplane provides PCIe 3 0 x4 for each drive slot Drive cage 1 can support...

Page 13: ...FPGA carrier NVIDIA carrier Blank one slot GPU carrier GPU Installation Order The figure shows the order for installing GPUs if the chassis is not fully populated This installation sequence must be f...

Page 14: ...lue The power button LED lights blue to indicate system power is on The power button is used to apply power to server components Pressing the power button initiates a request to the Baseboard Manageme...

Page 15: ...a steady amber color for all fatal errors that are detected during processor initialization A steady amber color indicates that an unrecoverable system failure condition has occurred Off Solid On Nor...

Page 16: ...over current or communication error Off No link or GPU is not detected On red PCIe link is up 8 0 GT s Gen3 Blinking 2 Hz PCIe link is up 5 0 GT s Gen2 Blinking 1 Hz PCIe link is up 2 5 GT s Gen1 The...

Page 17: ...bscrew secures the PCIe card release in place Dual rail configuration default PCIe 3 0 slots 1 and 2 Quad rail configuration optional PCIe 3 0 slots 1 4 Additional slots 3 and 4 come from the center s...

Page 18: ...Support Drive bay 2 supports up to 4 NVMe drives Internal SATA SSD Drives Up to 2 internal fixed SATA SSDs are supported with different cabling and or changes to drive support in bay 2 This drive bay...

Page 19: ...MDC MDC IFB SMBus I2C Bus PMBus Motherboard Slot 4 Slot 3 Slot 2 Slot 1 Bridge Mini SAS HD Fan VGA CPU 2 CPU 1 Fixed Internal Drives PCIe x16 PCIe x16 PCIe x16 PCIe x16 PCIe x16 Bridge Board I2C Contr...

Page 20: ...PU x16 GPU x16 GPU x16 Add on Card x16 PCIe Switch Board PCIe Switch 3 Network Card x16 NVMe 4x Optional x8 Optional x8 Optional x16 Optional x16 x16 x16 x16 x16 x16 x16 x16 x16 x16 x16 Network Card x...

Page 21: ...paddle board connectors on bottom of switch board CN1 CN6 CN1 CN2 CN3 CN4 CN5 CN6 Switch board connector Motherboard PCIe slot CN1 Slot 1 CN2 Slot 2 CN3 Slot 3 CN4 Slot 4 CN5 PCIe add in slot 1 CN6 PC...

Page 22: ...Figure 11 PCIe Connectors Bottom View CS Storm 500GT PCIe switch board Chassis bottom Bottom cover Twin ax cable paddle connectors PCIe Connections and Cabling H 6150 Rev C 22...

Page 23: ...r backplane distributes power and monitors all printed circuit assemblies PCAs in the chassis including the motherboard front panel controls and indicators fans and accelerators The power backplane pr...

Page 24: ...d over this bus The power supplies can receive 277VAC or 208VAC 200 277VAC input The 500GT balances the load on all available PSUs If a PSU fails the load is rebalanced across the remaining PSUs If us...

Page 25: ...hydrad The hydrad daemon is the main part of the hydra utility and runs as a service daemon on Linux OS It starts and stops by the init script at runlevel 3 4 and 5 When the service starts hydrad pars...

Page 26: ...nd to display verify the current GPU environment settings Modify the etc hydrad conf then restart the hydrad service The etc hydra conf file contains following parameters activefan on off default is o...

Page 27: ...Controls what events hydrad logs to the tmp hydrad log file nodepower on off default is off hydrad monitors motherboard power consumption normaltemp 0 C hightemp default is 60 C The minimum temperatur...

Page 28: ...s fan on off display fan status set duty cycle active control manual control power node gpu clear display PSU motherboard and GPU power status or reset the energy counter hydra config Display Configur...

Page 29: ...ds to enable or disable power to the GPUs Args non Display GPU status Bus PCI Mux Power Type Product ID FWVer for MIC GPU slave address Temperature and Status on Turn on the all GPU power off Turn off...

Page 30: ...on Set Active Fan control mode off Set Manual Fan control mode Set FAN speed duty 5 100 root hydra3 hydra fan ADT7462 Rev 04h ADT7462 Addr b0h Active Fan on Fan Stat RPM Duty FAN1 ok 9591 50 FAN2 ok...

Page 31: ...27 6144 5248 11 9 43 207 572 PSSH16220 H 02 Power 84 0 A 1122 W Peak 1226 W Average 1129 W Energy 3457 5 Wh in last 11013secs 3h 3m 33s ok hydra power node PMDev ADM1276 3 0 ok p 368 0 a 187 6 Power...

Page 32: ...or a group does not affect the other group s fan speed The fan speeds are determined by the GPUs within the same group No Power or Unknown GPU States If there is no power or the GPU state is unknown h...

Page 33: ...S P N G and F options can not be combined Usage hscan py options command Options h display this message w time waiting time for response packet i nic specific ethernet IF port eth0 eth1 m display Mac...

Page 34: ...lcP 00000 00 192 168 100 74 1A 1B 2A K40 2B K40 3A K20 3B K20 4A 4B Fan Information Fan information cannot be displayed with GPU information on the same command line The F option displays fan informa...

Page 35: ...to monitor the state of the server The MDC provides a CLI that can be used for server management No CDs or additional installation steps are required to use the CLI The MDC can be inserted and remove...

Page 36: ...r to the MDC DIP Switch Configuration section This is a standard full half duplex 10 100 base T Ethernet port with link and activity LEDs Blinking LED is On and Off for equal amount of time Winking LE...

Page 37: ...Figure 15 MDC DIP Switches CS Storm 500GT ON Up 1 Off Down 0 IDSW1 IDSW2 Slot ID Rack ID Management Daughter Card MDC H 6150 Rev C 37...

Page 38: ...flash 256 MB Internal Boot Device 3 3 3 4 Off Off On On SPI EEPROM 16 Mb N A SW3 Option Switches 4 1 4 2 Off On Off Off Debug mode MDC will log debug messages into tmp mdcd log No password validation...

Page 39: ...Slot 1 x8x8 SW2200 Default Slot 2 x16 Slot 3 x16 Slot 4 x16 Bifurcated Slot 2 x8x8 Slot 3 x8x8 Slot 4 x8x8 SW3200 Default Slot 5 x16 Slot 6 x16 Slot 7 x16 Bifurcated Slot 5 x8x8 Slot 6 x8x8 Slot 7 x8...

Page 40: ...ximum temperature of 95 F 35 C by 1 8 F 1 C 1 C per 1 000ft 305m of altitude above 5 000ft 1525m Temperature rate of change must not exceed 18 F 10 C per hour Operating humidity 8 to 80 non condensing...

Page 41: ...65W 40 lanes of Integrated PCIe 3 0 low latency I O Memory 16 DIMM slots in total across six memory channels Support for DDR4 DIMMs only DDR3 DIMMs are not supported Registered DDR4 RDIMM Load Reduced...

Page 42: ...VGA Bracket is installed on Riser slot 1 as a standard Riser Slot 2 Rear Left Side providing a x24 PCIe 3 0 lanes CPU1 Riser Slot 3 Front Left Side providing a x24 PCIe 3 0 lanes CPU2 Riser Slot 4 Mi...

Page 43: ...lot 4 Front panel Bridge board Fan connectors 1 2 3 Socket P LGA 3647 CPU 2 Socket P LGA 3647 CPU 1 Fabric sideband CPU1 Fabric sideband CPU2 M 2 SATA PCIe J4B1 J4B4 J6B3 USB 2 0 A1 A2 B1 D2 D1 C1 F1...

Page 44: ...tus LED on the chassis front panel and this motherboard Status LED are tied together and show the same state When the server is powered down transitions to the DC off state or S5 the Integrated BMC is...

Page 45: ...e overall power consumption of the system is referred to as System Power States There are a total of six different power states ranging from S0 the system is completely powered ON and fully operationa...

Page 46: ...ate no manageability blinking blue indicates u Boot is running but has not transferred control to BMC Linux System remains in this state 6 8 seconds after BMC reset while it pulls the Linux image into...

Page 47: ...mmon across all Intel server boards and systems that use the same generation of chipset are listed in the following table Each digit in the code is represented by a LED lit off sequence of whose count...

Page 48: ...and pin1 insert colors The left half keying insert is the same color as the body the right half inserts are white The LGA 3647 socket also supports both the E5 2600 v5 processors with embedded Intel...

Page 49: ...et Heatsink The heatsink is integrated into the PHM which is attached to the bolster plate springs by two captive nuts on either side of the heatsink The bolster plate is held in place around the sock...

Page 50: ...nment keys Corner standoff Heatsink leveling stud Processor Heat Sink Module PHM Assembly S2600BP Architecture The architecture of Intel Server Board S2600BP is developed around the integrated feature...

Page 51: ...lot 4 PCIe 3 0 x16 32 GB s PCIe 3 0 x8 16 GB s PCIe 3 0 x16 32 GB s SATA 6 GB s sSATA 6 GB s Port 2 USB 3 0 Port 0 Riser slot 1 Riser slot 2 S2600BP Processor Population Rules Although the Intel S2600...

Page 52: ...DIMM sizes of 4 GB 8 GB 16 GB 32 GB 64 GB and 128 GB depending on ranks and technology DIMM speeds of 1600 1866 2133 2400 2666 MT s Only Error Correction Code ECC enabled RDIMMs or LRDIMMs are support...

Page 53: ...B3 which will force the BMC into the proper update state The following procedure should be followed in the event the standard BMC firmware update process fails Normal BMC functionality is disabled wit...

Page 54: ...rds This is the only method by which the Administrator and User passwords can be cleared unconditionally Other than this jumper passwords can only be set or cleared by changing them explicitly in BIOS...

Page 55: ...he BIOS boots up to the Error Manager Page in the BIOS Setup utility In the BIOS Setup utility boot device Shell or Linux for example could be selected to perform the BIOS update procedure under Shell...

Page 56: ...not prompted to press the Hot Key After the POST process has completed and handed off the system boot process to the OS BIOS supported Hot Keys are no longer recognized Table 12 Intel S2600BP Boot tim...

Page 57: ...ed initially However in the event that a serious error occurs during POST the system will enter the BIOS Setup Utility and display the Error Manager screen instead of the Main screen Typically changin...

Page 58: ...ed when prompted the user has the ability to perform the following actions Access the F2 BIOS Setup Utility Configure all BIOS setup options in the F2 BIOS Setup Utility Clear both the Administrator a...

Page 59: ...he Power button System Reset button If Enabled the power and reset buttons on the server front panel are locked and they must be controlled via a system management interface S2600BP Motherboard Descri...