Cray ClusterStor L300, Installation Manual

Page 1: ...ClusterStor L300 and L300N Field Installation Guide 3 0 0 H 6163...

Page 2: ...LDN with Dual 48 Port LMN Switches 40 11 3 Cable SMU and 36 Port 40GbE LDN and Dual 24 Port LMN Switches 41 11 4 Cable SMU and 36 Port 40GbE LDN with Dual 48 Port LMN Switches 43 11 5 Cable SMU and 24...

Page 3: ...rs 101 16 3 4 ClusterStor L300 and L300N SSU Physical and Logical Port Numbering 101 16 3 5 Internal Rack Cabling Local Management Network 102 16 3 6 Internal Rack Cabling Local Data Network 107 16 3...

Page 4: ...tribution for PX2 5551 N4V2 138 16 5 17 Base Rack Optional AMMU Standard Power Distribution for PX2 5965X3 V2 139 16 5 18 Base Rack Optional AMMU Standard Power Distribution for PX2 5104X2 V2 140 16 5...

Page 5: ...dministration Guide S 2648 Scope and Audience This publication is written for trained Cray technicians and provides instructions to unpack and install a ClusterStor L300 and L300N system at the custom...

Page 6: ...AYPORT DATAWARP ECOPHLEX LIBSCI NODEKARE The following system family marks and associated model number marks are trademarks of Cray Inc CS CX XC XE XK XMT and XT The registered trademark LINUX is used...

Page 7: ...of the following procedures 1 Complete the pre installation checklist 2 Unload the shipping crates 3 Uncrate the cabinet s 4 Move each rack into position 5 Install the disk drives 6 Cable the system 7...

Page 8: ...Client Network LCN Local management network LMN Lustre client network LCN unspecified network connection Customer client node Customer server node Server node Chassis AMMU MDS MGS MDS MGS EAC 1 EAC 0...

Page 9: ...lifting capacity Forklift with tine side travel is strongly recommended Rack Weight and Dimensions If the system cabinets do not fit through all access doors you may need to uncrate the system cabine...

Page 10: ...commands require access to the administrative admin user account However root access is required to perform the system installation procedure Contact Cray Support for root access 3 1 Service Area Clea...

Page 11: ...system is identified with R1 BASE Storage racks installed in manufacturing test alongside the base rack are numbered sequentially R2 to Rn where n is the highest number that was physically installed w...

Page 12: ...Precautions Check List Before unpacking the ClusterStor L300 and L300N system complete this prescribed checklist and observe the stated precautions CAUTION Read and understand these instructions compl...

Page 13: ...0 kg fully populated WARNING Personal injury or equipment property damage The rack side panels can potentially be a hazard when they become unlatched It is recommended that the side panels remain lock...

Page 14: ...ircuit Protection of capacity equal to the current rating of the distribution unit must be provided and must meet all applicable codes and regulations The plugs on the PDUs shall be installed near the...

Page 15: ...figuration of the rack at 1720 lbs 780 kg can enable the cabinet to roll and will not be damaged About this task This task describes how to safely remove the ClusterStor L300 and L300N rack from the s...

Page 16: ...rained person in front of the rack to guide it an slow its decent 5 Position the other trained person behind the rack to push CAUTION Personal Injury or Damage to Equipment The rack weighs 1720 lbs 78...

Page 17: ...ing it over the edge of the aluminum plate so that only 1 wheel makes contact with the plate About this task WARNING Personal Injury or Equipment Damage If the rack is fully populated with SSUs and eq...

Page 18: ...ation and lower the leveling feet using a 13mm wrench 4 Use a bubble level to verify the rack is level 5 From the bottom up repeat the steps above to insert the disk drives into the remaining enclosur...

Page 19: ...ramps to its final destination There are two methods of loading disks using the speedloader device which loads seven disks at once to speed disk population and the manual method which loads one disk a...

Page 20: ...ltiple disk method or the individual disk method to install the disk drives 3 Use a T 20 Torx screwdriver to unlock the two drawer latches on the front of each drawer in the lowest empty 5U84 enclosur...

Page 21: ...t slots as before 6 Repeat the process for the remaining 5U84 enclosures working upwards Individual Disk Load Method 7 To manually install disk drives remove the correct disk drive from the box of dri...

Page 22: ...e the two drawer latches with a T 20 Torx screwdriver rotate until the arrow points to the lock icon 13 If During Initial Rack Installation Until the rack is in its final position repeat for the secon...

Page 23: ...n the underside of the speedloader IMPORTANT Do not force the locking arm to close If necessary apply slight pressure to the drive s to align them and then close the locking arm 19 Carefully lift the...

Page 24: ...e the second drawer and secure the two drawer latches with a T 20 Torx screwdriver 26 If During Initial Rack Installation Until the rack is in its final position repeat for the second enclosure only w...

Page 25: ...e installed to maintain a balanced airflow CAUTION EQUIPMENT DAMAGE Observe ESD precautions when handling this equipment Failure to do so can result in equipment damage Procedure 1 If the disk drives...

Page 26: ...all the drives for the configuration IMPORTANT Any drive bay not occupied by a populated drive carrier must have a dummy drive carrier installed to ensure proper air circulation 7 Engage the drive ca...

Page 27: ...usterStor L300 and L300N system should be considered carefully as the loads are large Where possible balance the load equally across all 3 phases WARNING Serious Injury or Equipment Damage Always perf...

Page 28: ...B4 C19 B5 C19 B6 C19 A7 C19 A8 C19 A9 C19 B10 C19 B11 C19 B12 Line A Line B PX2 5965X3 V2 24A 208V US C19 A1 C19 A2 C19 A3 C19 B4 C19 B5 C19 B6 C19 C7 C19 C8 C19 C9 Line A Line B Line C 60A 208V US PX...

Page 29: ...4 C19 B5 C19 B6 C19 A7 C19 A8 C19 A9 C19 B10 C19 B11 C19 B12 Line A Line B PX2 5965X3 V2 24A 208V US C19 A1 C19 A2 C19 A3 C19 B4 C19 B5 C19 B6 C19 C7 C19 C8 C19 C9 Line A Line B Line C 60A 208V US PX2...

Page 30: ...Current 24A Phase 40A Phase 48A Phase 32A Phase Power Capacity 23 7kVA at 190V 25 9kVA at 208V 28 8kVA 208V 33 3kVA at 240V 17 3kVA at 208V 20 0kVA at 240V 42 2kVA at 380V 46 0kVA at 415V Image NOTE T...

Page 31: ...ge rack model and configuration being installed 8 1 Reset the Circuit Breaker Prerequisites The handle type circuit breaker located on the ClusterStor L300 and L300N power distribution unit PDU may re...

Page 32: ...rcuit Breaker Status Conditions 2 To trip the circuit breaker when required insert a small flat blade screwdriver into the small slot on the OFF side of the breaker and press The LEDs begin to alterna...

Page 33: ...Figure 18 Trip the Circuit Breaker Cable the External Power Connections 33...

Page 34: ...2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 3 9 4 0 4 1 4 2 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1 3 2 3 3 3 4...

Page 35: ...0 5 172 16 0 110 ssu0 n005 172 16 2 8 10 10 100 6 172 16 0 111 ssu1 n006 172 16 2 9 10 10 100 7 172 16 0 112 ssu1 n007 172 16 2 10 10 10 100 8 172 16 0 113 ssu2 n008 172 16 2 11 10 10 100 9 172 16 0 1...

Page 36: ...switches Base and storage racks Port 17 is reserved for connection between the base rack and 1 storage rack Ports 5 7 and 19 are also available Port 13 is the ISL and is not available When a 24 port s...

Page 37: ...21 22 23 24 X1 X2 X3 X4 X5 X6 X7 X8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 X1 X2 X3 X4 X5 X6 X7 X8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 X1 X2 X3 X4 X5...

Page 38: ...nd is not available IMPORTANT Switch port assignments in the configuration diagrams are provided for reference only For system specific inter rack switch cabling refer to the system specific storage c...

Page 39: ...itch IB0 available ports listed above and then to the customer core switch 2 In the base rack and each expansion rack connect a QSFP data cable to the upper InfiniBand switch IB1 available ports liste...

Page 40: ...nts in the configuration diagrams are provided for reference only For system specific inter rack switch cabling refer to the system specific storage configuration document SCD Figure 24 Cable Base Rac...

Page 41: ...he lower Ethernet switch SW0 in the base rack 4 In each expansion rack of a multi rack system connect a Cat 5 6 Ethernet cable to the upper Ethernet switch SW1 available ports listed above Connect the...

Page 42: ...5 16 17 18 19 20 21 22 23 24 X1 X2 X3 X4 X5 X6 X7 X8 SW1 SW0 LMN to storage racks 1 2 3 N SW1 switch LMN to storage racks 1 2 3 N SW0 switch Figure 27 Cable SMU to External Administration Network A B...

Page 43: ...at 5 6 Ethernet cable from the NIC card SMU node 1 Port 4 to the local network 11 4 Cable SMU and 36 Port 40GbE LDN with Dual 48 Port LMN Switches About this task For ClusterStor L300 and L300N inter...

Page 44: ...ct a QSFP data cable to the lower 40GS0 switch and then to the customer core switch 2 In the base rack and each storage rack connect a QSFP data cable to the upper 40GS1 switch and then to the custome...

Page 45: ...s are provided for reference only For system specific inter rack switch cabling refer to the system specific storage configuration document SCD Figure 31 Base Rack with 24 Port OPA LDN and Dual 24 Por...

Page 46: ...ted above and then to the customer core switch 3 In each storage rack of a multi rack system connect a Cat 5 6 Ethernet cable to the lower Ethernet switch SW0 available ports listed above Connect the...

Page 47: ...m specific inter rack switch cabling refer to the system specific storage configuration document SCD Figure 34 Cable Base Rack with 24 Port OPA LDN and Dual 48 Port LMN Using Core Switch OPA1 OPA0 LDN...

Page 48: ...other end of the cable to the upper Ethernet switch SW1 in the base rack 5 In the base rack connect a Cat 5 6 Ethernet cable from the NIC card SMU node 0 NIC 5 to the local network 6 In the base rack...

Page 49: ...ion network EAN EAC A Node n000 EAC B Node n001 pub0 interfaces NIC 5 SMU Figure 38 Cable Storage Rack with 24 Port OPA LDN and Dual 24 Port LMN Using Core Switch OPA1 OPA0 LDN to customer core switch...

Page 50: ...hernet cable from the NIC card SMU Node 1 NIC 5 to the local network 11 8 Cable SMU and Omni path 48 Port LDN with Dual 48 Port LMN Switches About this task For ClusterStor L300 and L300N inter rack 4...

Page 51: ...SMU to External Administration Network A B To external administration network EAN EAC A Node n000 EAC B Node n001 pub0 interfaces NIC 5 SMU Figure 41 Cable Storage Rack 48 Port OPA LDN Dual 24 Port L...

Page 52: ...base rack connect a Cat 5 6 Ethernet cable from the NIC card SMU node 0 NIC 5 to the local network 6 In the base rack connect another Cat 5 6 Ethernet cable from the NIC card SMU node 1 NIC 5 to the...

Page 53: ...ion network EAN EAC A Node n000 EAC B Node n001 pub0 interfaces NIC 5 SMU Figure 44 Cable Storage Rack with 32 Port 100G and 24 Port LAN Using Core Switch 100G1 100G0 LDN to customer core switch 1 2 3...

Page 54: ...e from the NIC card SMU node 1 NIC 5 to the local network 11 10 Cable SMU and 36 Port 100GbE LDN with Dual 48 Port LMN Switches About this task For ClusterStor L300 and L300N inter rack 100GbE LDN swi...

Page 55: ...100G0 in the diagram available ports listed above and then to the customer core switch 2 In the base rack and each storage rack connect a QSFP passive copper data cable to the lower 100GbE switch 100G...

Page 56: ...me browser is recommended Default username admin and password admin About this task Use this procedure to power on a ClusterStor L300 and L300N system version 3 0 0 or above IMPORTANT Do not use the b...

Page 57: ...lines with one power switch for each line some have none Verify that each line is powered on b For rack PDUs without power switches plug the PDU cords into the facility power receptacles 3 Enable pow...

Page 58: ...configured for 115 2 Kbps 8 data bits no parity 1 stop bit and no flow control 7 If the system includes an AMMU enable power to the AMMU by powering on the 1U servers and their EACs then power on the...

Page 59: ...empts which then skip to booting from the internal disk If another PXE server is present in the subnet cancel the PXE boot by pressing the ESC key or CTRL C The primary and secondary MGMT nodes and th...

Page 60: ...ctioning admin sn11100n000 sudo ping IPaddress If the statement Destination host unreachable appears in the output that node is not connected Repeat this step until both nodes appear or contact Cray s...

Page 61: ...ocf heartbeat XYRAID Started sn11100n001 sn111008n001_md66 fsys ocf heartbeat XYMNTR Started sn11100n001 sn11100n001_md66 stop ocf heartbeat XYSTOP Started sn11100n001 prm httpd lsb httpd Started sn11...

Page 62: ...XYRAID Started sn11100n001 sn11100n001_md66 raid ocf heartbeat XYRAID Started sn11100n001 sn11100n001_md66 fsys ocf heartbeat XYMNTR Started sn11100n001 sn11100n001_md66 stop ocf heartbeat XYSTOP Star...

Page 63: ...nown 18 Power on the remaining nodes as root root sn11100n000 pm 1 desired node 00x 00x sn11100n 00x 00y IMPORTANT For large systems to avoid system power surge and network timeouts limit selected nod...

Page 64: ...e system configuration or make configuration changes return system to Pre shipment mode admin sn11100n000 network scripts sudo cscli cluster_mode mode pre shipment Power On the ClusterStor L300 and L3...

Page 65: ...Google Chrome preferred 2 Enter the CSM wizard IP address in the browser s URL field The CSM wizard launches and provides prompts to perform the first run procedures beginning with entering the defaul...

Page 66: ...es error which can occur after selecting the Set Password and Proceed with Login button do not click the OK box on the error b Carefully inspect any errors generated by CSM Often the errors reveal the...

Page 67: ...al ClusterStor L300 and L300N systems support only privacy enhanced mail PEM format files If the SSL certificate file is in a different format convert it to PEM before continuing with this procedure T...

Page 68: ...d the license has been accepted and the SSL certificate has been uploaded if desired The Lustre Network screen has appeared About this task This procedure configures the Lustre network interfaces by s...

Page 69: ...he DNS Server s IP addresses if not already discovered 6 Enter the NTP Server s IP addresses This screen provides the ability to change optional parameters firewall settings IB subnet manager and netw...

Page 70: ...aux grep beSystem This command returns a list of currently running processes with a name matching the pattern 2 Check the status of the beSystemNetConfig sh script MGMT0 watch pstree la PID Observe if...

Page 71: ...0 cap 2000398934016 dev sdax parts 0 status Foreign Arrays dev1 sded t10 11100111120 slot 8 wwn 5000cca01c4c32e8 cap 2000398934016 dev sdee parts 0 status Foreign Arrays dev1 sday t10 1110011112 slot...

Page 72: ...system before running CSM MGMT0 cscli unmount f file_system_name If a node not unmounted error occurs check to see what is mounted MGMT0 pdsh a mount grep c lustre 2 SSH into the MGS node n002 3 Get t...

Page 73: ...ance tests on a ClusterStor L300 and L300N system When the tests complete the results display on the progress screen and are available for download Figure 56 CSSM Lustre Validation Tests Progress Scre...

Page 74: ...mgsnid lsn11022n002 o2ib0 request id 256 python2 6 m t0 backend fs_tests beRunSanity fsname sn11000 mpoint tmp tmphNDsuO mgsnid lsn11000n002 o2ib0 reque sh cPATH PATH usr sbin 040RUNAS_ID 301 040CLIEN...

Page 75: ...dbc t10 11100111120 slot 3 wwn 5000cca01b38b8a4 cap 2000398934016 dev sdek parts 0 status Foreign Arrays dev1 sdbe t10 11100111120 slot 4 wwn 5000cca01c4bd708 cap 2000398934016 dev sdef parts 0 status...

Page 76: ...ot 17 wwn 5000cca01b4ce748 cap 2000398934016 dev sdfm parts 0 status Foreign Arrays dev1 sdcg t10 11100111120 T10_key_begin GRD_CHK 1 APP_CHK 1 REF_CHK 1 ATO 1 RWWP 1 SPT 1 P_TYPE 1 PROT_EN 1 DPICZ 1...

Page 77: ...performed either on the primary MGMT node via CLI or CSM a From the primary MGMT node MGMT0 cscli mount f file_system_name Sample output admin dvtrack00 cscli mount f testfs mount MGS is starting mou...

Page 78: ...ocedure 1 Prior to configuring any nodes stop Lustre MGMT0 cscli unmount f file_system_name 2 In the Lustre Users screen select the Configure tab to select the desired Lustre user authentication metho...

Page 79: ...tory Figure 59 CSM Active Directory Screen 2 In the AD Server field enter the AD server s hostname or IP address if not already displayed The AD server can be specified either as a hostname or as an I...

Page 80: ...ge cluster must use authentication to bind to this AD server box b In the Bind DN field enter a valid bind DN value c In the Password field enter a valid password The password corresponds to the Authe...

Page 81: ...rver is being specified using the hostname the DNS must first be configured 3 In the Port field enter the port number value The default value of the port is 389 LDAP can be configured to run on a non...

Page 82: ...LDAP Storage Cluster Authentication Checkbox Screen IMPORTANT Use TLS SSL port 636 if TLS SSL is used with the LDAP a Check the Use TLS SSL for accessing LDAP box b Click the Choose a file button to...

Page 83: ...he Group DN field updates to the base DN entry The auto filled group DN entry may be replaced with customer specific settings If needed specify additional group DNs by clicking the green button and en...

Page 84: ...field enter a valid password The password corresponds to the Authentication DN d In the Password Retype field re enter the password 11 Click Save 13 7 3 Specify Lustre User Authentication via the NIS...

Page 85: ...s of the server s and the NIS Domain name Click the red button to remove NIS servers if required Figure 67 CSM NIS Additional Servers Screen 5 Click Save 13 7 4 Specify Lustre User Authentication via...

Page 86: ...AP Parameters IMPORTANT Carefully inspect any errors generated by CSM In many cases the errors reveal the exact problem through backend script output middleware issues etc Some of the parameters are l...

Page 87: ...thrhi 1408 size 256 1024 rszmax 128 1024 rszlo 4096 rszhi 4096 obdfilter survey 4 Set the time zone a Check if the current time and zone require setting MGMT0 date b List the available time zones MGMT...

Page 88: ...mount f testfs mount MGS is starting mount MGS is started mount testfs is started on sn11000n002 sn11000n003 mount testfs is started on sn11000n004 sn11000n005 mount All start commands for filesystem...

Page 89: ...mtest400 ssh lmtest402 lctl list_nids 172 18 1 3 o2ib root lmtest400 root lmtest400 mkdir tmp testmount root lmtest400 root lmtest400 mount t lustre 172 18 1 3 o2ib testfs tmp testmount root lmtest400...

Page 90: ...MGMT0 cscli unmount f testfs Perform the First Run Configuration 90...

Page 91: ...plains how to configure a list of users to be sent notifications and how to enable disable an individual user s notifications IMPORTANT Email notifications to the specified user list are sent only if...

Page 92: ...T0 cscli service_console configure email disable 6 Send a test email to verify that the email address has been properly added MGMT0 cscli service_console configure email send_test_email If the SMTP re...

Page 93: ...MGMT0 cscli service_console enable IMPORTANT ClusterStor L300 and L300N are shipped with service console capabilities enabled by default To disable the service console MGMT0 cscli service_console dis...

Page 94: ...he ClusterStor L300 and L300N systems to take advantage of the Lustre distributed namespace DNE feature Additional MMUs are configured in manufacturing Contact Cray Support for more information about...

Page 95: ...name 4 Verify the resources have been stopped On an even numbered OSS node run MGMT0 ssh nodename crm_mon 1r grep fsys Example output shows the nodes are stopped MGMT0 ssh sn11000n006 crm_mon 1r grep...

Page 96: ...03_md65 fsys ocf heartbeat XYMNTR Stopped 6 Power off the diskless nodes MGMT0 pdsh f 100 g diskless poweroff 7 Check the power off status of the diskless nodes MGMT0 pm q Repeat this step until all n...

Page 97: ...rmined at the factory using the SCD The SCD lists all of the internal cable connections for the system and includes cable label information and each cables source and designation device Labels denote...

Page 98: ...rhs Right hand side RAS port Scalable storage unit SSU ssu 5U84 storage enclosure w controllers Expansion storage unit ESU esu Expansion storage EBOD for SSU 1 or SSU 2 Embedded application controller...

Page 99: ...t on the right has a yellow cable SMU and MMU node references shown in the network interconnect lists are numbered using physical node numbers same as the front control panel See ClusterStor L300 and...

Page 100: ...connections 12Gbit s 16 3 2 ClusterStor L300 and L300N MMU Physical and Logical Port Numbering ClusterStor L300 and L300N MMU physical and logical port numbering Figure 70 MMU Physical and Logical Po...

Page 101: ...the same functions and storage resources as the MMU in a ClusterStor L300 and L300N base rack Figure 71 Cable AMMU SAS 1U Servers 16 3 4 ClusterStor L300 and L300N SSU Physical and Logical Port Number...

Page 102: ...nagement switch is reserved for connection to a storage rack Ports 5 7 and 19 are also available When all available ports are used for connecting storage racks the system may consist of from 1 to 5 ra...

Page 103: ...2 Eth SW1 22 Blue RH Side 25 SSU2 B 2 Eth SW1 20 Blue RH Side 26 SSU3 B 2 Eth SW1 18 Blue RH Side 27 SSU4 B 2 Eth SW1 23 Blue RH Side 28 RHS PDU 1 Eth SW1 12 Blue RH Side 29 SSU5 B 2 Eth SW1 21 Blue...

Page 104: ...ide 7 Unused Unused Blue LH Side 8 Unused Unused Blue LH Side 9 SSU6 A 2 Eth SW1 10 Blue LH Side 10 Optional Additional MMU A 2 Eth SW1 9 White LH Side 11 SSU0 A 1 Eth SW0 2 White LH Side 12 SSU1 A 1...

Page 105: ...0 22 White RH Side 38 SSU2 B 1 Eth SW0 20 White RH Side 39 SSU3 B 1 Eth SW0 18 White RH Side 40 SSU4 B 1 Eth SW0 23 White RH Side 41 SSU5 B 1 Eth SW0 21 White RH Side 42 Unused Unused Eth SW0 19 White...

Page 106: ...0 34 White LH Side 21 SMU A 1 Eth SW0 33 White LH Side 22 RAS KVM LHS Eth SW0 35 Blue RH Side 23 SSU0 B 2 Eth SW1 48 Blue RH Side 24 SSU1 B 2 Eth SW1 46 Blue RH Side 25 SSU2 B 2 Eth SW1 44 Blue RH Sid...

Page 107: ...ptional Additional MMU 48 port Lustre 40GbE Data Network Interconnect List Base Rack Mellanox 36 port Lustre 40GbE Data Network Interconnect List Storage Rack with Optional Additional MMU Mellanox 36...

Page 108: ...SSU2 A I IB SW0 6 White LH Side 4 SSU3 A I IB SW0 8 White LH Side 5 SSU4 A I IB SW0 10 White LH Side 6 SSU5 A I IB SW0 12 White LH Side 7 SSU6 A I IB SW0 14 White LH Side 8 Optional Additional MMU B...

Page 109: ...e 16 SMU B 1 OPA SW1 9 Table 16 Lustre Omni Path Data Network Interconnect List Storage Rack with Optional MMU 24 port Loom Label Component Side Port Switch Port White LH Side 1 SSU0 A 1 OPA SW0 1 Whi...

Page 110: ...SW1 16 Blue RH Side 10 SSU1 B 1 OPA SW1 15 Blue RH Side 11 SSU2 B 1 OPA SW1 14 Blue RH Side 12 SSU3 B 1 OPA SW1 13 Blue RH Side 13 SSU4 B 1 OPA SW1 12 Blue RH Side 14 SSU5 B 1 OPA SW1 11 Blue RH Side...

Page 111: ...LH Side 3 SSU2 A I 40G SW0 10 White LH Side 4 SSU3 A I 40G SW0 12 White LH Side 5 SSU4 A I 40G SW0 14 White LH Side 6 SSU5 A I 40G SW0 16 White LH Side 7 MMU AMMU A I I 40G SW0 18 White LH Side 8 SMU...

Page 112: ...al MMU B I 40G SW1 22 NOTE The Arista 40GbE switch has 36 ports however ports 1 through 4 are 10GbE and are not used in the ClusterStor L300 and L300N system Table 21 Lustre 40GbE Data Network Interco...

Page 113: ...SU3 A I 40G SW0 12 White LH Side 5 SSU4 A I 40G SW0 14 White LH Side 6 SSU5 A I 40G SW0 16 White LH Side 7 SSU6 A I 40G SW0 18 White LH Side 8 Optional Additional MMU A I 40G SW0 20 Blue RH Side 9 SSU...

Page 114: ...Data Network Interconnect List Storage Rack with Optional MMU Arista 32 port Loom Label Component Side Port Switch Port White LH Side 1 SSU0 A I 100G SW0 2 White LH Side 2 SSU1 A I 100G SW0 4 White LH...

Page 115: ...spares A single file system cluster may have from 0 to 8 MMUs installed An MMU includes the following cables and connections when installed in a storage rack occupying 2U of rack space 2 x power cable...

Page 116: ...4 36 2 4 8 10 12 6 16 3 8 Cable SSU ESU SAS Ports Expansion storage Units ESUs on ClusterStor L300 and L300N systems extend the storage capacity of an SSU with controllers Each SSU in the system is pa...

Page 117: ...3 2 1 B 4 3 2 1 C 4 3 2 1 A 4 3 2 1 B 4 3 2 1 C 4 3 2 1 1 1 2 2 1 1 3 4 5 2 2 1 1 2 2 1 1 3 4 5 2 2 EAC 0 EAC 1 SSU EBOD 0 EBOD 1 ESU 0 EBOD 0 EBOD 1 ESU 1 Figure 76 Cabling SSU ESU SAS 12Gb SSU 1 1...

Page 118: ...vides two Lustre metadata servers MDS one running on each of the 2 MMU EACs and operating together as an HA failover pair Normally each MDS owns and operates 1 of the 2 MDTs provided by the MMU during...

Page 119: ...the cabling tables in this publication for specific port assignments Figure 78 Cable AMMU Networks 1 2 3 4 2 1 3 1 2 3 4 2 1 3 A B 1 1 2 2 1 1 3 4 5 2 2 1 1 2 2 1 1 3 4 5 2 2 AMMU To local data networ...

Page 120: ...IC 5 pub0 external admin network Ethernet NIC 1 mgmt1 primary management network Figure 80 EAC for MMU and SSU Nodes Ethernet NIC 3 eth2 not used SAS port 1 SAS port 2 SAS port 3 SAS port 4 Reset Powe...

Page 121: ...anel 5U84 The front of the ClusterStor L300 and L300N 5U84 enclosure incorporates an operator s control panel OPS with five LEDs and the Unit ID LCD display The OPS panel provides a consolidated view...

Page 122: ...fy the component with a fault Logical Status LED Amber Indicates a change of status or fault from something other than the enclosure management system This may be from an internal or external source a...

Page 123: ...L300 and L300N 2U24 enclosure incorporates an operator s control panel OPS with three LEDs and the Unit ID LCD display The OPS panel provides a consolidated view of the operational status of the enclo...

Page 124: ...300 and L300N base rack configuration PX2 5965X3 V2 Figure 84 Base Rack Standard Power Distribution for PX2 5965X3 V2 ClusterStor L300 and L300N 16 5 2 Base Rack Inverted Power Distribution for PX2 59...

Page 125: ...Figure 85 Base Rack Inverted Power Distribution for PX2 5965X3 V2 ClusterStor L300 and L300N Reference 125...

Page 126: ...k Optional MMU Standard Power Distribution for PX2 5965X3 V2 ClusterStor L300 and L300N 16 5 4 Storage Rack Optional MMU Inverted Power Distribution for PX2 5965X3 V2 Cabling diagram for ClusterStor L...

Page 127: ...Figure 87 Storage Rack Optional MMU Inverted Power Distribution for PX2 5965X3 V2 ClusterStor L300 and L300N Reference 127...

Page 128: ...16 5 5 Base Rack Standard Power Distribution for PX2 5104X2 V2 Figure 88 Base Rack Standard Power Distribution for PX2 5104X2 V2 ClusterStor L300 and L300N Reference 128...

Page 129: ...16 5 6 Base Rack Inverted Power Distribution for PX2 5104X2 V2 Figure 89 Base Rack Inverted Power Distribution for PX2 5104X2 V2 ClusterStor L300 and L300N Reference 129...

Page 130: ...6 5 7 Storage Rack Optional MMU Standard Power Distribution for PX2 5104X2 V2 Figure 90 Storage Rack Optional MMU Standard Power Distribution for PX2 5104X2 V2 ClusterStor L300 and L300N Reference 130...

Page 131: ...16 5 8 Storage Rack Optional MMU Inverted Power Distribution for PX2 5104X2 V2 Figure 91 Storage Rack Optional MMU Inverted Power Distribution for PX2 5104X2 V2 Reference 131...

Page 132: ...16 5 9 Base Rack Standard Power Distribution for PX2 5100X2 V2 Figure 92 Base Rack Standard Power Distribution for PX2 5100X2 V2 ClusterStor L300 and L300N Reference 132...

Page 133: ...16 5 10 Base Rack Inverted Power Distribution for PX2 5100X2 V2 Figure 93 Base Rack Power Inverted Distribution for PX2 5100X2 V2 ClusterStor L300 and L300N Reference 133...

Page 134: ...16 5 11 Storage Rack Optional MMU Power Distribution for PX2 5100X2 V2 Figure 94 Storage Rack Optional MMU Power Distribution for PX2 5100X2 V2 ClusterStor L300 and L300N Reference 134...

Page 135: ...6 5 12 Storage Rack Optional MMU Inverted Power Distribution for PX2 5100X2 V2 Figure 95 Storage Rack Optional MMU Inverted Power Distribution for PX2 5100X2 V2 ClusterStor L300 and L300N Reference 13...

Page 136: ...16 5 13 Base Rack Standard Power Distribution for PX2 5551 N4V2 Figure 96 Base Rack Standard Power Distribution for PX2 5551 N4V2 Sonexion 3000 Reference 136...

Page 137: ...16 5 14 Base Rack Inverted Power Distribution for PX2 5551 N4V2 Figure 97 Base Rack Inverted Power Distribution for PX2 5551 N4V2 ClusterStor L300 and L300N Reference 137...

Page 138: ...6 5 15 Storage Rack Optional MMU Standard Power Distribution for PX2 5551 N4V2 Figure 98 Storage Rack Optional MMU Standard Power Distribution for PX2 5551 N4V2 ClusterStor L300 and L300N Reference 13...

Page 139: ...6 5 16 Storage Rack Optional MMU Inverted Power Distribution for PX2 5551 N4V2 Figure 99 Storage Rack Optional MMU Inverted Power Distribution for PX2 5551 N4V2 ClusterStor L300 and L300N Reference 13...

Page 140: ...Rack Optional AMMU Standard Power Distribution for PX2 5965X3 V2 ClusterStor L300 and L300N 16 5 18 Base Rack Optional AMMU Standard Power Distribution for PX2 5104X2 V2 Figure 101 Base Rack Optional...

Page 141: ...16 5 19 Base Rack Optional AMMU Standard Power Distribution for PX2 5100X2 V2 Figure 102 Base Rack Optional AMMU Standard Power Distribution for PX2 5100X2 V2 ClusterStor L300 and L300N Reference 141...

Page 142: ...16 5 20 Base Rack Optional AMMU Standard Power Distribution for PX2 5551 N4V2 Figure 103 Base Rack Optional AMMU Standard Power Distribution for PX2 5551 N4V2 ClusterStor L300 and L300N Reference 142...

Page 143: ...16 5 21 Base Rack Optional AMMU Inverted Power Distribution for PX2 5965X3 V2 Figure 104 Base Rack Optional AMMU Inverted Power Distribution for PX2 5965X3 V2 ClusterStor L300 and L300N Reference 143...

Page 144: ...Rack Optional AMMU Inverted Power Distribution for PX2 5104X2 V2 ClusterStor L300 and L300N 16 5 23 Base Rack Optional AMMU Inverted Power Distribution for PX2 5100X2 V2 Figure 106 Base Rack Optional...

Page 145: ...16 5 24 Base Rack Optional AMMU Inverted Power Distribution for PX2 5551 N4V2 Figure 107 Base Rack Optional AMMU Inverted Power Distribution for PX2 5551 N4V2 ClusterStor L300 and L300N Reference 145...