Myricom Myrinet-2000, Manual

Page 1: ...ack mounting holes conform to the EIA 310 standard for 19 inch racks Line cards and the fan tray may be hot swapped inserted or removed with the power on However Insert and remove line cards gently according to the instructions that start on page 4 Do not operate a switch for extended periods with missing line cards Use M3 BLANK panels to fill in any empty line card slots The front of the enclosur...

Page 2: ... Myrinet 2000 switch products The M3 E16 M3 E32 M3 E64 and M3 E128 were certified in accordance with the IEC System for Conformity Testing and Certification of Electrical Equipment IECEE CB Scheme In addition TUV certified these products to the US and Canadian safety standards Electromagnetic Compatibility EMC Subject to the limitations listed below Myrinet 2000 switch products based on the M3 E16...

Page 3: ... cables and connectors exhibit exceptionally high reliability Myrinet Fiber cables may be up to 200m in length Myrinet Fiber components and cables operate within Class A limits for the emission of electromagnetic interference EMI Myrinet Fiber links carry packet data at a 2 2 Gb s data rate on industry standard 50 125 multimode fiber pair cables with LC connectorization Myricom ships Fiber compone...

Page 4: ... include a locking mechanism These locking springs should both be depressed when inserting and when removing a SAN cable end connector SAN links are for in cabinet applications only Switch configurations with any SAN line cards M3 SW16 8M or M3 SW16 4DM exceed Class A limits for EMI and should be used only within a shielding enclosure Also it is best to restrict the use of SAN cables to within an ...

Page 5: ... the middle like this You ll notice from the illuminated LEDs that this switch is powered That s OK and why it s called hot swapping To remove the line card first press the red tabs to unlock its handles Then turn both handles outward together You should be able to see and feel the lever action as the front panel of the line card is ejected toward you It is normal that the ejection is a little sti...

Page 6: ...arefully into the card guides The signal pins and ground blades on the high density connectors on the line card align themselves by means of alignment grounding posts on the backplane Conical depressions on the line card connectors center themselves on the posts to align the connectors with great precision Nevertheless it is best to insert the line cards slowly and gently When you use the lever ha...

Page 7: ... be removed by loosening the two locking screws and pulling the fan tray out with the handle When a fan try is removed it should be replaced within approximately one minute or else line cards may power themselves off in response to an over temperature condition ...

Page 8: ... 1 Introduction 9 2 The Family of Enclosures 11 3 Other Features of the Enclosures and Line Cards 12 4 Port Line Cards 14 5 Monitoring Line Card 17 6 Configurations up to 128 Hosts 19 7 Topology Concepts 21 8 Clos Networks 22 9 Clos Networks for more than 128 Hosts 23 ...

Page 9: ... hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts Clos spreader network Ports to up to 128 hosts line cards The network pictured above provides routes from any host to any other host There is a unique shortest route between hosts connected to the same XBar16 The eight minimal routes between hosts connected to different XBar16s traverse t...

Page 10: ...part of the backplane A port line card with connectors to the backplane on the back and connectors to external links on the front panel The topology is a full bisection Clos network with any combination of port line cards inserted or omitted For example you could plug 10 port line cards into the M3 E128 enclosure to support up to 80 hosts Ten of the ports on each of the 8 spine XBar16s would be us...

Page 11: ...switches for the spine An XBar16 can serve the purpose of two 8 port switches with some additional communication provided free Thus a suitable topology for a 64 host Clos network of 16 port switches is 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 8 hosts 2 links each in which each of the thicker lines represents two links If we wanted to get theoretical we d point out that the paired li...

Page 12: ...e powered off Other features of the enclosure products include a slot for an optional line card for monitoring and control of the switch This monitoring line card product code M3 M is the same size as the port line cards but has different backplane connectors such that it can be inserted only into the top line card slot The M3 M includes a big microcontroller that communicates externally via eithe...

Page 13: ... flexibility manufacturing economies and most importantly easy expansion or reconfiguration of switches and switch networks already installed and the ability to repair switches by replacing just a line card or the fan tray Although the MTBF of Myrinet components is exceptionally high field failures do occur The monitoring line card allows nearly any fault to be identified unambiguously A line card...

Page 14: ...ne cards or 4 legacy LAN ports M3 SW12 4L line card depending upon the Physical level conversion circuits The port numbers 8 15 8 11 for the M3 SW12 4L line card appear on the front panel silkscreen and correspond to the port numbering of the XBar16 Inasmuch as the native Physical level PHY of the XBar16 ports is SAN there are no conversion circuits for the M3 SW16 8M or the M3 SW16 4DM The only d...

Page 15: ...ch port The port LEDs become green when a port is connected through a cable to an active port and will blink when traffic is flowing The M3 SW16 8S M3 SW16 8F and M3 SW12 4L also include a Status LED for the line card When a line card is inserted the Status LED shows green within 0 5 seconds if the line card has passed its self test and all voltages temperatures and internal status bits are at nom...

Page 16: ...ny indicator LEDs The M3 SPINE 8F and M3 SPINE 8S port line cards are similar to their M3 SW16 counterparts except for the absence of the XBar16 µC Clocks Backplane Interface 0 1 2 3 4 5 6 7 Front panel Ports Serial link Sense Control Power hot swap circuits Physical Level Conversion Circuits dual 12V 0 1 2 3 4 5 6 7 The front panel ports are labeled 0 7 corresponding to the ports of the backplane...

Page 17: ...nclosure via 26 separate and independent serial links 16 serial links to µCs on port line cards 8 serial links to µCs associated with backplane XBars 1 serial link to the fan monitoring µC and 1 serial link to its own small µC The front panel includes the usual Status LED green for operating and yellow for fault There are two LEDs for the 12V A and B internal power buses Each of the RJ 45 10 Base ...

Page 18: ...page The following collection of screen snapshots from a web browser will give you some impression of the capabilities of the monitoring and of the web browser interface Home page of an M3 E32 Beginning of page for first port line card End of page for first port line card A part of the fan monitor page ...

Page 19: ...odular assembly M3F SW16M Fiber switch M3 E16 M3 M M3 SPINE 8F M3 SW16 8F M3S SW16M Serial switch M3 E16 M3 M M3 SPINE 8S M3 SW16 8S M3M SW16M SAN switch M3 E16 M3 M 2 M3 SW16 8M There is no M3 SPINE 8M product due to signal integrity limitations on carrying SAN signals across backplanes and then across line cards to cables Hence the 16 port all SAN switch has a redundant layer of switching see pa...

Page 20: ... space is at a premium So you order the following components for the switch network Qty Modular component 1 M3 E32 enclosure 1 M3 M monitoring card highly recommended or M3 BLANK 3 M3 SW16 8M port line card with 8 SAN ports 1 M3 SW16 8F port line card with 8 Fiber ports Example 2 You plan to build a 64 host cluster that you expect to expand soon to 128 hosts and perhaps to an even larger size even...

Page 21: ...mum bisection suppose that we wished to connect 8 hosts and the largest crossbar switches available had 5 ports One way to connect the 8 hosts would be 4 hosts 4 hosts bisecting cut 1 link The bisecting cut that exhibits the minimum bisection is evident If the 4 hosts on the left were sending messages to the 4 hosts on the right and vice versa the total traffic handling capacity of the network wou...

Page 22: ...s the path formation latency of Myrinet switches is much smaller than software latency the maximal or average diameter measured in switches traversed is a relatively unimportant metric of the network topology 8 Clos Networks Clos networks are named for Charles Clos who introduced them in a paper titled A Study of Non Blocking Switching Networks published in the Bell System Technical Journal in Mar...

Page 23: ...k at this necessity is to consider a bisecting cut that crosses the 8 host links There is another adjacent bisecting cut across the 8 upper links Thus any structure used as a building block for larger networks must have this property of preserving link count and data rate The proof that the 128 host Clos network of 16 port switches is a rearrangeable network a stronger statement than that it exhib...

Page 24: ...rk preserves 64 link data rate between the hosts and the deeper parts of the network fabric so that if these 64 hosts are communicating entirely with hosts outside their own group there are enough links to carry all the traffic This network is referred to as a Clos64 64 and it can be implemented with an M3 E128 enclosure with 8 M3 SW16 line cards and 8 M3 SPINE line cards In analogy to the structu...

Page 25: ... enclosures and 40 M3 SPINE line cards If you refer to the port layout and numbering linked from http www myri com myrinet m3switch guide you will see that each M3 SPINE line card in an M3 E128 connects to the same port of all 8 backplane switches Thus if an M3 E128 is populated with 15 M3 SPINE line cards it provides 8 15 port switches which can be used as 24 5 port switches Two such units and an...