© 2001, Myricom, Inc.
-- 25 --
Revision: 27 August 2001
Note that the diameter of this network is 5 switches. A packet may traverse two switches in its
upward path through a 64, one spine switch, and two switches in its downward path
through a 64.
For the spreader network, the 64 upper ports of each 64 should connect to one port on
each of the 64 16-port spine switches. Any cabling plan that satisfies this description is correct;
however, systematic cabling plans can be found linked from
http://www.myri.com/myrinet/m3switch/guide/
.
The most compact and economical way to implement the spine is with 8 M3-E128 enclosures,
each with 16 M3-SPINE line cards, providing 8 16-port switches in each M3-E128.
If you recall the way in which a networks smaller than the 128-host Clos network are constructed
(page 11) by using a 16-port switch as if it were 2 8-port switches, or 4 4-port switches, this
same technique can be used with the 64 to build networks for 512 or for 256 hosts.
For example, a network for 256 hosts would require 4 64 units and 64 4-port switches.
Instead of 64 4-port switches, we could instead use 16 16-port spine switches. The 64 upper
ports of each 64 should connect to four ports on each of the 16 16-port spine switches.
This basic scheme is suitable for any multiple of 64 hosts from 192 to 1024 hosts. For example,
a 320-host network requires 5 of the 64 units. In the abstract, the spine should consist of
64 5-port switches. The minimal-cost implementation of the spine requires 3 M3-E128
enclosures and 40 M3-SPINE line cards. If you refer to the port layout and numbering linked
from
http://www.myri.com/myrinet/m3switch/guide/
, you will see that each M3-SPINE line
card in an M3-E128 connects to the same port of all 8 backplane switches. Thus, if an M3-E128
is populated with 15 M3-SPINE line cards, it provides 8 15-port switches, which can be used as
24 5-port switches. Two such units, and another with 10 M3-SPINE line cards for 8 10-port
switches used as 16 5-port switches, provide a total of 64 5-port switches. This optimized
implementation would need to be recabled to expand the network to 384, 448, or 512 hosts,
whereas the more symmetrical implementation of the spine with 4 M3-E128 enclosures, each
with 10 M3-SPINE line cards, can be expanded without recabling.
The Clos networks for 16 <
N
≤
128 are particularly efficient because the switch-to-switch links
are all contained within the “network in a box.” The discontinuity in the switch-network cost
from 16 <
N
≤
128 to
N
> 128 is due in part to the diameter of the Clos network increasing from
3 to 5, and in part to the need for a spreader network implemented with cables. A Clos network
for
N
= 8192 can be implemented with 128 of the 64 units connected to a spine of 64
Clos128 networks. This network has a diameter of 7, but still requires only one external spreader
network built with cables between the 64 units and the Clos128 units. Thus, the
cost/host of Clos networks in the 1024 <
N
≤
8192 range is similar to the cost/host of Clos
networks in the 128 <
N
≤
1024 range, and the optimization techniques are the same.
For additional information about Myrinet Clos networks for 128 <
N
≤
8192, please see the
supplemental information at
http://www.myri.com/myrinet/m3switch/guide/
.
