- Recall a
Message-Passing MIMD
is connected as follows:
- CPUs
in a Message-Passing MIMD
communicate with each other by
sending computer message .
- The
computer message
contains:
- Sender's ID
- Receiver's ID
- Message body
The message is usually variable size (not like memory data in Shared-Memory MIMD which is fixed sized)
In fact, the date items transmitted through the interconnection network are computer messages exchanged on a computer network (but the distances traveled is usually very short (in a room) in the Message-Passing MIMD.
Because the size of the messages are variable, switching networks used in the Shared Memory MIMD are not well-suited for Message-Passing MIMD, because the control of the timing in the switching fabric is very complex.
- Every processor in
a
Message-Passing MIMD
is also a
router
- A router is a processor (program + CPU) than can forward a message to its (final) destination.
- Although routing
in general is a complex problem,
it is very trivial in a
Message-Passing MIMD because the
interconection network is
very simple/regular
- We will look at some commonly used interconection networks for Message-Passing MIMDs
- Fully connected:
- Routing:
- No routing is needed (direct link)
- Advantages:
- Fast communication (messages travels only one hop)
- Disvantages:
- High complexity/cost : For N nodes, you need N(N-1)/2 links
- This is only feasible for a small number of processors
- Routing:
- Ring:
- Routing:
- In a ring of N nodes, each node is assigned an ID from 0 to N-1
- A node with ID = u will route a message to its right neighbor if destination ID v is between u+1 (mod N), u+2 (mod N), ... u+N/2 (mod N)
- Otherwise, it routes to its left
- Advantages:
- Very very cheap to make... (each nodes has 2 connections)
- Disvantages:
- Very very long routes for messages.
- Worst case: (N-1)/2 (half the circle)
- Routing:
- Mesh (Grid):
- Routing:
- Each node is assigned an ID consisting of a
(row index,column index) pair
- Route message
first along the row ,
then along a column
Example: o
- Message from (1,1) sent to (3,3)
- Route: (1,1) -> (2,1) -> ( 3 ,1) -> (3,2) -> (3, 3 )
- Each node is assigned an ID consisting of a
(row index,column index) pair
- Advantages:
- Very cheap to make... (each nodes has <= 4 connections)
- Disvantages:
- Very long routes for messages.
- Worst case: 2*(Sqrt(N)-1) (corner to corner)
- Routing:
- Hypercube:
- Connection property:
- A node in a k dimensional hypercube is assigned
an ID consisting of
k bits
- The node xxxx and yyyy are neighbors
(connected by a direct link) if and only if
their IDs differs
in
one single bit
(in any position)
- E.g.: nodes 000 are connected to: 001, 010 and 100
- A node in a k dimensional hypercube is assigned
an ID consisting of
k bits
- Routing:
- When a node with ID xxxx need to route
a message destined for node
yyyy, it compares the strings:
- xxxx
- yyyy
and find the first bit in which the pattern differ.
- The message to yyyy is route to the neighbor node of xxxx that has the differing bit flipped.
Example:
- Message: from node 000 to node 101
- The message will be sent as follows:
- 000 ----> 100 (101 and 000 differ in first bit)
- 100 ----> 101 (101 and 100 differ in third bit)
- When a node with ID xxxx need to route
a message destined for node
yyyy, it compares the strings:
- Advantages:
- Cheap to make... (each nodes has k connections in a k-dimensional hyber cube)
- Disvantages:
- Long routes (but pretty reasonable length)
- Worst case: log(N) (0000 ---> 1111)
1D, 2D and 3D Hypercubes 4D Hypercube 
- Connection property:
- The message-passing MIMD was thought to be a means to
build faster and faster computers by dividing a task
into smaller tasks and let each computer handle a subtask
- One of the more revolutionary design in
message-passing MIMD was the
transputor
- From Wikipedia:
-
The transputer was a pioneering concurrent computing
microprocessor design of the 1980s from INMOS,
a British semiconductor company based in Bristol.
The transputer (transistor computer) was the first general purpose microprocessor designed specifically to be used in parallel computing systems. The goal was to produce a family of chips ranging in power and cost that could be wired together to form a complete computer.
The name was selected to indicate the role the individual transputers would play: numbers of them would be used as basic building blocks, just as transistors had earlier.
Originally the plan was to make the transputer cost only a few dollars per unit.
- In other words:
People envisioned to make a very inexpensive complete computer (very limited instruction set and a small memory - the first transputer only has 4K memory) A computer would consists of hundred of thousands of transputers - interconnected with one another
- Modern day Message-passing MIMD:
- More powerful CPUs with a lot of memory
- Basically a full computer (like a high-end PC)
- Computers interconnected with highspeed networks, such as: InfiniBand and Gigibit Ethernet
- More powerful CPUs with a lot of memory
- Message-passing MIMDs are generally cheaper than Shared-memory
MIMDs
The price tag of Message-passing MIMDs depends a lot on how much you want to pay for the interconnection network - which can be cheap (like Ethernet) or quick expensive (InfiniBand).
- InfiniBand -
click here
So far InfiniBand has become the de-facto interconnect of choice for high performance computing, and its adoption as seen in the TOP500 supercomputers list is faster than Ethernet. (Wikipedia) The InfiniBand specification defines the raw bandwidth of the base 1x connection at 2.5Gb per second. It then specifies two additional bandwidths, referred to as 4x and 12x, as multipliers of the base link rate. (4x rates - 10 Gbps are already available) (OreillyNet)
New switch systems based on the InfiniScale IV architecture supporting up to 40Gb/s per port and 120Gb/s for inter-switch links are expected to be available in the latter part of 2008 from several leading server and infrastructure system OEMs. ( click here )
- Gigabit Ethernet (cheaper alternative) - click here