Connecting Message Passing Multi-Processor

Message passing MIMD: System Organization
- System organization:
- Each computer has a unique ID

Messages exchanged by the multiple processors

Besides the memory not being shared among the multi-processors, another difference between a Shared-Memory MIMD and Message-Passing MIMD multi-processor is the information sent through the interconnection network.

Shared-memory MIMD:

Information sent in a Shared-Memory MIMD are data items between one CPU and one memory module.
Two CPUs "communicates" with each other by writing/reading a value to/from a (agreed) location in the SHARED memory
Therefore, the date items transmitted through the interconnection network is small and fixed sized

In contrast, Message-passing MIMD:

Information sent in a Message-Passing MIMD are data items between one CPU and another CPU.
Two computers in a Message-Passing MIMD communicates with each other by sending a message that contains:
- Sender's ID (ID of the sending computer)
- Receiver's ID (ID of the receiving computer)
- Message
The message sent is usually large and of variable size
(Because there is a lot of overhead to send a single message

In fact, the date items transmitted through the interconnection network is same as messages exchanged on a computer network (but the distances traveled is usually smaller in the MIMD.

Because the size of the messages are variable, switching networks used in the Shared Memory MIMD are not well-suited for Message-Passing MIMD, because the control of the timing in the switching fabric is very complex.

Communication Processing in the Message-Passing multiple processors
- In addition to performing computation, every processor in a Message-Passing MIMD is also a router
- A router is a processor (program + CPU) than can forward a message to its (final) destination.
- Although routing in general is a complex problem, it is very trivial in a Message-Passing MIMD because the interconection network is very simple/regular
- We will look at some commonly used interconection networks for Message-Passing MIMDs
Interconnection Networks for Message-Passing MIMD.
- Fully connected:
  - Routing:
    - No routing is needed (direct link)
  - Advantages:
    - Fast communication (messages travels only one hop)
  - Disvantages:
    - For N nodes, you need N(N-1)/2 links
    - This is only feasible for a small number of processors
- Ring:
  - Routing:
    - In a ring of N nodes, each node is assigned an ID from 0 to N-1
    - A node with ID = u will route a message to its right neighbor if destination ID v is between u+1 (mod N), u+2 (mod N), ... u+N/2 (mod N)
    - It routes to its left otherwise.
  - Advantages:
    - Very very cheap to make... (each nodes has 2 connections)
  - Disvantages:
    - Very very long routes for messages.
    - Worst case: (N-1)/2 (half the circle)
- Mesh (Grid):
  - Routing:
    - Each node is assigned an ID consisting of a (row index,column index) pair
    - Route message along the row first, then along a column
      Example: o
      - (1,2) -> (3,1)
      - Route: (1,2) -> (2,2) -> (3,2) -> (3,1)
  - Advantages:
    - Very cheap to make... (each nodes has <= 4 connections)
  - Disvantages:
    - Very long routes for messages.
    - Worst case: 2*(Sqrt(N)-1) (corner to corner)
- Hypercube:
  - Connection property:
    - A node in a k dimensional hypercube is assigned an ID consisting of k bits
    - The node xxxx and yyyy are neighbors (connected by a direct link) if and only if their IDs differs in one single bit (in any position)
    - E.g.: nodes 000 are connected to: 001, 010 and 100
  - Routing:
    - When a node with ID xxxx need to route a message destined for node yyyy, it compares the strings:
      - xxxx
      - yyyy
      and find the first bit in which the pattern differ.
    - The message to yyyy is route to the neighbor node of xxxx that has the differing bit flipped.
    Example:
    - Node 000 wants to route a message to node 101
    - The message will be sent as follows:
      1. 000 ----> 100 (101 and 000 differ in first bit)
      2. 100 ----> 101 (101 and 100 differ in third bit)
- Advantages:
  - Cheap to make... (each nodes has k connections in a k-dimensional hyber cube)
- Disvantages:
  - Not very short routes (but pretty reasonable length)
  - Worst case: log(N) (0000 ---> 1111)

A (abadoned) vision of parellel processing

The message-passing MIMD was thought to be a means to build faster and faster computers by dividing a task into smaller tasks and let each computer handle a subtask
One of the more revolutionary design in message-passing MIMD was the transputor

From Wikipedia:

The transputer (transistor computer) was the first general purpose microprocessor designed specifically to be used in parallel computing systems. The goal was to produce a family of chips ranging in power and cost that could be wired together to form a complete computer.

The name was selected to indicate the role the individual transputers would play: numbers of them would be used as basic building blocks, just as transistors had earlier.

Originally the plan was to make the transputer cost only a few dollars per unit.

In other words:

People envisioned to make a very inexpensive complete computer (very limited instruction set and a small memory - the first transputer only has 4K memory)

A computer would consists of hundred of thousands of transputers - interconnected with one another

Modern day Message-passing MIMD:
- More powerful CPUs with a lot of memory
- Basically a full computer (like a high-end PC)
- Computers interconnected with highspeed networks, such as: InfiniBand and Gigibit Ethernet
Message-passing MIMDs are generally cheaper than Shared-memory MIMDs
The price tag of Message-passing MIMDs depends a lot on how much you want to pay for the interconnection network - which can be cheap (like Ethernet) or quick expensive (InfiniBand).

Popular Interconnection Networks for Message-Passing MIMD

InfiniBand - click here

So far InfiniBand has become the de-facto interconnect of choice for high performance computing, and its adoption as seen in the TOP500 supercomputers list is faster than Ethernet. (Wikipedia)

The InfiniBand specification defines the raw bandwidth of the base 1x connection at 2.5Gb per second. It then specifies two additional bandwidths, referred to as 4x and 12x, as multipliers of the base link rate. (4x rates - 10 Gbps are already available) (OreillyNet)

New switch systems based on the InfiniScale IV architecture supporting up to 40Gb/s per port and 120Gb/s for inter-switch links are expected to be available in the latter part of 2008 from several leading server and infrastructure system OEMs. ( click here )

Gigabit Ethernet (cheaper alternative) - click here