- When I first encountered algorithms for Message-Passing MIMD computers,
it struck me as "weird"...
at least, much "weirder" than Shared-Memory MIMD algorithm.
- The reason is probably because all of us
"grew up" in a
Shared-memory programming environment;
and have never seen
message passing algorithm before.
- So here is a brief intro to a "weird" message passing
algorithm
(We don't have time to study parallel algorithms at length, this is ment to make you aware of a new type of thinking about solving problems).
- Given a sequence of values:
a0, a1, a2, ...., aN-1
- Problem:
- a0
- a0 + a1
- a0 + a1 + a2
- a0 + a1 + a2 + a3
- ...
- a0 + a1 + a2 + a3 + ... + aN-1
-
Find ALL the partial sums:
- Traditional Solution:
PartialSum[0] = a[0]; for (i = 1; i < N; i++) PartialSum[i] = PartialSum[i-1] + a[i];The run time complexity is N
- Overview of the Partial Sum Message Passing
Algorithm:
- Uses N processors
(think: transputer - a large number of processors available) - Processor i holds the number a[i] (only one number)
- Processors exchange information to compute the partial sums...
- Uses N processors
- Example of the Partial Sum Message Passing
Algorithm:
- I think the algorithm can best be explained by using
an example first.
- Consider N = 8
- The following diagrams shows how
8 messages passing processors can compute all the partial sums:
The goal is to compute ALL the partial sums:
- Processor 0: a0
- Processor 1: a0 + a1
- Processor 2: a0 + a1 + a2
- Processor 3: a0 + a1 + a2 + a3
- ...
- Processor 7: a0 + a1 + a2 + a3 + ... + a7
(I used array notation a[i] in the figures for ai)
- Step 1:
- processor i sends its data to
processor i + 1 ,
for i = 0, 1, ..., 6
- processor i receive some data and and add the received value to its data, for i = 1, 2, ..., 7
What happens is the following:
Resulting state:
- processor i sends its data to
processor i + 1 ,
for i = 0, 1, ..., 6
- Step 2:
- processor i sends its data to
processor i + 2 ,
for i = 0, 1, ..., 5
- processor i receive some data and and add the received value to its data, for i = 2, 3, ..., 7
What happens is the following:
Resulting state:
- processor i sends its data to
processor i + 2 ,
for i = 0, 1, ..., 5
- Step 3: (final step)
- processor i sends its data to
processor i + 4 ,
for i = 0, 1, ..., 3
- processor i receive some data and and add the received value to its data, for i = 4, 5, ..., 7
What happens is the following:
Resulting state:
- Processor 0: a0
- Processor 1: a0 + a1
- Processor 2: a0 + a1 + a2
- Processor 3: a0 + a1 + a2 + a3
- ...
- Processor 7: a0 + a1 + a2 + a3 + ... + a7
- processor i sends its data to
processor i + 4 ,
for i = 0, 1, ..., 3
OK, how can we formulate this as an compuetr algorithm ?
- I think the algorithm can best be explained by using
an example first.
- Message Passing Algorithm for processor i:
// k = loop index // k runs: 1, 2, 4, 8, .... etc !!! for (k = 1; k < N; k = 2*k) { // Check if I am a processor that needs to send data if ( i + k < N ) send a[i] to processor (i+k); // Check if I am a processor that needs to receive data if ( i >= k ) { recevive message x; a[i] = a[i] + x; } } -
Run time complexity:
- # iterations = O(log(N)) !!!
- This kind of message passing algorithms will only work
when you have a
massive amount of processor
and a
super-high speed interconnection network
(because you are sending an
insane number of messages
and if you have a slow network, the running time will
be hampered by the message transmissions).
- Most message-passing MIMD's available today consists of
a relatvely small number (16 - a few hundred) of powerful
computers interconnected by highspeed network.
- The most commonly used message-passing programming paradigm is
"job-shop" where one processor (master) doles out pieces of work
(tasks) to other processors (slaves).
- The Message Passing Interface (MPI)
is ideal for this kind of message passing programming paradigm:
- Process 0 in MPI is the master process
- All other processes are slave processes