- Executing instructions on a computer require 2 things:
- Instructions
- Operands
- Parallel computing
or parallel execution involves
performing multiple computations in parallel
(i.e., at the same time, simultaneously).
- This can happen in
two different ways:
- A single operation (= instruction)
is applied to
multiple (different) operands (= data)
This form of parallelism is known as:
SIMD: Single Instruction (stream) - Multiple Data (stream) - Multiple operations (= instructions)
are applied to
multiple (different) operands (= data)
This form of parallelism is known as:
MIMD: Multiple Instruction (stream) - Multiple Data (stream)
- A single operation (= instruction)
is applied to
multiple (different) operands (= data)
- From the basic building blocks of computing
(it requires instructions and operands),
the following taxonomy of computers can be formulated:
-
This is the most common computer produced - workstations, PC, etc
In fact, this is the type of computer that we have been studying all along...
- Schematically:
- One CPU present in computer
- CPU fetches 1 instruction at a time
- Instruction operates on 1 data item at a time
- The machines executes
ONE instruction
at a time
(single instruction stream) but on
MULTIPLE (different) data items
(multiple data streams)
NOTE: obviously, it's kinda stupid to execute the same instruction on the same data - you will waste CPU time getting the same result multiple times.
- One CPU present in computer
- The CPU has:
- one Instruction Register, but
- multiple ALUs (and multiple system busses to get multiple operands simultaneously)
- Main memory are divided into multiple banks that are accessed
independently
- The CPU also has multiple data paths that connect to the different
memory banks allowing it to fetch multiple operands
simultaneously
- The CPU fetches 1 instruction at a time
- Some instruction can operates on multiple data items
at a time - these instructions are known as
vector instructions
-
This type of computer is known as a
vector computer.
It is commonly used in speeding up numerical applications that
use many vector operations such as
matrix-matrix and matrix-vector
multiplications.
- SIMD computers were the first systems to be implemented
with a massive amount of processors, and were among
the first systems to provide computational power above the
Giga FLOP range.
- One CPU present in computer
- SIMD parallel computers executes
MULTIPLE instruction
at the same time
(multiple instruction stream).
The (different) instruction will usually operate of different operands (data) and hence it naturally require MULTIPLE data items (multiple data streams)
- There are
two common varieties
of
MIMD computers:
- Shared Memory MIMD
- Non-Shared (private) Memory MIMD (A.K.A.: message passing multi-processor)
- Shared Memory MIMD
- In a
Shared-Memory MIMD
computer, all memory locations are
accessible by all processors (CPUs)
- Schematically:
- Multiple CPUs present in computer
- Each CPU fetches 1 instruction at a time and each CPU may fatch
a different instruction; thus, there are
multiple instruction streams
- The memory must be equiped with
multiple access points
(i.e., organized into multiple
INDEPENDENT
memory banks)
that allows multiple CPU to access the different items in memory
simultaneously
(barring conflict of access, i.e., different CPU want to access
the same memory bank).
- Instruction operates on 1 data item at a time, because
each CPU fetches its own operand, there will be multiple
data streams
-
This type of computer is a
multi-processor computer
and they are quite common today.
Most workstations and high-end PC are multi-processor based.
- Multiple CPUs present in computer
- NOTE
- A common problem in
Shared-Memroy MIMD
is the problem of
asynchronous updates
CPU 1 CPU 2 ------------ ------------ mov x, D0 mov x, D0 add #1, D0 add #1, D0 mov D0, x mov D0, xThe interleaved execution will result in error (x should be increased by 2, but at the end of the above execition, x is increased by 1)
- Shared-Memroy MIMD
computers always provide
a synchronization methods
to allow programmers to write statements in their programs
to
synchronize updates to SHARED variables
A commonly used Shared-Memory programming API is posix-threads or pthreads for short...
- Here's the Wikipedia page on Posix Threads: click here
- A common problem in
Shared-Memroy MIMD
is the problem of
asynchronous updates
- In a Message-passing MIMD,
a number of
SISD
computers are inter-connected by a
communication network:
- Each CPU has its own
private memory
In orther words: Memory is not shared among the different CPUs.
- Naturally, in order to solve a problem together,
you need
information
-
So from time to time, programs running on different CPUs
in a
Message-passing MIMD
will
EXCHANGE information
each other.
- These information are exhanged using
message
- Hence the name Message-passing MIMD
- Each CPU has its own
private memory
- NOTE
- The "Network Switch" used to interconnect the different
SISD computers is usually a
dedicated high speed network switching fabric
- However, it is quite common to use
cheaper interconnection networks
- e.g., Highspeed Ethernet
A MIMD multi-processor using "cheaper interconnection networks" is typically known as networked computers
- Message-passing MIMD
computers always provide
a message-passing API (application programming interface)
to allow programmers to write statements in their programs
to exchange messages
A commonly used Message-passing API is MPI (Message Passing Interface):
- The standardization organization of MPI is here: click here
- The "Network Switch" used to interconnect the different
SISD computers is usually a
dedicated high speed network switching fabric
- Before we can talk about SIMD and MIMD computers,
we need to look at a more basic problem.
- Since SIMD/MIMD computers require
MULTIPLE operands/instructions
to be used inside the CPU (that's still the place where
the execution takes place),
we need to solve the problem of
getting multiple data/operands
to the CPU.
- As an integral part of parallel computers, we will also discuss interconnection networks for each type of parallel computer.