- The most important aspect of parallel programming
that is different from sequential
(i.e., using 1 thread) programming is
synchronization among the various parallel executing
threads that access a common resource
(in our case: shared variables) concurrently.
- Some access operations are
conflicting and
these access operations
cannot be executed simulateneously
Not all access operations are conflicting, however.
- Knowing which accesses are conflicting and
when to protect shared variables from concurrent access
is the key in
writing fast and correct
parallel programs.
- There are 2 access operations:
read and
write
- One common conflicting operations are:
- Reading by one thread and writing by another thread
- Writing by one thread and writing by another thread
- Recall that the global variables can be accessed by all threads
- Recall that threads executes simultaneously, meaning
the (assembler) instructions at multiple places in the program
are executed (by multiple CPUs)
- Recall that when 2 parallel threads try to
update a
shared (global) variable,
some updates can be lost:
Thread 1 on Thread 2 on Memory CPU 1 CPU 2 ============== =================== ================= N = 1234 Read N --> 1234 Add 1 --> 1235 Read N --> 1234 N = 1235 Write N Add 1 --> 1235 N = 1235 Write N
- This phenomenon can be
observed in the following
program:
#include <pthread.h> int N; <========= Shard variable N pthread_t tid[100]; // Each thread executes the following function: void *worker(void *arg) { int i, k, s; for (i = 0; i < 10000; i = i + 1) { N = N + 1; <======= Multiple threads access shared variable N } cout << "Added 10000 to N" << endl; return(NULL); /* Thread exits (dies) */ } /* ======================= MAIN ======================= */ int main(int argc, char *argv[]) { int i, num_threads; num_threads = atoi(argv[1]); /* ------ Create threads ------ */ for (i = 0; i < num_threads; i = i + 1) { if ( pthread_create(&tid[i], NULL, worker, NULL) ) { cout << "Cannot create thread" << endl; exit(1); } } N = 0; // Wait for all threads to terminate for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); cout << "N = " << N << endl << endl; exit(0); }
- Example Program:
(Demo above code)
- Prog file: click here
How to run the program:
- Right click on link and
save in a scratch directory
- To compile: g++ -pthread thread02.C
- To run: ./a.out
-
Obviously, there is a
need to ensure that the parallel thread
execution will
not produce results that is different
from the "normal" result.
- This problem when parallel thread execution can produce abnormal results is called (thread) synchronization problem
- Fact:
- There are many different kinds
of synchronization problems
- Each kind of problem may need to be solved with a special synchronization mechanism (that has certain properties)
- There are many different kinds
of synchronization problems
- Commonly used
synchronization methods (for commonly solved synchronization
problems):
- Mutex (or Mutually exclusive locks):
-
Used to control updates to shared variables
- Read/Write Locks
-
Also used to control access to shared variables, but they are
more "distinguishing" than mutex locks
- Binary and counting semaphores
-
Can be used to synchronize sharing in a more general manner
(not only updating shared variables)
Example: reader/writer problem
- Barriers
-
Can be used to synchronize event horizons
- make sure all involved threads reach a certain point
before proceeding
- Conditional variables (test-and-set operation)
-
Very low-level (hard to use), but very powerful ---
is the basis to implement all the above synchronization methods
in a shared memory programming
model.
- Mutex (or Mutually exclusive locks):
- Not every synchronization methods in CS theory are
implemented in PThreads
- Available synchronization methods in POSIX Threads:
- Mutex lock
- Read-write lock
- Semaphores
- Conditional variable --- too low level for CS355... skipped
- Mutex lock