Posix Thread (pthread) Programming

Introduction to Multi-threaded (Parallel) Programming

Multi-processors
- Recall that a Shared Memory multi-processor has the following (simplified) architecture:
  (NOTE: the inter-connection network is irrelevant to the programming and has been omitted).
  - The memory is shared by all CPUs
  - Each CPU fetches its own instruction and executes it

Processes and Threads

A process (in any operating system, e.g. UNIX) is created whenever a program is run

Property

Processes do not share memory space
Consequently: Different process cannot modify each other's variables stored in memory
This property is enforced by the Operating System (e.g., UNIX)

The reason for this is safety and reliability:

How do you feel if your program crashes because a bug in someone else's program updated one of the variables in your program ???

Different processes can be executed simultaneously in a multi-processor computer (computer with multiple CPUs)

Example:

Definition:

thread of execution

thread

location in the program

processor (CPU)

fetching and executing instructions

In other words: a thread makes progress in executing a program

Conclusion:

In the example above, each process has a single thread of execution
In the figure, the instructions at the arrow in both processes may be executed simultaneously
But within one process , there is only one thread of execution

Each process has at least one thread of execution.
Most (if not all) programs that you have seen probably have ONE SINGLE thread of execution
These are called SINGLE threaded programs
We will soon see MULTI threaded programs

Summary of Properties of processes:

Different processes do not share instruction (program) code
Different processes do not share variables
Different processes may share a minimum amount of variables that allow them to communicate with each other through the operating system kernel using a system call (i.e., with the aid of the Operating System)
- E.g., Parent and Child UNIX processes share file descriptors which allow them to create "pipes" to communicate with each other.

Multi-threaded programs
- Another look at a thread :
  There is at least one thread within every process (otherwise, the process will not run :-))
- "Traditional" processes are single threaded
  A single threaded process has exactly one thread...
- A multi-threaded program has more than one thread
- Example:
- If the computer is a multi-processor (multiple CPUs), the computer can simultaneously execute instructions at multiple places inside the same program !!!
- In the above picture, the computer is executing two instructions from process 1 and three instructions from process 2 simultaneously
  (This is assuming we have at least 5 CPUs !!!)

Example Multi-Threaded Program

Just to get you exicted on writting multi-threaded programs (later, I'll get you acquainted with the caveats of parallel program execution), here is a very simple example of a multi-threaded program

Example Multi-threaded program:

#include <pthread.h> pthread_t tid[100]; // Each thread executes the following function: void *worker(void *arg) { int i; i = pthread_self(); cout << "Hello, I am thread # " << i << endl; return(NULL); /* Thread exits (dies) */ } /* ======================= MAIN ======================= */ int main(int argc, char *argv[]) { int i, num_threads; num_threads = atoi(argv[1]); /* ------ Create threads ------ */ for (i = 0; i < num_threads; i = i + 1) { if ( pthread_create(&tid[i], NULL, worker, NULL) ) { cout << "Cannot create thread" << endl; exit(1); } } // Wait for all threads to terminate for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); exit(0); }

The above program makes use of two functions from the PThreads API

pthread_create(&tid[i], NULL, worker, NULL) will:

Create a new thread of execution (new beginning of execution)
The new thread executes the function worker()
Each thread has a unique thread ID
The thread ID of the new thread will be returned in the variable tid[i] (the first parameter of the function call)
Notice the address of tid[i] is passed, so the pthread_create(&tid[i], NULL, worker, NULL) can update the variable !

pthread_join(tid[i], NULL, worker, NULL) will:

Example Program: (Demo above code)
- Prog file: click here
Compile with: CC -mt thread01.C
Run the program several times and you will notice that the threads prints things out in different order...
We do not have control on when each thread is run....

Synchronization problems in multi-threaded programs

Computers execute assembler instructions, not high level language statements, such as "i = i + 1".
A high level language statements, such as "i = i + 1". results in a sequence of assembler instructions

Example:

High level programming language: i = i + 1; (Assuming variable i is stored at address 2000) Assembler code: 1. read value from memory address 2000 into the CPU 2. add 1 to this value 3. write value from CPU to memory address 2000

This detail will be important when we have multiple threads executing in a program.

Example: Asynchronous update of a variable

#include <pthread.h> int N; pthread_t tid[100]; // Each thread executes the following function: void *worker(void *arg) { int i, k, s; for (i = 0; i < 10000; i = i + 1) { N = N + 1; } cout << "Added 10000 to N" << endl; return(NULL); /* Thread exits (dies) */ } /* ======================= MAIN ======================= */ int main(int argc, char *argv[]) { int i, num_threads; num_threads = atoi(argv[1]); /* ------ Create threads ------ */ for (i = 0; i < num_threads; i = i + 1) { if ( pthread_create(&tid[i], NULL, worker, NULL) ) { cout << "Cannot create thread" << endl; exit(1); } } N = 0; // Wait for all threads to terminate for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); cout << "N = " << N << endl << endl; exit(0); }

Example Program: (Caveat of multi-threaded programs) --- click here
- Compile with: CC -mt thread02.C
- Run to see some interesting results....
The effect is explained below....

Properties of threads:

Threads are flows of control within the SAME program

As such, all threads share the same memory space !!!

All threads in a program share the global variables in the program
Threads do not share their local variables (unless a thread has convey the address of their local variable to another threads)

Threads can be in one of 2 states
Threads that are ready to run will be executed in an order determined by the thread scheduling system
(The programmer does not (nor should ) have control over the therad scheduling --- you could do some thead scheduling, but it is not worth the effort)
Different threads communicate with each other through global variables
The global variables are shared by all threads and can thus be used by threads to "deposit" values for one another --- through the writing and reading of global variables

Exaplanation of the asnchronous update problem in threads

The statement "N = N + 1" is executed by multiple assembler instructions:

Instruction 1: Read (get) the value from variable N in memory into the CPU
Instruction 2: Add 1 from this value
Instruction 3: Write (put) the result from the CPU in variable N in memory

Remember that in a multi-processor (such as "compute.mathcs.emory.edu"), all processors work simultaneously.
When multiple processors execute the statement "N = N + 1", incorrect outcome can result due to interspersed execution

Example: interspersed execution of 2 threads that result in a "missed update":

Thread 1 on Thread 2 on Memory CPU 1 CPU 2 ============== =================== ================= N = 1234 Read N --> 1234 Add 1 --> 1235 Read N --> 1234 N = 1235 Write N Add 1 --> 1235 N = 1235 Write N

The second value N = 1235 should have been N = 1236" if execution of "N = N + 1" was not interspersed !!!

Parallel programs: some lessons learned

Multi-threaded programs do NOT behaved like single threaded programs where only one thing happens at a time.
Due to concurrent execution unexpected behavior may arise
Parallel program are very hard to write and especially hard to DEBUG (because the behavior is not repeatable - different executions can produce different outcomes)
Always try to limited variable sharing in parallel programming - the more variables are shared, the more opportunities for problems (but variables sharing is inevitable in parallel programs)

Share and non-shared variables

When you write multi-threaded (parallel) programs, you have to understand the difference between SHARED and NON-SHARED variables
A thread executes some function...

The scoping rules that we learned in this webpage (click here) HAS NOT CHANGED

If a variable is accessible under single threaded programming, the variable is also accessible under multi-threaded programming.
If a variable is NOT accessible under single threaded programming, the variable is also NOT accessible under multi-threaded programming.

Therefore:

All global variables are shared (accessible) by all threads
All local variables (defined in functions) are not shared
(NOT accessible) by threads other than the one that executes that function

Example: non-shared local variable

void *worker(void *arg) { int N; int i; // Local i (non-shared) N = 0; for (i = 0; i < 10000; i = i + 1) { N = N + 1; } cout << "Added 10000 to N, N = " << N << endl << endl; return(NULL); /* Thread exits (dies) */ } /* ======================= MAIN ======================= */ int main(int argc, char *argv[]) { int i, num_threads; num_threads = ... /* ------ Create threads ------ */ for (i = 0; i < num_threads; i = i + 1) { if ( pthread_create(&tid[i], NULL, worker, NULL) ) { cout << "Cannot create thread" << endl; exit(1); } } for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); }

Example Program: (Demo above code)
- Prog file: click here
Compile and run:
All threads prints N = 10000
Because the variable i is non-shared , this variable kept an accurate count within the loop

Now consider: shared global variable

int i; // Global i (shared !!!) void *worker(void *arg) { int N; N = 0; for (i = 0; i < 10000; i = i + 1) { N = N + 1; } cout << "Added 10000 to N, N = " << N << endl << endl; return(NULL); /* Thread exits (dies) */ } /* ======================= MAIN ======================= */ int main(int argc, char *argv[]) { int i, num_threads; num_threads = ... /* ------ Create threads ------ */ for (i = 0; i < num_threads; i = i + 1) { if ( pthread_create(&tid[i], NULL, worker, NULL) ) { cout << "Cannot create thread" << endl; exit(1); } } for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); }

Example Program: (Demo above code)
- Prog file: click here
Compile and run:
This program prints unpredictable values for N by all threads...
The variable i is used and updated by all threads
It did NOT keep an accurate count within the loop.