- A common
parallel execution
(especially in
numerical analysis
is as follows:
main() { set up for parallel execution (sequential) parallel computational section gather result and set up for next parallel execution (sequential) parallel computational section and so on... }
- The follow program fragment depicts the
common motif
of parallel programs:
void *worker (void *arg) { .... } int main(int argc, char *argv[]) { // SEQUENTIAL Section .... Prepare problem (setup shared variables) .... // PARALLEL Section // Start workers for (i = 0; i < NUM_PROCESSORS; i = i + 1) { param[i] = ....; pthread_create(&tid[i], &attr, worker, ¶m[i]) } // Wait for all workers to finish for (i = 0; i < NUM_PROCESSORS; i = i + 1) pthread_join(tid[i], NULL); // SEQUENTIAL Section .... Gather result (post-processing) and set up variables for next parallel section and so on.... }
- A sequential program to find the minimum in array:
#define N_Items 1000000 double x[N_Items]; // Input array double min; int i; min = x[0]; for ( i = 1; i < N_Items; i = i + 1 ) if ( x[i] < min ) min = x[i];
- Example Program:
(Demo above code)
- Prog file: click here
- When writing a parallel program, you
must divide the work
into
(preferrably) non-overlapping
subtasks
- Division of labor:
- There are usually
many different ways
to
divide a task into subtasks
- Some way to divide the work
may lead to
easier coding
- Other way to divide the work
may lead to
faster execution
- And sometimes, it is not possible to divide the task into non-overlapping tasks and you may have to repeat some portion by multiple threads - that is the necessary evil in parallel programming...
- There are usually
many different ways
to
divide a task into subtasks
- We will compare two different parallel programs to "find the minimum value" in an array.
- Division of labor scheme:
(For simplicity of discussion, I used 2 threads)
- Split the array into 2 (approximate) equal halfs
- Thread 1 finds the minimum in the first half of the array
- Thread 2 finds the minimum in the second half of the array
- Main thread waits for the results and find the actual minimum.
- Split the array into 2 (approximate) equal halfs
- Graphically:
start[0] start[1] | | | values handled by | values handled by V thread 0 V thread 1 |<--------------------->|<--------------------->| | | | | V V min[0] min[1] \ / \ / \ / \ / \ / main thread | | V Actual minimum - In general:
start[0] = N_Items/N_Threads; start[1] = 2 * N_Items/N_Threads; start[2] = 3 * N_Items/N_Threads; .... ^ | | Pass this "ID" to a thread so the thread can determine its share of labor
Program - Main Thread:
#define N_Items 1000000 /* Shared Variables */ double x[N_Items]; // Must be SHARED (accessed by worker threads !!) int start[100]; // Contain starting array index of each thread double min[100]; // Contain the minimum found by each thread int num_threads; // ----------------------------------- // Create worker threads.... // ----------------------------------- for (i = 0; i < num_threads; i = i + 1) { start[i] = i; // Pass ID to thread in a private variable if ( pthread_create(&tid[i], NULL, worker, (void *)&start[i]) ) { cout << "Cannot create thread" << endl; exit(1); } } // ----------------------------------- // Wait for worker threads to end.... // ----------------------------------- for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); // ---------------------------------------- // Post processing: Find actual minimum // ---------------------------------------- my_min = min[0]; for (i = 1; i < num_threads; i++) if ( min[i] < my_min ) my_min = min[i];
Worker Thread:
void *worker(void *arg) { int i, s; int n, start, stop; double my_min; n = N_Items/num_threads; // number of elements to handle /* ------------------------------------ Get thread's ID (value = 0 or 1 or 2..) ------------------------------------ */ s = * (int *) arg; /* --------------------------------- Locate the starting index --------------------------------- */ start = s * n; // Starting index /* --------------------------------- Locate the ending index --------------------------------- */ if ( s != (num_threads-1) ) { stop = start + n; // Ending index } else { stop = N_Items; // Ending index } /* --------------------------------- Find min in my section of array --------------------------------- */ my_min = x[start]; for (i = start+1; i < stop; i++ ) // Find min in my range { if ( x[i] < my_min ) my_min = x[i]; } min[s] = my_min; // Store min in private slot return(NULL); /* Thread exits (dies) */ }
- Example Program:
(Demo above code)
- Prog file: click here
Compile with: CC -mt min-mt1.C
- The
interleaved labor division
scheme is a
commonly used labor division method
for its
coding simplificity
- Interleaved labor division method:
(Again, for simplicity of discussion, I used 2 threads)
- Split the array into 2 (approximate) equal halfs
- Thread 0 performs the task on the
odd-indexed elements
(I.e.: x[0], x[2], x[4], etc) - Thread 1 performs the task on the
even-indexed elements
(I.e.: x[1], x[3], x[5], etc) - Main thread waits for the results and find the actual minimum.
Graphically:
values handled by thread 0 | | | | | | | | | | | | | | V V V V V V V V V V V V V V |-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-| ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | | | | | | | | | | | | | | values handled by thread 1 Thread 0 Thread 1 | | | | V V min[0] min[1] \ / \ / \ / \ / \ / main thread | | V Actual minimum - Split the array into 2 (approximate) equal halfs
- In general:
elements processed by thread[0]: 0, 0+N_Threads, 0+2*N_Threads ... elements processed by thread[1]: 1, 1+N_Threads, 1+2*N_Threads ... elements processed by thread[2]: 2, 2+N_Threads, 2+2*N_Threads ... .... ^ | | Pass this "ID" to a thread so the thread can determine its share of labor
Program - Main Thread: (UNCHANGED)
// ----------------------------------- // Create worker threads.... // ----------------------------------- for (i = 0; i < num_threads; i = i + 1) { start[i] = i; // Pass ID to thread in a private variable if ( pthread_create(&tid[i], NULL, worker, (void *)&start[i]) ) { cout << "Cannot create thread" << endl; exit(1); } } // ----------------------------------- // Wait for worker threads to end.... // ----------------------------------- for (i = 0; i < num_threads; i = i + 1) pthread_join(tid[i], NULL); // ---------------------------------------- // Post processing: Find actual minimum // ---------------------------------------- my_min = min[0]; for (i = 1; i < num_threads; i++) if ( min[i] < my_min ) my_min = min[i];
Worker Thread: changed !!!
void *worker(void *arg) { int i, s; double my_min; s = * (int *) arg; // -------------------------------------- // Find min in my range // -------------------------------------- my_min = x[s]; for (i = s + num_threads; i < N_Items; i = i + num_threads) { if ( x[i] < my_min ) my_min = x[i]; } min[s] = my_min; // Store min in private slot return(NULL); /* Thread exits (dies) */ } - Interleaved labor division algorithm
is
much easier to code
- Example Program:
(Demo above code)
- Prog file: click here
Compile with: CC -mt min-mt2.C
- Try running the programs using
different number
of threads
(the program prints the elapsed time)
- Notice that the first version has the
best performance
on multi-processors (e.g. on compute)
- The second version performs
poorly compared to
first vserion
- Computer programs store information
in main memory
- The computer
does not store
the
entire program
(instructions + variables)
in main memory
Rather:
- The main memory contains the most recently accessed instructions and variables used by the computer program
- The reason is:
- The main memory
of the computer is a
scarse resource
whose utilization needs to be optimized
- There is no need to store
instructions and
variables
that the program does not used
in the memory
- The best indicator
that
instructions and
variables
will be used in the
near future
is the fact that
they
have been used recently
due to fact that
computer programs contains
many loops :
for ( ....... ) { .... same instructions and variables used over and over again in loop ... }
- The main memory
of the computer is a
scarse resource
whose utilization needs to be optimized
- Paging:
- Paging is a
technique
where the computer
store program instructions and variable
on disk and
selectively reads in
the
needed "pages" (blocks) of
program instructions and variables
- Only a portion of the entire program resides in the main memory at any one time
- Paging is a
technique
where the computer
store program instructions and variable
on disk and
selectively reads in
the
needed "pages" (blocks) of
program instructions and variables
- What paging does:
- When a page (block, ~4 Kbytes)
is
no longer needed ,
the page is
written to disk
- The occupied memory is re-used to store instruction/variable from a different part of the program
Graphically:
- When a page (block, ~4 Kbytes)
is
no longer needed ,
the page is
written to disk
- NOTE:
- To implement
paging ,
the computer provides a
mapping facility
that
translates a
program address to a memory address
- The
address mapping facility
is beyond the scope of
this course
- Suffice to say that
there is a black box
function available that:
+----------+ | | | Address | Program address ------> | | -----> Memory address | mapping | | | +----------+
- To implement
paging ,
the computer provides a
mapping facility
that
translates a
program address to a memory address
- Access pattern in block-wise division:
start[0] start[1] | | | values handled by | values handled by V thread 0 V thread 1 |---------------------->|---------------------->| Page access pattern: |----|----|----|... |----|----|----|... ^ ^ | |Conclusion:
- Pages that store the array elements are accessed exactly once
- Access pattern in block-wise division:
values handled by thread 0 | | | | | | | | | | | | | | V V V V V V V V V V V V V V |-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-| ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | | | | | | | | | | | | | | values handled by thread 1 Page access pattern: | V |-------|-------|-------|... ^ |Conclusion:
- Pages that store the
array elements
may need to be
accessed multiple times
- Sample scenario:
- Thread 1 access page 0, reads in page 0
- Thread 1 finishes processing elements in page 0, and goes on to page 1 - replacing page 0
- Thread 0 access page 0, reads in page 0 again
- Pages that store the
array elements
may need to be
accessed multiple times
- Summary:
- When processing large amount of data in memory, choose an access pattern that minimizes the number of page replacements