Asynchronous Communication Primitives of MPI

Synchronous Communication

Synchronous send operation:

A send operation is called synchronous when the operation will only complete after the message sent has been received
- i.e., A synchronous send operation is the blocking send operation that we have studied previously

Synchronous receive operation:

Similarly, a receive operation is called synchronous when the operation will only complete after a message has been received
- i.e., A synchronous receive operation is the blocking receive operation that we have studied previously

The MPI_Send() and MPI_Recv() are synchronous communication primitives :
- MPI_Send will not return until the destination processor has received the message.
- MPI_Recv will not return a message has been received.

Asynchronous Communication

Asynchronous send operation:

A send operation is called ASYNCHRONOUS if the operation will complete WITHOUT waiting for the message reception by the receiver
- i.e., the asynchronous send operation will return IMMEDIATELY
Note: later in the program, you may need to test if the message has been sent successfully

Asynchronous receive operation:

Similarly, a receive operation is called ASYNCHRONOUS when the receive operation complete WITHOUT having received any message !!!
- i.e., the asynchronuous receive operation returns IMMEDIATELY
Note: here also, later in the program, you may need to test if the some message has been receive yet before proceeding

Asynchronous communication primitives in MPI

The MPI_Isend() and MPI_Irecv() are the ASYNCHRONOUS communication primitives of MPI

Function Name Usage

MPI_Isend(void *buff,
     int count,
     MPI_Datatype type,
     int dest,
     int tag,
     int comm,
     MPI_Request *request )

Send a point-to-point message ASYNCHRONOUSLY to process dest in the communication group comm

The message is stored at memory location buff and consists of count items of datatype type

The message will be tagged with the tag value tag

The MPI_Isend() function will return IMMEDIATELY
The current sending status is available through the request variable (see MPI_Test() function)

NOTE: DO NOT change/update the variable buff until the message has been sent !!!

MPI_Irecv(void *buff,
     int count,
     MPI_Datatype type,
     int source,
     int tag,
     int comm,
     MPI_Request *request )

Receive a point-to-point message ASYNCHRONOUSLY
The message MUST BE from the process source in the communication group comm AND the message MUST BE tagged with the tag value tag

The message received will be stored at memory location buff which will have space to store count items of datatype type

The MPI_Irecv() function will return IMMEDIATELY -
The current RECEIVE status is available through the request variable (see MPI_Test() function)

NOTE: DO NOT use the variable buff until the message has been received !!!

Function Name	Usage
MPI_Isend(void buff, int count, MPI_Datatype type, int dest, int tag, int comm, MPI_Request request )	Send a point-to-point message ASYNCHRONOUSLY to process dest in the communication group comm The message is stored at memory location buff and consists of count items of datatype type The message will be tagged with the tag value tag The MPI_Isend() function will return IMMEDIATELY The current sending status is available through the request variable (see MPI_Test() function) NOTE: DO NOT change/update the variable buff until the message has been sent !!!
MPI_Irecv(void buff, int count, MPI_Datatype type, int source, int tag, int comm, MPI_Request request )	Receive a point-to-point message ASYNCHRONOUSLY The message MUST BE from the process source in the communication group comm AND the message MUST BE tagged with the tag value tag The message received will be stored at memory location buff which will have space to store count items of datatype type The MPI_Irecv() function will return IMMEDIATELY - The current RECEIVE status is available through the request variable (see MPI_Test() function) NOTE: DO NOT use the variable buff until the message has been received !!!

Note:

Both MPI_Isend() and MPI_Irecv() returns a MPI request variable "request"
This MPI request variable "request" identifies a particular asynchronous send/receive operation
The "request" variable is used to check on the completion status of the asychronous send/receive operation

Testing for the completion status of asynchronous send and receive

A program will ONLY operate correctly with valid data
So, after you have started an asynchronous send/receive operation, you can do some processing that does NOT used the data in the asynchronous communication (while the data is being sent/receive)
But sooner or later, you will need to work with the data that you have been sending/receiving asnynchronously
You can only proceed correctly if the asynchronous operation has completed

The following 2 MPI Primitives allows you to check/wait for the asynchronous operation:

Function Name Usage

MPI_Test(MPI_Request *request,
     int *flag,
     MPI_Status * status) This function returns immediately.
The function will return the status of an ASYNCHRONOUS send/receive request
The input parameter is request associated with an asynchronous send/receive operation.
The return values are:

flag:

if flag == 0, the send/receive operation is not yet complete
if flag != 0, the send/receive operation is complete and the variable status contains information about the message, e.g.,:

The received message tag is status.MPI_TAG and
The rank of the sending process (of the message) is status.MPI_SOURCE.

status: contains information about the message (use the information only if flag != 0

MPI_Wait(MPI_Request *request,
     MPI_Status * status) This function BLOCKS (waits) until the request is completed.
The input parameter is request associated with an asynchronous send/receive operation.
The function wait until that ASYNCHRONOUS send/receive request is COMPLETE

The return value is:

status: contains information about the message, e.g.,:

The received message tag is status.MPI_TAG and
The rank of the sending process (of the message) is status.MPI_SOURCE.

Function Name	Usage
MPI_Test(MPI_Request request, int flag, MPI_Status * status)	This function returns immediately. The function will return the status of an ASYNCHRONOUS send/receive request The input parameter is request associated with an asynchronous send/receive operation. The return values are: flag: if flag == 0, the send/receive operation is not yet complete if flag != 0, the send/receive operation is complete and the variable status contains information about the message, e.g.,: The received message tag is status.MPI_TAG and The rank of the sending process (of the message) is status.MPI_SOURCE. status: contains information about the message (use the information only if flag != 0
MPI_Wait(MPI_Request request, MPI_Status status)	This function BLOCKS (waits) until the request is completed. The input parameter is request associated with an asynchronous send/receive operation. The function wait until that ASYNCHRONOUS send/receive request is COMPLETE The return value is: status: contains information about the message, e.g.,: The received message tag is status.MPI_TAG and The rank of the sending process (of the message) is status.MPI_SOURCE.

Note: MPI_Isend() call followed immediately by a MPI_Wait() call in exactly a MPI_Send() call:

MPI_Isend(void *buff, int count, MPI_Datatype type, int dest, int tag, int comm, MPI_Request *request ) MPI_Wait( MPI_Request *request, MPI_Status * status)
Equivalent to: MPI_Send(void *buff, int count, MPI_Datatype type, int dest, int tag, int comm)

Similarly for the receive operation

Example: multi-staged processing using BLOCKING primitives

Suppose we want to find BOTH the minimum and the maximum in an array.

We can do that the following program:

int num_procs; double x[MAX]; // Input array int main(int argc, char *argv[]) { int start, stop; int myid; double my_min, others_min; // Minimum double my_max, others_max; // Maximum MPI_Init(&argc,&argv); // Initialize MPI_Comm_size(MPI_COMM_WORLD, &num_procs); // Get # processors MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Get my rank (id) /* -------------------------------------- Find the min. among my numbers -------------------------------------- */ n = MAX/num_procs; start = myid * n; if ( myid != (num_procs-1) ) { stop = start + n; } else { stop = MAX; } my_min = x[start]; for (i = start+1; i < stop; i = i + n ) { if ( x[i] < my_min ) my_min = x[i]; } if ( myid == 0 ) { /* ------------------------------------- Get the min from others and compare ------------------------------------- */ for (i = 1; i < num_procs; i++) { MPI_Recv(&others_min, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, NULL); if ( others_min < my_min ) my_min = others_min; } } else { MPI_Send(&my_min, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD); } ************************************************************ Synchronization point: all programs are stopped here due to the synchronous send/receive operations used ************************************************************ /* -------------------------------------- Now find the max. among my numbers -------------------------------------- */ my_max = x[start]; for (i = start+1; i < stop; i = i + n ) { if ( x[i] > my_max ) my_max = x[i]; } if ( myid == 0 ) { /* ------------------------------------- Get the max from others and compare ------------------------------------- */ for (i = 1; i < num_procs; i++) { MPI_Recv(&others_max, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, NULL); if ( others_max > my_max ) my_max = others_max; } } else { MPI_Send(&my_max, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

Example Program: (Demo above code)
- Prog file: click here
Demo instruction:
1. ssh2 puma
2. cd sunhome/teaching/web/355/Syllabus/92-MPI/Progs
3. mpiCC -o MPI-max-min1 MPI-max-min1.C
4. mpirun -np 4 ./MPI-max-min1

Example: multi-staged processing using NON-BLOCKING primitives

The previous example causes the parallel processing to synchronize after computing the minimum
The synchronization will cause some processors to idle (waste time)
It would be better to synchronize at the END of the processing
This can be achieved using ASYNCHRONOUS communication primitives.
But there is one caveat:

Example:

int num_procs; double x[MAX]; // Input array int main(int argc, char *argv[]) { int start, stop; int myid; double my_min; double others_min[100]; // Save minimum separately double my_max; double others_max[100]; // Save maximum separately MPI_Request rq_min[100], rq_max[100]; // Status variables MPI_Init(&argc,&argv); // Initialize MPI_Comm_size(MPI_COMM_WORLD, &num_procs); // Get # processors MPI_Comm_rank(MPI_COMM_WORLD, &myid); // Get my rank (id) /* -------------------------------------- Find the min. among my numbers -------------------------------------- */ n = MAX/num_procs; start = myid * n; if ( myid != (num_procs-1) ) { stop = start + n; } else { stop = MAX; } my_min = x[start]; for (i = start+1; i < stop; i = i + n ) { if ( x[i] < my_min ) my_min = x[i]; } if ( myid == 0 ) { /* ------------------------------------- Get the min from others and compare ------------------------------------- */ for (i = 1; i < num_procs; i++) { MPI_Irecv(&others_min[i], 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, &rq_min[i]); } } else { MPI_Isend(&my_min, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &rq_min[0]); } ***************************************************************** NO Synchronization: all programs proceeds to the next stage of computation independently and will not wait on each other due to the asynchronous send/receive operations used ***************************************************************** /* -------------------------------------- Now find the max. among my numbers -------------------------------------- */ my_max = x[start]; for (i = start+1; i < stop; i = i + n ) { if ( x[i] > my_max ) my_max = x[i]; } if ( myid == 0 ) { /* ------------------------------------- Get the max from others and compare ------------------------------------- */ for (i = 1; i < num_procs; i++) { MPI_Irecv(&others_max[i], 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, &rq_max[i]); } } else { MPI_Isend(&my_max, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, &rq_min[0]); } ***************************************************************** Now we must gather the min[] and max[] from different processes HERE we must synchronize (we have no choice) That's OK, we are at the END of the processing ! Waiting for others to finish at the END to complete the whole job is a necessity, it's not wasting time ***************************************************************** /* -------------------------------------- Now synchronize to compute results -------------------------------------- */ if ( myid == 0 ) { // The Master processor waits for all min[] and max[] // messages to arrive for ( i = 1; i < num_procs; i++) { MPI_Wait( &rq_min[i], NULL ); if ( others_min[i] < my_min ) my_min = others_min[i]; } for ( i = 1; i < num_procs; i++) { MPI_Wait( &rq_max[i], NULL ); if ( others_max[i] > my_max ) my_max = others_max[i]; } // Print result.... cout << "min = " << my_min << endl << endl; cout << "max = " << my_max << endl << endl; } else { // The other processes must wait until their messages // has been received before exiting !!! MPI_Wait( &rq_min[0], NULL ); MPI_Wait( &rq_max[0], NULL ); } MPI_Finalize(); }

Example Program: (Demo above code)
- Prog file: click here
Demo instruction:
1. ssh2 puma
2. cd sunhome/teaching/web/355/Syllabus/92-MPI/Progs
3. mpiCC -o MPI-max-min2 MPI-max-min2.C
4. mpirun -np 4 ./MPI-max-min2