Point-to-Point blocking Send and Receive in MPI

Point-to-point communication
- The term point-to-point communication is used to describe a communication between one source with one destination

Blocking and non-blocking communication

Communication can be blocking or non-blocking

Blocking communications

If a send operation is blocking , then the operation will not TERMINATE until the message has been received by the destination
If a receive operation is blocking , then the operation will not TERMINATE until a message has been received

NON-Blocking communications

If a send operation is NON-blocking , then the operation will not TERMINATE IMMEDIATELY WITHOUT waiting for the message reception by the destination
If a receive operation is NON-blocking , then the operation will not TERMINATE IMMEDIATELY WITHOUT WAITING a message to arrive

The reason for providing NON-blocking send and receive operations is to allow the process to do something else while waiting for the message to be transmitted or for some message to arrive...
Obviously, after invoking NON-blocking send or receive operations, the process needs to have a way to make sure that the data has been sent or some data has arrived before proceeding...
In this webpage, we will look at (the easier) blocking send and receive MPI operation first
In the next webpage, we will study the (more difficult) NON-blocking send and receive.

Point-to-point communication primitives

Recall that the MPI point-to-point blocking communication primitives are:

Function Name	Usage
MPI_Send(void *buff, int count, MPI_Datatype type, int dest, int tag, int comm)	Send a point-to-point message to process dest in the communication group comm The message is stored at memory location buff and consists of count items of datatype type The message will be tagged with the tag value tag The MPI_Send() function will only return if the message sent has been received by the destination. (It's safe to reuse the buffer buff right away).
MPI_Recv(void buff, int count, MPI_Datatype type, int source, int tag, int comm, MPI_Status status)	Receive a point-to-point message The message MUST BE from the process source in the communication group comm AND the message MUST BE tagged with the tag value tag The message received will be stored at memory location buff which will have space to store count items of datatype type When the function exits, the exit status is stored in status. Information about the received message is returned in a status variable, e.g.,: The received message tag is status.MPI_TAG and The rank of the sending process (of the message) is status.MPI_SOURCE. In most cases, you know the structure of data received and then you can ignore the status value... If you pass NULL as status parameter, MPI will not return the status value. The MPI_Recv() function will only return if the desired message (from source with tag tag) has been received - or exits with an error code...

Function Name

Usage

MPI_Send(void *buff,
     int count,
     MPI_Datatype type,
     int dest,
     int tag,
     int comm)

Send a point-to-point message to process dest in the communication group comm

The message is stored at memory location buff and consists of count items of datatype type

The message will be tagged with the tag value tag

The MPI_Send() function will only return if the message sent has been received by the destination. (It's safe to reuse the buffer buff right away).

MPI_Recv(void *buff,
     int count,
     MPI_Datatype type,
     int source,
     int tag,
     int comm,
     MPI_Status *status)

Receive a point-to-point message

The message MUST BE from the process source in the communication group comm AND the message MUST BE tagged with the tag value tag

The message received will be stored at memory location buff which will have space to store count items of datatype type

When the function exits, the exit status is stored in status.

Information about the received message is returned in a status variable, e.g.,:

The received message tag is status.MPI_TAG and
The rank of the sending process (of the message) is status.MPI_SOURCE.

In most cases, you know the structure of data received and then you can ignore the status value... If you pass NULL as status parameter, MPI will not return the status value.

The MPI_Recv() function will only return if the desired message (from source with tag tag) has been received - or exits with an error code...

Be careful that:

The MPI_Recv() function is a selective receive function.
It will only receive (and return) a message from a specific source and with a specific tag
If you want to receive a message from any source , then use the value MPI_ANY_SOURCE for the "source" parameter
If you want to receive a message with any tag value, then use the value MPI_ANY_TAG for the "tag" parameter

Example: sending and receiving with the wrong tag

/* This is a modified "Hello World" program */ #include "mpi.h" #include <iostream.h> #include <string.h> int main(int argc, char **argv) { char reply[100]; char buff[128]; int numprocs; int myid; int i; MPI_Status stat; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); if (myid == 0) { /* ------------------------------------------------------------ This portion of code is executed by MPI 0 process - send ------------------------------------------------------------ */ printf("WE have %d processors\n", numprocs); for( i=1; i < numprocs; i++) { sprintf(buff, "Hello %d", i); MPI_Send(buff, 128, MPI_CHAR, i, 1234 /* tag !!!! */, MPI_COMM_WORLD); } for(i=1; i < numprocs; i++) { MPI_Recv(buff, 128, MPI_CHAR, i, 0, MPI_COMM_WORLD, &stat); cout << buff << endl; } } else { /* ------------------------------------------------------------ This portion of code is executed by the other MPI processes ------------------------------------------------------------ */ MPI_Recv(buff, 128, MPI_CHAR, 0, 0 /* tag !!!! */, MPI_COMM_WORLD, &stat); sprintf(reply, " |--> Hello 0, Processor %d is present and accounted for !", myid); strcat(buff, reply); MPI_Send(buff, 128, MPI_CHAR, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

Example Program: (Sends a message with tag 1234, but receiver is receiving a message with tag 0)
- Prog file: click here
Demo instruction:
Another Example Program:
(This program demos that the source node parameter must also match for a messaeg to be received)
- Prog file: click here
Demo instruction:
1. ssh lab1a
2. cd sunhome/MPI
3. mpiCC -o Comm02 Comm02.C
4. mpirun -np 4 ./Comm02
5. Try changing source to MPI_ANY_SOURCE and run it again

The MPI_Status data structure
- The MPI_Status data structure is one of the output paramters in the MPI_Recv() function.
  The "status" variable is used by MPI mesage receiving functions to obtain information about a received message.
- The MPI_Status variable status contains at least the following fields:
- There may also be other fields in the stucture, but these are reserved for the implementation and you are not supposed to access them directly.

Name of field	Usage
`MPI_SOURCE`	id of processor sending the message (integer)
`MPI_TAG`	tag of the message (integer)
`MPI_ERROR`	error code (integer)

Data Types in messages

As you know from CS255:

must have a type

Without the type information, it is impossible to know the value conveyed in a representation.

Example:

What does 11111111 represent ?

The answer depends on the type information:

11111111 represents -1 if the type is signed integer (2's complement encoding)
11111111 represents 255 if the type is UNsigned integer

Without the type information, you cannot answer correctly

One of the main problem in data communication is to convey the TYPE INFORMATION to the receiver.
In MPI, all you can do is:
It is upto the program (better: programmer) on how to interprete the content
Next, we will learn how to specify the amount of data to send and receive.

Sending/Receiving SIMPLE data types in MPI
- The MPI communication primitives allow you to send simple data types without much effort
- To send N items of the type TYPE stored at location buff, use the call:
  All this function does is:
- Use the following pre-defined MPI symbolic constants to indicate the most commonly used C/C++ type:
- To receive N items of the type TYPE stored at location buff, use the call:
  All this function does is:
  - Wait until N*k bytes has been received.
  - Save these bytes in memory starting at buff
    (k is the size (number of bytes) of the type TYPE)

C/C++ Type	MPI symbolic constant
`char`	`MPI_CHAR`
`int`	`MPI_INT`
`float`	`MPI_FLOAT`
`double`	`MPI_DOUBLE`

Illustrative Example 1: send and receive types can be different

Process 1 sends 4 characters to process 0

Process 0 receives an integer (4 bytes)

int main(int argc, char **argv) { char in[4]; // Send 4 characters int out; // Interprete as an integer int numprocs; int myid; int i; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); if(myid == 0) { cout << "We have " << numprocs << " processors" << endl; MPI_Recv(&out, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, NULL); cout << "Received this number from proc 1: " << out << endl; } else if ( myid == 1 ) { in[0] = '2'; in[1] = 1; in[2] = 0; in[3] = 0; MPI_Send(in, 4, MPI_CHAR, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

Program will print (256+50 = 306) - because ASCII code for '2' is 50...

(Note: lab1a is a Intel based machine and uses little endian storage, that's why we need to put the '2' in the first byte)

Example Program: (Demo above code)
- Prog file: click here
Demo instruction:
1. ssh lab1a
2. cd sunhome/MPI
3. mpiCC -o Comm03 Comm03.C
4. mpirun -np 4 ./Comm03

Illustrative Example 2: strange behavior when send and receive SIZE are different

Process 1 sends 1 character to process 0

Process 0 receives an integer (4 bytes)

int main(int argc, char **argv) { char in[1]; // Send 1 character int out; // Interprete as an integer int numprocs; int myid; int i; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); if(myid == 0) { cout << "We have " << numprocs << " processors" << endl; MPI_Recv(&out, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, NULL); cout << "Received this number from proc 1: " << out << endl; } else if ( myid == 1 ) { in[0] = '2'; // ONLY 1 Character MPI_Send(in, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

Program prints a random number - because the last 3 characters are "random" - the first ASCII code is 50, re[resenting the character '2' - see the output of the program:

c[0] = 2 <- this was initialized. c[1] = H <- the next 3 characters ar "random"... c[2] = 3 c[3] =

Example Program: (Demo above code)
- Prog file: click here
Demo instruction:
1. ssh lab1a
2. cd sunhome/MPI
3. mpiCC -o Comm04 Comm04.C
4. mpirun -np 4 ./Comm04

Cooperation to complete a common task: A REAL Example in MPI programming

Consider the problem of computing the value of Pi by numerical integration

Scheme to divide the work:

Divide the the computational labor into N portions and each processor will compute its own (partial) sum.
Then at the end, the master (processor 0) collects all (partial) sums and forms a total sum.

Pseudo code:

/* ============================================================ Prepare MPI ============================================================ */ MPI_Init(&argc,&argv); // Initialize MPI_Comm_size(MPI_COMM_WORLD, &num_procs); // Get # processors MPI_Comm_rank(MPI_COMM_WORLD, &myid); N = # intervals used to do the integration... w = 1.0/(double) N; mypi = 0.0; // My partial sum (from a MPI processor) Compute my part of the partial sum based on 1. myid 2. num_procs if ( I am the master of the group ) { for ( i = 1; i < num_procs; i++) { receive the partial sum from MPI processor i; Add partial sum to my own partial sum; } Print final total; } else { Send my partial sum to the master of the MPI group; } MPI_Finalize();

MPI program to compute Pi by integration:

double f(double a) { return( 2.0 / sqrt(1 - a*a) ); } int main(int argc, char *argv[]) { int N; // Number of intervals double w, x; // width and x point int i, myid; double mypi, others_pi; MPI_Init(&argc,&argv); // Initialize MPI_Comm_size(MPI_COMM_WORLD, &num_procs); // Get # processors MPI_Comm_rank(MPI_COMM_WORLD, &myid); N = atoi(argv[1]); w = 1.0/(double) N; mypi = 0.0; /* --------------------------------------------------------- Every MPI process computes a partial sum for the integral --------------------------------------------------------- */ for (i = myid; i < N; i = i + num_procs) { x = w*(i + 0.5); mypi = mypi + w*f(x); } /* ----------------------------- Now put the sum together... ----------------------------- */ if ( myid == 0 ) { /* ---------------------------------------------------- Proc 0 collects and others send data to proc 0 ---------------------------------------------------- */ for (i = 1; i < num_procs; i++) { MPI_Recv(&others_pi, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, NULL); mypi += others_pi; } cout << "Pi = " << mypi<< endl << endl; // Output... } else { /* --------------------------------------------- The other processors send their partial sum to processor 0 --------------------------------------------- */ MPI_Send(&mypi, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

NOTE:
Example Program: (Demo above code)
- Prog file: click here
Demo instruction:
1. ssh lab1a
2. cd sunhome/MPI
3. mpiCC -o MPI-Pi MPI-Pi.C
4. mpirun -np 4 ./MPI-Pi