MPI Point-to-Point in Practice: Ring Relay and Latency Measurement

1. Scenario: Making Processes Actually Talk

Hello World only proves that all processes can start; no messages flow between them. Point-to-point (P2P) is the most basic form of MPI communication: one process sends, one receives. This article runs two compile-ready examples — a ring relay and a ping-pong latency probe — to get P2P communication working and measurable.

Environment: the oneAPI setup from MPI Hello World in Practice in this knowledge base.

mpirun -perhost 4 -np 8 -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER tcp -hostfile hostfile ./ring

-perhost 4 starts 4 processes per node; hostfile lists one hostname per line.

2. Example 1: Ring Relay

Eight processes form a ring: rank 0 sends to rank 1, rank 1 to rank 2, ..., rank 7 back to rank 0. A single MPI_Sendrecv (receive, then send) avoids deadlock.

// ring.c -- ring relay: each process increments and passes on
#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int next = (rank + 1) % size;
    int prev = (rank - 1 + size) % size;
    int recv_buf = 0, send_buf = rank + 1;

    if (rank == 0) {
        // rank 0 starts the first leg
        MPI_Send(&send_buf, 1, MPI_INT, next, 0, MPI_COMM_WORLD);
        MPI_Recv(&recv_buf, 1, MPI_INT, prev, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        printf("Rank %d: received relay value %d, full loop done\n", rank, recv_buf);
    } else {
        // everyone else: receive first, then send
        MPI_Recv(&recv_buf, 1, MPI_INT, prev, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
        MPI_Send(&recv_buf, 1, MPI_INT, next, 0, MPI_COMM_WORLD);
        printf("Rank %d: received %d, sent to rank %d\n", rank, recv_buf, next);
    }

    MPI_Finalize();
    return 0;
}

Build and run:

mpicc -O2 -o ring ring.c
mpirun -perhost 4 -np 8 -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER tcp -hostfile hostfile ./ring

Observed output (key lines):

Rank 7: received 7, sent to rank 0
Rank 3: received 3, sent to rank 4
Rank 0: received relay value 8, full loop done

Key point: MPI_Sendrecv completes "send the next leg + receive the previous leg" in one call, avoiding the classic deadlock where two processes both block in Send. The separate Send/Recv version here demonstrates the ordering trap: if every process calls Send before Recv, large -np runs can stall when buffers fill up.

3. Example 2: Ping-Pong Latency Probe

The standard microbenchmark for network/communication quality is ping-pong: rank 0 sends a message to rank 1, rank 1 echoes it back; one round trip is timed. Small messages approximate latency; large messages reveal bandwidth.

// pingpong.c -- round-trip latency/bandwidth
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (size != 2) {
        if (rank == 0) printf("Run with -np 2\n");
        MPI_Finalize();
        return 1;
    }

    int iters = 1000;
    int max_bytes = 1 << 20;  // 1 MB
    for (int nbytes = 4; nbytes <= max_bytes; nbytes *= 2) {
        char *buf = (char *)malloc(nbytes);
        double t0 = MPI_Wtime();
        for (int i = 0; i < iters; i++) {
            if (rank == 0) {
                MPI_Send(buf, nbytes, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
                MPI_Recv(buf, nbytes, MPI_CHAR, 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
            } else {
                MPI_Recv(buf, nbytes, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
                MPI_Send(buf, nbytes, MPI_CHAR, 0, 0, MPI_COMM_WORLD);
            }
        }
        double t1 = MPI_Wtime();
        double sec_per_pingpong = (t1 - t0) / iters;    // one round trip
        double latency_us = sec_per_pingpong / 2 * 1e6; // one way
        double bw_mbps = (double)nbytes * 8 / sec_per_pingpong / 1e6;
        if (rank == 0)
            printf("%8d bytes | latency %8.1f us | bandwidth %8.1f Mbps\n",
                   nbytes, latency_us, bw_mbps);
        free(buf);
    }

    MPI_Finalize();
    return 0;
}

Run:

mpicc -O2 -o pingpong pingpong.c
mpirun -perhost 1 -np 2 -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER tcp -hostfile hostfile ./pingpong

Typical observations (2 nodes, TCP path):

Message size	One-way latency	Bandwidth	Note
4 B	~10-30 us	low	close to network latency floor (TCP path is higher)
4 KB	improves with fabric	hundreds of Mbps-Gbps	bandwidth competition zone
1 MB	bandwidth-limited	near link ceiling	large messages show throughput

Takeaway: with shm:ofi, same-node processes use shared memory (lowest latency), while cross-node traffic goes through FI_PROVIDER tcp (clearly higher latency than direct IB/RoCE). This is why large HPL runs must configure fabric/provider carefully — see MPI + Slurm in Practice: Scheduling and Performance Tuning.

4. Practical Checklist

• [ ] mpicc compiles (if command not found, run source /opt/intel/oneapi/setvars.sh first)
• [ ] hostfile has one hostname per line and passwordless SSH works
• [ ] Ring runs from 2 to N processes (-np 1 needs special handling for next==prev; start at 2)
• [ ] Ping-pong must use -np 2, otherwise it exits with an error
• [ ] If cross-node latency is abnormally high, check I_MPI_FABRICS/FI_PROVIDER and firewall first

Next, MPI Collectives in Practice runs Bcast/Allreduce and explains why the HPL grid parameters P×Q depend on collective performance.