MPI Collectives in Practice: From Bcast to Allreduce

1. Scenario: One Process Talking to Everyone

Point-to-point is "one to one". Real computation is full of "one-to-many / many-to-one / all-to-all": the root process broadcasts input to every process, each process computes a slice, then results are combined. MPI packages these patterns as collective operations — one call coordinates the whole group, and it is much harder to deadlock.

This article runs three examples (broadcast, parallel sum, global statistics) and connects the results to the HPL grid parameters measured on real clusters.

2. Example 1: MPI_Bcast

The root process (rank 0) distributes an array to all processes. Every process must call MPI_Bcast at the same time with matching arguments; only the root provides real data.

// bcast.c -- root broadcasts a config string
#include <mpi.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    char msg[64];
    if (rank == 0) {
        strcpy(msg, "HPL.N=3145728 NB=384 P*Q=80");
    }
    MPI_Bcast(msg, 64, MPI_CHAR, 0, MPI_COMM_WORLD);

    printf("Rank %d got config: %s\n", rank, msg);

    MPI_Finalize();
    return 0;
}

mpicc -O2 -o bcast bcast.c
mpirun -perhost 4 -np 8 -hostfile hostfile ./bcast

Key point: collectives are collective actions — if any process skips the call, everyone else waits forever. This is the biggest difference from point-to-point and the first thing to check when a job appears "hung".

3. Example 2: MPI_Reduce for Parallel Sum

Split "sum 1..N" across P processes: each process adds its own slice, then MPI_Reduce combines partial sums at the root.

// reduce.c -- parallel sum 1..N
#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    long long N = 100000000;            // 100 million
    long long n_per = N / size;
    long long start = rank * n_per + 1;
    long long end = (rank == size - 1) ? N : start + n_per - 1;
    long long local = 0;
    for (long long i = start; i <= end; i++) local += i;

    long long total = 0;
    MPI_Reduce(&local, &total, 1, MPI_LONG_LONG, MPI_SUM, 0, MPI_COMM_WORLD);

    if (rank == 0) {
        long long expect = N * (N + 1) / 2;
        printf("parallel sum = %lld, expected = %lld, %s\n",
               total, expect, total == expect ? "MATCH" : "MISMATCH");
    }

    MPI_Finalize();
    return 0;
}

mpicc -O2 -o reduce reduce.c
mpirun -perhost 8 -np 16 -hostfile hostfile ./reduce

Observed with 16 processes: parallel sum = 5000000050000000, expected = 5000000050000000, MATCH. This is the seed of HPL: decompose a large matrix into a P×Q process grid, eliminate locally, then reduce.

4. Example 3: MPI_Allreduce for Global Statistics

MPI_Reduce delivers the result only to the root; MPI_Allreduce gives the result to all processes (used for convergence checks and global max/min/avg).

// allreduce.c -- every process reports one value; global max/min/avg
#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    double local = (rank + 1) * 1.5;      // simulated per-process metric
    double g_max, g_min, g_sum;
    MPI_Allreduce(&local, &g_max, 1, MPI_DOUBLE, MPI_MAX, MPI_COMM_WORLD);
    MPI_Allreduce(&local, &g_min, 1, MPI_DOUBLE, MPI_MIN, MPI_COMM_WORLD);
    MPI_Allreduce(&local, &g_sum, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);

    printf("Rank %d: global max=%.1f min=%.1f avg=%.2f\n",
           rank, g_max, g_min, g_sum / size);

    MPI_Finalize();
    return 0;
}

Allreduce internally uses tree or ring algorithms whose communication cost grows logarithmically (or stays near-constant) with process count — that is why it can support training/evaluation at tens of thousands of processes.

5. Collectives and the HPL Grid: The Real Mapping

Review the key HPL.dat parameters from CPU HPL in Practice: Single Node to 80 Nodes:

HPL.dat parameter	Measured value	Communication meaning
PMAP (process mapping)	1 (Column-major)	how the P×Q grid is numbered, affects data layout
Ps Qs	10 × 8 (80 procs)	process grid; P×Q = total processes
BCASTs	0 (1rg)	panel broadcast algorithm, built on MPI_Bcast
DEPTHs (lookahead)	0	pipeline depth, hides broadcast latency

Experience: the closer P×Q is to a square, the fewer communication steps; but it must also match node topology (e.g., 8 nodes × 10 processes) to avoid cross-node hotspots. Grid shape, broadcast algorithm, and measured efficiency are the core of HPL tuning — see the tuning retrospective in the next article.

6. Practical Checklist

• [ ] All processes must call collectives in the same order with matching arguments, or the job "collectively hangs"
• [ ] MPI_Reduce gives the result only to the root; use MPI_Allreduce when everyone needs it
• [ ] For large arrays use MPI_Scatter/MPI_Gather (chunk/collect), not Bcast
• [ ] Use MPI_LONG_LONG for 64-bit integers across platforms to avoid overflow
• [ ] Before a full-scale HPL or training run, validate collectives with the small bcast + allreduce examples