Parallel programming

Students are asked to bring there own computer.

The course follows a single path-from getting onto an HPC cluster to extracting scalable performance from a scientific application - and every step along that path is driven by measurement. Rather than presenting OpenMP and MPI as collections of language features, the course focuses on understanding performance: forming hypotheses, validating them through profiling and benchmarking, and using quantitative evidence to guide optimization.

 

During the taught week, participants first acquire the practical skills required to work effectively on a cluster: connecting to the system, transferring files, compiling and debugging applications, and using Slurm to manage computational resources. Alongside these fundamentals, the course introduces the hardware and software concepts that underpin performance engineering, including processor and memory architectures, the roofline model, profiling techniques, and strong and weak scaling. Participants also learn how to distinguish between theoretical hardware capabilities and the performance that can actually be achieved in practice.

 

Before optimizing any code, the course establishes a quantitative baseline. Standard micro-benchmarks are used to characterize the capabilities of the hardware, while a deliberately unoptimized finite-difference Poisson solver is profiled and analyzed to identify its performance bottlenecks. This baseline provides the reference against which every subsequent optimization is evaluated.

 

Building on this foundation, participants progressively improve the application at three levels. They first optimize serial performance through techniques such as improving memory access patterns and exploiting vectorization. They then introduce shared-memory parallelism with OpenMP, exploring thread placement, scheduling strategies, and scalability on multicore systems. Finally, they extend the application to distributed-memory systems using MPI, investigating domain decomposition, communication overheads, and techniques that improve scalability across multiple nodes. Throughout, the emphasis remains on understanding why an optimization succeeds or fails, rather than simply applying programming constructs.

 

The taught week is followed by a project period spanning two calendar weeks, during which participants apply the methodology to a provided scientific application. The project represents approximately one week of full-time work and gives participants the opportunity to independently profile, optimize, parallelize, and characterize the performance of a larger code.

 

By the end of the course, participants will have developed a structured workflow for analyzing and improving the performance of scientific applications. More importantly, they will have learned how to make optimization decisions based on measurement and evidence, enabling them to assess not only whether a program runs faster, but why it does so and how close it comes to the capabilities of the underlying hardware.

 

Please note that accelerator programming (e.g. GPUs) is outside the scope of this course.

 

Course Content

 

Week 1: Lectures and practical sessions

1. Basic concepts — getting productive on the machine

• Connecting to HPC systems
• Transferring data to and from the cluster
• Managing the software environment (modules and available software)
• Compilation workflows
• Debugging applications

2. Cluster architecture — understanding the hardware

• What a cluster is: from the system level down to individual compute resources
• Resource management with Slurm
• Compute nodes, CPUs, memory, and processor architecture
• Detailed analysis of the Jed node
• Comparison with other cluster architectures (e.g., Kuma and Lyra)

3. Performance measurement — understanding the software side

• Performance metrics and their interpretation
• Profiling: identifying where execution time is spent
• Scaling analysis: strong and weak scaling
• The roofline model
• The Pareto principle for prioritizing optimization effort

4. Node and application characterization — establishing a baseline

• Measuring the real capabilities of the machine:
• OSU benchmarks for network latency and bandwidth
• STREAM for sustainable memory bandwidth
• GEMM for floating-point performance
• Profiling and characterizing a deliberately unoptimized finite-difference Poisson solver

5. Single-core optimization

• Making the application efficient before introducing parallelism.
• Basic optimization principles
• Memory hierarchy and data locality
• SIMD and vectorization
• Measuring the impact of low-level optimizations

6. OpenMP - shared-memory parallelism

• Introduction to OpenMP concepts and programming model
• Thread affinity and pinning
• Scheduling strategies
• Scaling studies
• Profiling and analyzing threaded applications

7. MPI -€” distributed-memory parallelism: fundamentals

• Domain decomposition of the Poisson solver
• Initial 1D decomposition approach
• Scaling and profiling distributed applications
• Communication versus computation trade-offs
• Identifying the limitations of simple decompositions

8. MPI -€” advanced techniques

• MPI derived datatypes
• Topology-aware Cartesian decompositions
• Comparing decomposition strategies using performance measurements

Weeks 2 and 3: Project period

 

A scientific application is provided and must be parallelized and characterized
Profiling, optimization, and performance analysis are carried out using the methods introduced during the course

 

Expected workload:
Approximately one week of full-time work

 

Final presentation:
15-minute project presentation
5-minute questions and discussion