3.2 High Throughput Computing

Source repo: sdsc-summer-institute-2025 | Branch: main | Last synced: 2026-04-24 10:27:17.425 UTC

SDSC Summer Institute 2025

Session 3.5 High Throughput Computing

Date: Tuesday, August 5th, 2025

Summary: High-throughput computing (HTC) workloads are characterized by large numbers of small jobs. These frequently involve parameter sweeps where the same type of calculation is done repeatedly with different input values or data processing pipelines where an identical set of operations is applied to many files. This session covers the characteristics and potential pitfalls of HTC, job bundling, the Open Science Grid and the resources available through the Partnership to Advance Throughput Computing (PATh).

Presented by: Marty Kandes (mkandes @sdsc.edu)

Reading and Presentations:

• Lecture material:

  • COMPLECS: Batch Computing 3: High-Throughput and Many-Task Computing: Slurm Edition

• Source Code/Examples:

  • Batch job arrays
  • Batch job dependencies
  • Batch job bundling
  • Preemptible batch jobs
  • Distributed high-throughput computing

Additional References

• COMPLECS: Batch Computing 1: Working with the Linux Scheduler
• COMPLECS: Batch Computing 2: Getting Started with Batch Job Scheduling: Slurm Edition

Marty Kandes, Computational & Data Science Research Specialist, HPC User Services Group, SDSC