2.1 Data Management

Source repo: sdsc-summer-institute-2025 | Branch: main | Last synced: 2026-04-24 10:27:17.425 UTC

SDSC Summer Institute 2025

Session 3.2 Data Management

Date: Monday, August 4, 2025

Summary: Proper data management is essential to make effective use of high-performance computing (HPC) systems and other advanced cyberinfrastructure (CI) resources. This session will cover an overview of filesystems, data compression, archives (tar files), checksums and MD5 digests, downloading data using wget and curl, data transfer and long-term storage solutions.

Presented by: Marty Kandes (mkandes @sdsc.edu)

Data has a lifecycle. Data management is a lifestyle.

Image Credit: Harvard Biomedical Data Management

Reading and Presentations:

• Lecture material:

  • COMPLECS: Data Management 1: Data Storage and File Systems
  • COMPLECS: Data Management 2: Data Transfer

• Source Code/Examples:

  • https://github.com/sdsc-complecs/data-storage-filesystems
  • https://github.com/sdsc-complecs/data-transfer

Additional References

• Data Management & File Systems (2023)
• Data Management & Job Submission (2022)
• Data Management & File Systems on Expanse (2021)
• Implementing Research Data Management for Labs & Grants (2021)