Source repo: sdsc-summer-institute-2024 | Branch: main | Last synced: 2026-04-24 10:27:17.425 UTC

Dask Tutorial

Watch the full SciPy 2020 tutorial

Dask provides multi-core execution on larger-than-memory datasets.

We can think of dask at a high and a low level

• High level collections: Dask provides high-level Array, Bag, and DataFrame collections that mimic NumPy, lists, and Pandas but can operate in parallel on datasets that don't fit into main memory. Dask's high-level collections are alternatives to NumPy and Pandas for large datasets.
• Low Level schedulers: Dask provides dynamic task schedulers that execute task graphs in parallel. These execution engines power the high-level collections mentioned above but can also power custom, user-defined workloads. These schedulers are low-latency (around 1ms) and work hard to run computations in a small memory footprint. Dask's schedulers are an alternative to direct use of threading or multiprocessing libraries in complex cases or other task scheduling systems like Luigi or IPython parallel.

Different users operate at different levels but it is useful to understand both. This tutorial will interleave between high-level use of dask.array and dask.dataframe (even sections) and low-level use of dask graphs and schedulers (odd sections.)

On Expanse

Use the singularity container at:

/expanse/lustre/projects/sds166/zonca/dask-numba-si21.sif

Prepare

conda env create -f binder/environment.yml conda activate python-hpc jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter labextension install @bokeh/jupyter_bokeh jupyter labextension install dask-labextension jupyter serverextension enable dask_labextension

Links

• Reference
  • Docs
  • Examples
  • Code
  • Blog
• Ask for help
  • dask tag on Stack Overflow, for usage questions
  • github issues for bug reports and feature requests
  • gitter chat for general, non-bug, discussion
  • Attend a live tutorial

Outline

0. Overview - dask's place in the universe.

1. Delayed - the single-function way to parallelize general python code.

2. Array - blocked numpy-like functionality with a collection of numpy arrays spread across your cluster.

3. Dataframe - parallelized operations on many pandas dataframes spread across your cluster.

4. Distributed - Dask's scheduler for clusters, with details of how to view the UI.

5. Advanced Distributed - further details on distributed computing, including how to debug.

6. Dataframe Storage - efficient ways to read and write dataframes to disc.

7. Machine Learning - applying dask to machine-learning problems.