Config Management Camp 2024 Ghent

Creating Throwaway Supercomputers in the Cloud with Magic Castle
2024-02-05, 17:10–17:35, B.3.039

Magic Castle is an open-source software project that makes it easy to create your very own supercomputer on cloud resources (OpenStack, AWS, Azure, GCP, or OVH).

Using Terraform, cloud-init, and Puppet, it autonomously creates and configures a complete High Performance Computing (HPC) cluster infrastructure, including a login and management node, a set of worker nodes, the job scheduler (Slurm), a shared filesystem, a data transfer node, JupyterHub, and a shared software stack like EESSI that includes thousands of scientific software installations compiled by experts and distributed via CernVM-FS.
The cluster can be configured with autoscaling enabled, so that worker nodes are spun up on-demand when jobs are submitted, and are automatically destroyed again when the job queue is empty, which helps to reduce operation costs.

Magic Castle can be used by both experienced HPC system administrators and novices to create a dedicated HPC cluster in a matter of minutes for specific research or development projects, for training sessions, or just for fun.

In this talk you will learn how to deploy and configure your own virtual supercomputer on your preferred cloud provider, and how to get rid of it in a heartbeat when you no longer need it.

See also: slides (PDF)

Kenneth Hoste, a.k.a. boegel, is a computer scientist and FOSS enthusiast from Belgium. He holds a Masters (2005) and PhD (2010) in Computer Science from Ghent University (Belgium). His dissertation topic was "Analysis, Estimation and Optimization of Computer System Performance Using Machine Learning".

Since October 2010, he is a member of the HPC team at Ghent University where he is mainly responsible for user support & training. As a part of his job, he is also the lead developer and release manager of EasyBuild, a software build and installation framework for (scientific) software on High Performance Computing (HPC) systems.

He is actively involved in the EuroHPC Centre-of-Excellence MultiXscale, in which the European Environment for Scientific Software Installations (EESSI) is being developed.

In his free time, he is a family guy and a fan of loud music and beer, frequently attending gigs and festivals.
He enjoys helping people & sharing his expertise, and likes joking around.
He has a weak spot for stickers.