We expect the HPC clusters users to know what an HPC cluster is and what parallel computing is. As all HPC clusters are different, it is important for any users to have a general understanding of the clusters they are working on, what they offer and what are their limitations.

This section gives an overview of the technical HPC infrastructure and how things work at the University of Geneva. More details can be found in the corresponding sections of this documentation.

The last part of this page gives more details for advanced users.

The University of Geneva owns three HPC clusters or supercomputers : Baobab, Yggdrasil and Bamboo.

As for now, they are completely separated entities, each of them with their own private network, storage, and login node. What is shared is the accounting (user accounts and job usage).

Choose the right cluster for your work:

• You can use every clusters, but try to stick to one of them.
• Use the cluster where the private partition you have access to is located.
• If you need to access other servers not located in Astro, use Baobab or Bamboo to save bandwidth.
• As your data are stored locally on each cluster, avoid to use both clusters if this involves a lot of data moving between cluster.

You can't submit jobs from one cluster to the other one. This may be done in the future.

Boabab is physically located at Uni Dufour in Geneva downtown, while Yggdrasil is located at the Observatory of Geneva in Sauverny and Bamboo is located in campus Biotech

Each cluster is composed of :

• a login node (aka headnode) allowing users to connect and submit jobs to the cluster. Aach user is limited to 2 CPU cores and 8 GB of RAM on the login node.
• many compute nodes which provide the computing power. The compute nodes are not all identical ; they all provide CPU cores (from 8 to 128 cores depending on the model), and some nodes also have GPUs or more RAM (see below).
• management servers that you don't need to worry about, that's the HPC engineers' job. The management servers are here to provide the necessary services such as all the applications (with EasyBuild / module), Slurm job management and queuing system, ways for the HPC engineers to (re-)deploy compute nodes automatically, etc.
• BeeGFS storage servers which provide “fast” parallel file system to store the data from your $HOME and for the scratch data ($HOME/scratch).

All those servers (login, compute, management and storage nodes) :

• run with the GNU/Linux distribution Rocky.
• are inter-connected on high speed InfiniBand network
  • 40Gbit/s (QDR) for Baobab.
  • 100Gbit/s (EDR) for Yggdrasil.
  • 100Gbit/s (EDR) for Bamboo.

In order to provide hundreds of software and versions, we use EasyBuild / module. It allows you to load the exact version of a software/library that is compatible with your code. Learn more about EasyBuild/module

When you want to use some cluster's resources, you need to connect to the login node and submit a job to Slurm which is our job management and queuing system. The job is submitted with an sbatch script (a Bash/shell script with special instructions for Slurm such as how many CPU you need, which Slurm partition to use how long your script will run and how to execute your code). Slurm will place your job in a queue with other users' jobs, and find the fastest way to provide the resources you asked for. When the resources are available, your job will start.
Learn more about Slurm

One important note about Slurm is the concept of partition. When you submit a job, you have to specify a partition that will give you access to some specific resources. For instance, you can submit a job that will use only CPU or GPU nodes.

**CPU and GPU server example pricing **

See below the current price of a compute node (without the extra 15% and without VAT)

The big difference in term of cost isn't the number of cores but the Memory and SSD disk.

The bandwidth lowers if you have empty memory slots:

• 12 memory (i.e. no empty slots) = 460.8 GB/s
• 8 memory (i.e. 4 empty slots) = 307 GB/s

Key differences:

• + 9754 has higher memory performance of up to 460.8 GB/s vs 7763 which has 190.73 GB/s
• + 9754 has a bigger cache
• - 9754 is more expensive
• - power consumption is 400W for 9754 vs 240W for 7763
• - 9754 is more difficult to cool as the inlet temperature for air colling must be 22° max

• 2 x 64 Core AMD EPYC 9554 3.15GHz Processor
• 768GB DDR4 4800MHz ECC Server Memory (24x 32GB / 0 free slots)
• 1 x 7.68TB NVMe Intel 24×7 Datacenter SSD (14PB written until warranty end)
• 4 x nVidia Ampere H100 94GB PCIe GPU (max. 8 GPUs possible)
• ~ 124k CHF HT
• ~ 28,5k CHF HT per extra GPU

• 2 x 64 Core AMD EPYC 9554 3.10GHz Processor
• 384 GB DDR4 4800MHz ECC Server Memory
• 8 x nVidia RTX 4090 24GB Graphics Controller
• ~ 44k CHF HT

We usually install and order the nodes twice per year.

If you want to ask a financial contribution from UNIGE you must complete submit a request to the COINF.

Baobab, our HPC infrastructure, supports educators in providing students with hands-on HPC experience.

Teachers can request access via [dw.unige.ch](final link to be added later, use hpc@unige.ch in the meantime), and once the request is fulfilled, a special account named <PI_NAME>_teach will be created for the instructor. Students must specify this account when submitting jobs for course-related work (e.g., --account=<PI_NAME>_teach).

A shared storage space can also be created optionally, accessible at /home/share/<PI_NAME>_teach and/or /srv/beegfs/scratch/shares/<PI_NAME>_teach.

All student usage is free of charge if they submit their job to the correct account.

We strongly recommend that teachers use and promote our user-friendly web portal at OpenOndDemand which supports tools like Matlab, JupyterLab, and more. Baobab helps integrate real-world computational tools into curricula, fostering deeper learning in HPC technologies.

Everyone has different needs for their computation. A typical example of usage of the cluster would consists of these steps :

• connect to the login node
• this will give you access to the data from your $HOME directory
• execute an sbatch script which will request resources to Slurm for the estimated runtime (i.e. : 16 CPU cores, 8 GB RAM for up to 7h on partition “shared-cpu”). The sbatch will contain instructions/commands :
  • for Slurm scheduler to access compute resources for a certain time
  • to load the right application and libraries with module for your code to work
  • on how to execute your application.
• the Slurm job will be placed in the Slurm queue
• once the requested resources are available, your job will start and be executed on one or multiple compute nodes (which can all access the BeeGFS $HOME and $HOME/scratch directories).
• all communication and data transfer between the nodes, the storage and the login node go through the InfiniBand network.

If you want to know what type of CPU and architecture is supported, check the section For Advanced users.

Both clusters contain a mix of “public” nodes provided by the University of Geneva, a “private” nodes in general funded 50% by the University through the COINF and 50% by a research group for instance. Any user of the clusters can request compute resources on any node (public and private), but a research group who owns “private” nodes has a higher priority on its “private” nodes and can request a longer execution time.

Several CPU models are available across the three clusters. The table below summarizes the available resources.

Several GPU models are available across the three clusters. The table below summarizes the available resources.

See here how to request GPU for your jobs.

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

(baobab)-[root@admin1 slurm] (master *)$

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

In the following table you can see which type of GPU is available on Baobab.

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

Since our clusters are regularly expanded, the nodes are not all from the same generation. You can see the details in the following table.

The “generation” column is just a way to classify the nodes on our clusters. In the following table you can see the features of each architecture.

Click here to Compare Intel CPUS.

In the following table you can see which type of GPU is available on Yggdrasil.

Link to see the GPU details https://developer.nvidia.com/cuda-gpus#compute

In order to follow system ressources, you can go to https://monitor.hpc.unige.ch/dashboards

You can reach node metrics for the last 30 days and BeeGFS metrics for the last 6 months.

For checking resources on a specific node, go to “Baobab - General” or “Yggdrasil - General” and click on “Host Overview - Single”. You will be able to choose the node you want to check in the form at the top :

For going back to the dashboard list, click on the four squares on the left panel :

The “General” dashboard in “Yggdrasil - General” and “Baobab - General” folders gives an overview of the cluster : total load and memory used, and how many nodes are up/down.

You can see GPU metrics too under “Cuda - GPU” dashboards.