eResearch Doc

User Tools

Site Tools

You are here: start » hpc » applications_and_libraries
hpc:applications_and_libraries

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

hpc:applications_and_libraries [2025/04/11 09:17] – [Conda] Adrien Alberthpc:applications_and_libraries [2025/06/11 12:27] (current) – external edit 127.0.0.1
Line 316: Line 316:
  
  
-Using **Conda** directly on HPC systems or shared servers can cause performance issues and storage bloat because Conda environments create thousands of small files. This often results in:+Using **Conda** directly on HPC systems or shared servers can cause performance issues and storage overload because Conda environments create thousands of small files. This often results in:
  
   * Slow job startup times   * Slow job startup times
-  * File system limitations being hit+  * Filesystem limitations being hit
   * High I/O load on the cluster   * High I/O load on the cluster
   * Complex environment management   * Complex environment management
  
-A better solution is to **encapsulate Conda environments inside a container**. This way, the entire environment is packaged into a single file (such as a `.sif` image used by Apptainer/Singularity), which can be easily deployed, shared, and reused without polluting the cluster's filesystem.+A better solution is to **encapsulate Conda environments inside a container**. This way, the entire environment is packaged into a single file (such as a `.sif` image used by Apptainer/Singularity), which can be easily deployed, shared, and reused without polluting clusters.
  
  
Line 338: Line 338:
  
  
-This guide explains how to build such a container using [`cotainr`](https://hpc-community.unige.ch/t/new-software-installed-cotainr-version-2025-3-0/3917), a tool that simplifies container creation.+This guide explains how to build such a container using [[https://hpc-community.unige.ch/t/new-software-installed-cotainr-version-2025-3-0/3917|cotainr]], a tool that simplifies container creation.
  
  
 === Step 1 – Define the Conda Environment === === Step 1 – Define the Conda Environment ===
-Create a file `env.yml` that contains the definition of your environment: +Create a file ''env.yml'' that contains the definition of your environment: 
-(As exemple we will use `bioenv.yml`)+(As exemple we will use ''bioenv.yml'')
  
 <code bash> <code bash>
Line 363: Line 363:
 [...] [...]
  
-prefix:/home/users/a/alberta/env+prefix:/home/users/a/alberta/env     # <==== To delete/edit
 </code> </code>
  
Line 371: Line 371:
 # 1. (optional) create your environment (or not if you already have one) # 1. (optional) create your environment (or not if you already have one)
 $ conda create -n bioenv -c bioconda -c conda-forge spades exonerate diamond blast mafft trimal numpy joblib scipy -y $ conda create -n bioenv -c bioconda -c conda-forge spades exonerate diamond blast mafft trimal numpy joblib scipy -y
 +
 # 2. Activate your environment # 2. Activate your environment
 $ conda activate bioenv $ conda activate bioenv
 +
 # 3. Export the settings of your environment  # 3. Export the settings of your environment 
-#    It’s recommended to manually remove the `prefix:` line at the bottom of the file before using it with cotainr.+# It’s recommended to manually remove the `prefix:` line at the bottom of the file before using it with cotainr.
 $ conda env export > bioenv.yml $ conda env export > bioenv.yml
 </code> </code>
  
-<note tip></note>+
  
 === Step 2 – Build the Container === === Step 2 – Build the Container ===
-Now use `cotainr` to create the image:+Now use ''cotainr'' to create the image:
  
 <code bash> <code bash>
Line 389: Line 391:
 </code> </code>
  
-You can replace `ubuntu:latest` with any other base image, such as `rockylinux:latest`.+You can replace ''ubuntu:latest'' with any other base image, such as ''rockylinux:latest''.
  
 === Step 3 – Use the Container === === Step 3 – Use the Container ===
Line 403: Line 405:
  
  
-=== References === + 
-  * [cotainr tutorial](https://hpc-community.unige.ch/t/tutorial-use-conda-in-a-container/3529) + 
-  * [Apptainer documentation](https://apptainer.org/docs/)+ 
 + 
 +==== Conda environment management ====
  
  
Line 413: Line 417:
 module load Anaconda3 module load Anaconda3
 </code> </code>
- 
- 
-==== Conda environment management ==== 
- 
  
 Create Create
Line 1150: Line 1150:
 With the Baobab upgrade to CentOS 7 (cf. https://hpc-community.unige.ch/t/baobab-migration-from-centos6-to-centos7/361 ) we do not provide anymore a central RStudio. With the Baobab upgrade to CentOS 7 (cf. https://hpc-community.unige.ch/t/baobab-migration-from-centos6-to-centos7/361 ) we do not provide anymore a central RStudio.
  
-Instead, you can download the upstream Open Source binary RStudio Desktop version (cf. https://rstudio.com/products/rstudio/download/ ) and directly use it, here the instructions: +Instead, we provide Rstudio on [[hpc:how_to_use_openondemand|OpenOnDemand]].
- +
-  - install it in your ''${HOME}'' folder: <code console> +
-capello@login2:~$ mkdir Downloads +
-capello@login2:~$ cd Downloads +
-capello@login2:~/Downloads$ wget ${URL_FOR_rstudio-${VERSION}-x86_64-fedora.tar.gz} +
-[...] +
-capello@login2:~/Downloads$ tar axvf rstudio-${VERSION}-x86_64-fedora.tar.gz +
-[...] +
-capello@login2:~/Downloads$  +
-</code> +
-  - launch an interactive graphical job: +
-    - connect to the cluster using [[hpc:access_the_hpc_clusters#gui_accessdesktop_with_x2go|GUI access / Desktop with X2Go]] or using ''ssh -Y'' from a machine with an X server such as [[hpc:access_the_hpc_clusters#from_linux_and_mac_os|Linux or Mac]]. \\  +
-    - start an interactive session on a node (see [[hpc/slurm#interactive_jobs|Interactive Slurm jobs]]): <code console> +
-capello@login2:~$ salloc -p debug-cpu -n 1 -c 16 --x11 +
-salloc: Pending job allocation 39085914 +
-salloc: job 39085914 queued and waiting for resources +
-salloc: job 39085914 has been allocated resources +
-salloc: Granted job allocation 39085914 +
-capello@node001:~$  +
-</code> Doing so, you will have 16 cores on one node of the partition ''debug-cpu'' for a max time of 15 minutes. Specify the appropriate duration time, partition, etc. like you would do for a normal job. +
-    - load one of the R version supported by RStudio, for example:<code console> +
-capello@node001:~$ module spider R/3.6.0 +
- +
----------------------------------------------------------------------------------- +
-  R: R/3.6.0 +
----------------------------------------------------------------------------------- +
-    Description: +
-      R is a free software environment for statistical computing and +
-      graphics.  +
- +
- +
-    You will need to load all module(s) on any one of the lines below +
-    before the "R/3.6.0" module is available to load. +
- +
-      GCC/8.2.0-2.31.1  OpenMPI/3.1.3 +
-[...] +
-capello@node001:~$ module load GCC/8.2.0-2.31.1  OpenMPI/3.1.3 +
-capello@node001:~$ module load PostgreSQL/11.3-Python-3.7.2 +
-capello@node001:~$ module load R/3.6.0 +
-capello@node001:~$  +
-</code> +
-    - run RStudio : <code console> +
-capello@node001:~$ ~/Downloads/rstudio-${VERSION}/bin/rstudio +
-</code> +
- +
-<note important>Latest version of Rstudio needs an aditional dependency loaded +
-<code> +
-module load PostgreSQL/11.3-Python-3.7.2 +
-</code> +
-</note> +
 ==== R packages ==== ==== R packages ====
  
hpc/applications_and_libraries.1744363077.txt.gz · Last modified: (external edit)