Action

Managing system packages, python, R and many other environments is easy with anaconda!

Note that you want to install miniconda in a network lab directory as it will quickly fill up your individual home directory!

Instructions

Connect to head.its.albany.edu via SSH (see: How-to: Connect via SSH (PuTTY, macOS terminal)
Gain an interactive session by running srun. This will ensure that you are operating on a worker node and not directly on the headnode.
Run the following commands, line by line. Make sure you change [lab_directory] to be your lab directory name.

Interactive Session

srun --partition=batch --propagate=NONE --nodes=1 --time=03:00:00 --cpus-per-task=2 --mem=30G --pty $SHELL -i

You can double check by invoking ‘hostname’ into the terminal. You should see the name of the node you have now spawned on.

Installing Miniconda

# download the install script
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh

# execute the installer
bash ~/miniconda.sh -b -p /network/rit/lab/[lab_directory]/miniconda

# initialize the environment to your shell 
/network/rit/lab/[lab_directory]/miniconda/bin/conda init

# If you are running bash, this is important
cat ~/.bashrc >> ~/.bash_profile

# finally, source your environment, you will only need to do this once. This will only work if your $SHELL is bash
source ~/.bash_profile

Using a centralized conda install for lab members.
1. Using a centralized conda install for lab members.
  1. There are some risks to be aware of, if a lab member updates a conda environment it could break other users code. Best practice is to have one environment per project, it will save you headaches in the long run!
    Steps to allow lab members to access conda
    # Make sure all contents of the folder are g+rwx chmod g+rwx -R /network/rit/lab/[lab_directory]/miniconda/ # now, each lab member must execute the following commands: # initialize the environment to your shell /network/rit/lab/[lab_directory]/miniconda/bin/conda init # If you are running bash, this is important cat ~/.bashrc >> ~/.bash_profile # finally, source your environment, you will only need to do this once. This will only work if your $SHELL is bash source ~/.bash_profile
2. Submitting jobs using sbatch with conda
  Steps to using conda in a shell script
  # To source a conda environment in a shell script add the following after you #SBATCH commands but before your code: source /network/rit/lab/[lab_directory]/miniconda/etc/profile.d/conda.sh conda activate some_env
Some helpful hints
1. Do not install packages in your (base) environment (e.g. conda activate base). It is best to create separate environments for different packages, or conda is at a risk of breaking (e.g. conda create -n tensorflow python=3)
2. Sharing conda across lab group members is possible, make sure the folder is g+rwx (chmod -R g+rwx /network/rit/lab/[lab_directory]/miniconda)
3. Always check if conda has a software package available. There are many different channels, but researchers and software developers commonly publish to conda-forge (e.g.
  conda install -c conda-forge ffmpeg)
4. Read the docs: https://conda.io/projects/conda/en/latest/user-guide/index.html

Lastly, to exit the interactive session, you can do:

ctrl+A+D

Or just scancel your interactive job ID. You can get your job ID by checking what jobs you have running under your own netID.

squeue -u netID
scancel jobID

How-to: Software Management with Anaconda (Software, Python, R, Etc.)

Action

Instructions