Airborne SMCE Parallel Cluster

The Airborne SMCE parallel cluster is a SLURM-based high-performance computing cluster that operates alongside JupyterHub. It provides users with access to additional compute resources beyond what is available in JupyterHub alone.

The two environments are fully integrated, allowing you to seamlessly read from and write to your JupyterHub home directory and shared storage without encountering permission issues.

To access the Airborne SMCE Parallel Cluster, contact an administrator to have your user added to the pcluster group.

Logging in for the First Time

Once you have been added to the pcluster group, your first login to the cluster must be through JupyterHub.

  1. Navigate to the Airborne SMCE.

  2. Select and start a server.

  3. During server startup, your PCluster user will be automatically configured, including the SSH configuration.

  4. Once inside JupyterLab, open a terminal and run: ssh pcluster

You will be connected to the PCluster head node.

Your PCluster user and your JupyterHub user are the same, allowing you to read and write files on both systems. The PCluster has a separate home directory, but you can also access your JupyterHub home directory as well as the shared drive via:

  • /efs/home/<username>

  • /efs/shared

Mounted S3 buckets can be found at /s3.

Non-mounted S3 buckets can be access via the aws cli. To list available buckets: aws s3 ls. Not all listed buckets can be used!

Logging in from Your Local Machine (Optional)

After your first login via JupyterHub, you can set up direct SSH access to the PCluster from your local machine.

  1. On your local machine, generate an SSH key pair: ssh-keygen -t ed25519 -f pcluster_key -N ""

This will create a private and a public SSH key.

  1. Open your public key (.pub file) and copy the contents to the PCluster:

  • Start a JupyterHub server.

  • Open a terminal and log into the PCluster: ssh pcluster

  • Navigate to your SSH directory: cd ~/.ssh

  • Edit the authorized_keys file: nano authorized_keys

  • Paste your public key on a new line, then save and exit (Ctrl+S followed by Ctrl+X).

  1. Now you can SSH directly from your local machine:

ssh -i <path-to-your-private-key> <username>@pcluster.airborne.smce.nasa.gov

This setup allows you to securely access the PCluster without having to go through JupyterHub each time.

Using the Pcluster

Compute Options

The cluster has 4 different partitions for use:

  • demand-8cpu

  • demand-16cpu

  • spot-8cpu

  • spot-16cpu

The 8cpu instances have 16GB of memory and the 16cpu instances have 32GB.

The spot instances are much cheaper to use, but they can be interrupted by AWS and should be used for testing purposes only. The demand instances cost more, but will run as long as you need them to.

You can list the different partitions by running: sinfo

Using the Module System

To list the available modules you can run module avail

To load modules run module load <module-name>

Using Conda Environments on the Cluster

Your pcluster user has access to the same conda environments as the Jupyterhub. New environments can be created through the Conda Store. DO NOT CREATE CONDA ENVIRONMENTS ON THE HEAD NODE VIA CONDA CLI.

To access your environments you must first load the miniconda module: module load miniconda3.

If this is your first time using the miniconda3 module run: conda init

If you have used the module before you can run source ~/.bashrc

Now, everything is read for you to use conda!

Running a SLURM JOB

There are many ways to use a SLURM-based cluster. Below are a few common usage patterns to help you get started. For more detailed and advanced workflows, refer to the official SLURM documentation.

Submitting a Job Using sbatch

Jobs are typically submitted to the scheduler using a batch script. Start by creating a script that specifies the required resources and the commands your job will execute.

#!/bin/bash
#SBATCH --partition=demand-8cpu # choose from the partitions above
#SBATCH --ntasks=1 # run one task with using all compute resources
#SBATCH --cpus-per-task=8 # use all available cpus
#SBATCH --mem=15GB # The memory needs to be slightly lower than the node max
#SBATCH --job-name=slurm_test
#SBATCH --output=slurm-%j.out
#SBATCH --error=slurm-%j.err

# Loads modules
module load miniconda3

# configure conda
conda init
source ~/.bashrc

# activate your conda env
conda activate global-airborne-core

# Run your python code
python #<your script here>

Now that you created your script, submit it to run on the cluster using sbatch:

sbatch <path-to-your-script>

Running an Interactive Job

Another common and useful way to work on the cluster is by running an interactive job. Interactive jobs allow you to request compute resources and work directly on a compute node in real time.

To start an interactive job, you can specify all required compute resources directly in an srun command and include the --pty bash argument. This tells SLURM to allocate the requested resources and open an interactive shell on the compute node once the allocation is granted.

When the resources become available, you will be placed directly onto the allocated compute node and can begin working immediately.

srun \
--partition=demand-8cpu \
--ntasks=1 \
--cpus-per-task=8 \
--mem=15G \
--time=01:00:00 \ # 1 hour of time HH:MM:SS
--job-name=slurm_test \
--pty bash

Other Useful Commands

  • squeue Shows the status of all jobs in the queue.

  • squeue -u <username> Shows only jobs submitted by a specific user.

  • scancel -j <job-id> Cancels a specific job.

  • scancel -u <username> Cancels all jobs submitted by a specific user.

  • sinfo Displays information about available partitions and node states.