Moving DGX Cloud Data to Local Lab Directory

The NVIDIA GPU Cloud Web Interface offers the simplest way to transfer data between DGX Cloud and local systems. For more advanced data transfer needs, particularly when working with lab directories, the NVIDIA GPU Cloud Command Line Interface (NGC CLI) provides a more robust solution. The NGC CLI can be run from:

lmm.its.albany.edu
dgx-head01.its.albany.edu
Your personal machine (requires installation)

This guide demonstrates connecting through lmm.its.albany.edu. For instructions on running NGC CLI from your personal machine, see How-to: NVIDIA DGX Cloud.

This guide explores multiple approaches to downloading data (workspaces and datasets), allowing you to choose the method that best fits your requirements. The options are presented in order of complexity, from the straightforward approach (A) to more sophisticated solutions (B & C).

Index

Prerequisites

Active VPN connection (if accessing from off-campus) - please refer to: VPN (Virtual Private Network)
SSH client (Terminal for macOS, PuTTY for Windows) - please refer to How-to: Connect via SSH

Downloading Workspaces

Workspaces can be dowloaded from both the web interface and the command line. From the command line, you can either use ngc workspace download or use ngc workspace mount to mount the volume and move the data back and forth.

Option A - Download Workspace from NGC Web Interface

The simplest and most straightforward one, recommended when you have a small workspace (under 10GB) and a small number of objects. Just open your workspace and click the Download button in the top right corner of the page.

Option B - Download Workspace using NGC CLI Download Command

Recommended when you have a larger workspace (above 10GB) and a small number of objects.

Step 01 - Connect to LLM Server

Connect to lmm.its.albany.edu via SSH using your NetID:

macOS: Use Terminal command ssh <NetID>@lmm.its.albany.edu
Windows: Use PuTTY

Step 02 - Navigate to Your Lab Directory

Navigate to your lab directory.

cd /network/rit/lab/<your_lab>

Then use the ngc workspace download command to download the workspace to this directory. You can either download the workspace as a folder with all the subdirectories and files or you can download it as a zip file when using the --zip flag.

# To download as a folder
ngc workspace download <your_workspace>

# To download as a zip file
ngc workspace download --zip <your_workspace>

Option C - Download Workspace using NGC CLI Mount Command

Recommended when you have a larger workspace (above 10GB) and a larger number of objects as this option allows you to resume the synchronization in case something goes wrong. The following video provides a step-by-step walkthrough of this tutorial.

Step 01 - Connect to LLM Server

Connect to lmm.its.albany.edu via SSH using your NetID:

macOS: Use Terminal command ssh <NetID>@lmm.its.albany.edu
Windows: Use PuTTY

Step 02 - Navigate to Your Lab Directory and Mount Workspace

Navigate to your lab directory (required for proper NGC workspace mounting) and from there create a directory to mount your workspace to.

cd /network/rit/lab/<your_lab>
mkdir ngc-mount

Then use the ngc workspace mount command to mount the workspace to this new directory. The --mode RW flag enables both read and write access, allowing data transfer in both directions.

Step 03 - Transferring Your Data

Basic File Access

Access your NGC remote workspace through the ngc-mount directory.
Transfer files as needed between local and cloud storage.

The ngc-mount directory is only a mount point to your cloud workspace - files in this directory are not physically stored on your lab directory. To permanently store files on your lab directory, you must copy them from the ngc-mount directory to your lab parent directory or another local directory.

Syncing Directories Using rsync

For efficient directory synchronization, rsync is recommended. Here's how to sync your cloud workspace to a local backup:

Create a backup directory: mkdir ngc-backup
Sync the directories with progress tracking: rsync -r ngc-mount/ ngc-backup/ --progress

The --progress flag allows you to monitor the transfer in real-time through the terminal.

If you are not familiar with rsync there is a very useful tutorial available here.

Step 04 - Cleanup

Remember to unmount the workspace when finished.

Lastly, make sure to verify your transfer:

The ngc-mount folder will appear empty after unmounting.
Your ngc-backup folder should contain all synced files.

If you have any questions on how to use ngc mount or ngc unmount, please refer to the official NVIDIA documentation on this topic.

Downloading Datasets

Datasets can be dowloaded from both the web interface and the command line using ngc dataset download.

Option A - Download Dataset from NGC Web Interface

The simplest and most straightforward one, recommended when you have a small dataset (under 10GB). Just open your dataset and click the Download button in the top right corner of the page.

Option B - Download Dataset using NGC CLI Download Command

Recommended when you have a larger dataset (above 10GB).

Step 01 - Connect to LLM Server

Connect to lmm.its.albany.edu via SSH using your NetID:

macOS: Use Terminal command ssh <NetID>@lmm.its.albany.edu
Windows: Use PuTTY

Step 02 - Navigate to Your Lab Directory

Navigate to your lab directory.

Then use the ngc dataset download command to download the dataset to this directory. Please note that unlike the workspace that you provided the name, for the dataset you have to provide its ID, which is conveniently located in the bottom left corner of the dataset page in the NGC web interface. You can either download the dataset as a folder with all the subdirectories and files or you can download it as a zip file when using the --zip flag.