Versions Compared

Old Version 1

changes.mady.by.user Karker, Nicholas

Saved on

compared with

New Version Current

changes.mady.by.user Vieira Sobrinho, Jose

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The NVIDIA GPU Cloud Command Line Interface (NGC CLI) provides Web Interface offers the simplest way to transfer data between DGX Cloud and local systems. You can run NGC CLI For more advanced data transfer needs, particularly when working with lab directories, the NVIDIA GPU Cloud Command Line Interface (NGC CLI) provides a more robust solution. The NGC CLI can be run from:

  • lmm.its.albany.edu

  • dgx-head01.its.albany.edu

  • Your personal machine (requires installation)

Panel
panelIconIdatlassian-info
panelIcon:info:
bgColor#F4F5F7

This guide demonstrates connecting through lmm.its.albany.edu. For instructions on running NGC CLI from your personal machine, see How-to: NVIDIA DGX Cloud.

The following video provides a step-by-step walkthrough of this tutorial.

...

This guide explores multiple approaches to downloading data (workspaces and datasets), allowing you to choose the method that best fits your requirements. The options are presented in order of complexity, from the straightforward approach (A) to more sophisticated solutions (B & C).

Index

Prerequisites

Downloading Workspaces

Workspaces can be dowloaded from both the web interface and the command line. From the command line, you can either use ngc workspace download or use ngc workspace mount to mount the volume and move the data back and forth.

Option A - Download Workspace from NGC Web Interface

The simplest and most straightforward one, recommended when you have a small workspace (under 10GB) and a small number of objects. Just open your workspace and click the Download button in the top right corner of the page.

...

Option B - Download Workspace using NGC CLI Download Command

Recommended when you have a larger workspace (above 10GB) and a small number of objects.

Step 01 - Connect to LLM Server

Connect to lmm.its.albany.edu via SSH using your NetID:

  • macOS: Use Terminal command ssh <NetID>@lmm.its.albany.edu

  • Windows: Use PuTTY

Step 02 - Navigate to Your Lab Directory

Navigate to your lab directory.

Code Block
languagebash
cd /network/rit/lab/<your_lab>

Then use the ngc workspace download command to download the workspace to this directory. You can either download the workspace as a folder with all the subdirectories and files or you can download it as a zip file when using the --zip flag.

Code Block
# To download as a folder
ngc workspace download <your_workspace>

# To download as a zip file
ngc workspace download --zip <your_workspace>

Option C - Download Workspace using NGC CLI Mount Command

Recommended when you have a larger workspace (above 10GB) and a larger number of objects as this option allows you to resume the synchronization in case something goes wrong. The following video provides a step-by-step walkthrough of this tutorial.

...

Step 01 - Connect to LLM Server

Connect to lmm.its.albany.edu via SSH using your NetID:

  • macOS: Use Terminal command ssh <NetID>@lmm.its.albany.edu

  • Windows: Use PuTTY

Step 02 - Navigate to Your Lab Directory and Mount Workspace

Navigate to your lab directory (required for proper NGC workspace mounting) and from there create a directory to mount your workspace to.

...

Code Block
ngc workspace mount <your_workspace> ./ngc-mount --mode RW

Step 03 - Transferring Your Data

Basic File Access

  • Access your NGC remote workspace through the ngc-mount directory.

  • Transfer files as needed between local and cloud storage.

Panel
panelIconIdatlassian-info
panelIcon:info:
bgColor#F4F5F7

The ngc-mount directory is only a mount point to your cloud workspace - files in this directory are not physically stored on your lab directory. To permanently store files on your lab directory, you must copy them from the ngc-mount directory to your lab parent directory or another local directory.

Syncing Directories Using rsync

For efficient directory synchronization, rsync is recommended. Here's how to sync your cloud workspace to a local backup:

...

Panel
panelIconIdatlassian-info
panelIcon:info:
bgColor#F4F5F7

If you are not familiar with rsync there is a very useful tutorial available here.

Step 04 - Cleanup

Remember to unmount the workspace when finished.

...

If you have any questions on how to use ngc mount or ngc unmount, please refer to the official NVIDIA documentation on this topic.

Downloading Datasets

Datasets can be dowloaded from both the web interface and the command line using ngc dataset download.

Option A - Download Dataset from NGC Web Interface

The simplest and most straightforward one, recommended when you have a small dataset (under 10GB). Just open your dataset and click the Download button in the top right corner of the page.

...

Option B - Download Dataset using NGC CLI Download Command

Recommended when you have a larger dataset (above 10GB).

Step 01 - Connect to LLM Server

Connect to lmm.its.albany.edu via SSH using your NetID:

  • macOS: Use Terminal command ssh <NetID>@lmm.its.albany.edu

  • Windows: Use PuTTY

Step 02 - Navigate to Your Lab Directory

Navigate to your lab directory.

Code Block
languagebash
cd /network/rit/lab/<your_lab>

Then use the ngc dataset download command to download the dataset to this directory. Please note that unlike the workspace that you provided the name, for the dataset you have to provide its ID, which is conveniently located in the bottom left corner of the dataset page in the NGC web interface. You can either download the dataset as a folder with all the subdirectories and files or you can download it as a zip file when using the --zip flag.

Code Block
# To download as a folder
ngc dataset download <your_dataset_id>

# To download as a zip file
ngc dataset download --zip <your_dataset_id>