...
Make sure your data is not zipped or archived. If your data is zipped or in .tar format, it will upload as such and will not be as accessible on the cloud. Unzip/untar your data before uploading. The option under --source should be the file or folder that contains your files. The --desc option has a descriptor and name section. Lastly the --share option will designate to which team you want to upload the data to if you are a part of multiple teams. Add a 2nd --share option if you want to upload to another team at the same time.
| Code Block |
|---|
ngc dataset upload --source <dir> --desc "my data" <dataset_name> --share <team_name> |
...
| Code Block | ||
|---|---|---|
| ||
ngc dataset upload --source world_happiness --desc "csvs of world happiness data ranging from 2015 to 2019" world_happiness --share awan_lab |
Now if you return to the base command dashboard, and look under 'Datasets', you should see your just uploaded file/folder. You can upload more files in the same fashion and they will appear in the same manner. The data you upload in this fashion is immutable, thus you do not need to worry about accidentally editing the data during your work. In practice, you should make a copy of the data when executing code, and have the code interface with the copy. This would be accomplished in a manner such as:
| Code Block | ||||
|---|---|---|---|---|
| ||||
import pandas as pd
df = pd.read_csv("/mount/data/folder/name_of_your_data.csv")
print(df) |
In this fashion, your original data is never truly changed, ensuring reproducibility of your work. If you are doing data cleaning, you can do so in a workspace and save it as a new csv, or clean the data locally if that is easier.
From your lab folder:
uhhh figure this out, maybe install nvidia CLI somewhere on lmm?
Starting a Job
The default time limit of a job is 30 days. There are 2 ways to start a job on DGX cloud, via web interface or via CLI. The web interface is graphical, easy to use, and also generates a CLI prompt for you to use if you wish. In this example, we will submit a job to launch a jupyternotebook instance where we can access our data from inside the notebook.
From the Web Interface:
In the create job section, there are templates created by people in your team, Nvidia, or UAlbany ITS. In this example, we'll be using a template to launch a jupyter notebook session. Here we see a templates tab, and one template available for us to use. The name of this template is tf_3.1_jupyter_notebook, uses 1 GPU, 1 node, and a container image made by Nvidia for tensorflow 3.1.
Once you load the template, you can edit the options in the web interface to suit your needs. You can swap the number of GPUs/nodes or container type to better suit your computing needs. Scroll down to 'Inputs' and you can see which datasets and workspaces you can load. You can load multiple of each.
From the CLI:
| Code Block |
|---|
ngc batch run --name "Job-univ-of-albany-iad2-ace-622835" --priority NORMAL --order 50 --preempt RUNONCE --min-timeslice 2592000s --total-runtime 2592000s --ace univ-of-albany-iad2-ace --instance dgxa100.80g.1.norm --commandline "jupyter lab --allow-root -port=8888 --no-browser --NotebookApp.token='' --NotebookApp.allow_origin='*' --notebook-dir=/" --result /results --image "nvaie/tensorflow-3-1:23.03-tf1-nvaie-3.1-py3" --org tt6xxv6at61b --port 8888 |