Text to Image with NVIDIA Visual Generative AI NIM

This tutorial leverages a container that houses Stable Diffusion 3.5 Large model that produces high-quality images, with Depth and Canny ControlNets offering controllability over image outputs. Choose this model when you want high quality, artistic style, and more fine-tuning flexibility. This tutorial is based on:

NVIDIA official documentation: https://docs.nvidia.com/nim/visual-genai/latest/getting-started.html?model=stable-diffusion-3.5-large.
NGC container image: https://catalog.ngc.nvidia.com/orgs/nim/teams/stabilityai/containers/stable-diffusion-3.5-large?version=1.0.0.
NVIDIA model card: https://build.nvidia.com/stabilityai/stable-diffusion-3_5-large/modelcard.
Stability AI release notes: https://stability.ai/news/introducing-stable-diffusion-3-5.

Buckle up, because this is going to be one of the coolest demos so far!

Step 1 - NGC Authentication

Since this is a self-hosted NVIDIA Inference Micro-service (NIM) available under the NVIDIA AI Enterprise (NVAIE) License, an NGC API key is required.

Note: This requires a Personal Key, different from the Legacy Key used in Enroot configurations.
For Enroot setup, refer to: DGX On-Prem How-To

To generate your NGC API Personal Key, visit: Generate Personal Key. When creating your key, make sure to select “NGC Catalog” under the “Services Included” dropdown. Additional services can be included if the key will be reused for other purposes. For detailed information on API key types: https://docs.nvidia.com/ngc/gpu-cloud/ngc-user-guide/index.html#ngc-api-keys. The key will have a prefix nvapi-. Keep it securely, as it will be used in the next step.

Step 2 - Hugging Face Authentication

To access Stable Diffusion 3.5 Large model read and accept Stable Diffusion 3.5 Large, Stable Diffusion 3.5 Large TensorRT and Stable Diffusion 3.5 Large ControlNet TensorRT License Agreements and Acceptable Use Policy.

Create a new Hugging Face token with Read access to contents of all public gated repos you can access permission. The token will have a prefix hf_. Once again, keep it securely, as it will be used in the next step.

Step 3 - Launching the NIM

The official NVIDIA documentation provides Docker-based instructions, but since DGX environments use Enroot as the container runtime, a SLURM script is needed to translate Docker parameters to the appropriate Pyxis/Enroot flags. Below is a template SLURM script to launch the Visual Generative AI NIM.

#!/bin/bash

#SBATCH --job-name=stable-diffusion
#SBATCH --output=%j.out
#SBATCH --time=8:00:00
#SBATCH --gpus=1
#SBATCH --container-image='docker://nvcr.io/nim/stabilityai/stable-diffusion-3.5-large:1.0.0'
#SBATCH --container-writable

# SLURM Parameters
SLURM_NODE_NAME="$SLURMD_NODENAME"
SLURM_JOB_ID="$SLURM_JOB_ID"

# NIM API Ports
NIM_HTTP_API_PORT=8000

# Construct URLs
NIM_HTTP_API_URL="http://${SLURM_NODE_NAME}.its.albany.edu:${NIM_HTTP_API_PORT}"

# Display URLs and job info
echo -e "\n================================================================================\n"
echo -e "SLURM Job ID ${SLURM_JOB_ID} running on ${SLURM_NODE_NAME}\n"
echo -e "HTTP URL: ${NIM_HTTP_API_URL}"
echo -e "\n================================================================================\n"

# Set NGC API Key (replace YOUR_API_KEY with your actual API key)
export NGC_API_KEY="YOUR_API_KEY"

# Set Hugging Face Token (replace YOUR_HUGGING_FACE_TOKEN with your actual token)
export HF_TOKEN="YOUR_HUGGING_FACE_TOKEN"

# Start the NIM server
sh /opt/nim/start_server.sh

You can copy the contents from this snippet and save it as stable-diffusion.slurm or you can simply download it from here: .

Connect to the DGX On-Prem cluster over SSH (ssh your_netid@dgx-head01.its.albany.edu). If off-campus, ensure VPN is connected before logging in.
Download and edit the above SLURM script for your use:
- Replace YOUR_LAB_NAME with your lab's actual name.
- Replace YOUR_API_KEY with the NGC API key (including nvapi- prefix).
- Replace YOUR_HUGGING_FACE_TOKEN with the Hugging Face token (including hf_ prefix).
Submit the job to SLURM (sbatch stable-diffusion.slurm). SLURM will assign and queue your job according to resource availability.
Monitor job status (squeue -u YOUR_NETID).
Watch job progress by checking the output file (tail -f YOUR_JOB_ID.out). Look for an entry similar to the following, as you will use the HTTP URL on the next step.

================================================================================
SLURM Job ID 23390 running on dgx22
HTTP URL: http://dgx22.its.albany.edu:8000
================================================================================

Step 4 - Running Inference

On your local machine, open a terminal and run the following command (replace http://dgx22.its.albany.edu:9000 with your HTTP URL):

invoke_url="http://dgx22.its.albany.edu:8000/v1/infer"

output_image_path="result.jpg"

response=$(curl -X POST $invoke_url \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -d '{
          "prompt": "A simple coffee shop interior.",
          "mode": "base",
          "seed": 0, 
          "steps": 30 
        }')
response_body=$(echo "$response" | awk '/{/,EOF-1')
echo $response_body | jq .artifacts[0].base64 | tr -d '"' | base64 --decode > $output_image_path

Look for a file named result.jpg and open it. You should expect to see something like the following.

Example 1

Example 2

Explore the official documentation for more details and parameters: https://docs.nvidia.com/nim/visual-genai/latest/getting-started.html?model=stable-diffusion-3.5-large. This tutorial provides a solid foundation to get started with NVIDIA NIM for Visual Generative AI on DGX systems. Feel free to experiment with client parameters and models to tailor the experience to your needs.