llama.cpp + Chat UI (HF) on DGX On-Prem: Another Personal Local LLM Experience

1. Introduction

Welcome to your guide to running powerful large language models (LLMs) right here on UAlbany's DGX On-Prem system! This tutorial will walk you through setting up llama.cpp (a high-performance LLM inference engine) along with Chat UI (a user-friendly HuggingFace interface) on our high-performance computing infrastructure.

What will you get out of this?

Your own private LLM environment running on UAlbany's powerful GPUs
A clean, intuitive web interface to interact with state-of-the-art models
Complete privacy - all your prompts and data stay within the university network
The ability to run models like Phi-3-mini-4k (and others) without sending sensitive data to external services

Think of this as running your own personal ChatGPT-like service, but with the processing happening right here on campus instead of on someone else's servers. It's perfect for research projects, exploring AI capabilities, or just learning about how these systems work!

2. Prerequisites

Before we dive in, let's make sure you have everything you need.

Required Access

Access to UAlbany's DGX On-Prem cluster
- If you don't have access yet, you'll need to request it through your faculty advisor or principal investigator by completing the DGX On-Prem Computation Request Form

Knowledge Requirements

Basic familiarity with terminal/command line (don't worry, we'll guide you through each step)
Basic understanding of SLURM job submission (if you've never used SLURM before, check out the How to Schedule via SLURM page)

Files Needed

Just the script provided in this tutorial (llamacpp.slurm) - the script will take care of all the required dependencies
You'll be able to copy-paste it directly or download it

3. Understanding What We're Building

Before we jump into the technical details, let's understand what we're setting up.

llama.cpp: llama.cpp is a highly efficient inference engine for running LLMs on consumer hardware. Originally designed for Meta's Llama models, it now supports a wide variety of models through the GGUF format. It's C/C++ based with a focus on performance and efficiency. Think of llama.cpp as the "engine" of our setup - it's what actually processes your prompts and generates responses.

Chat UI (HuggingFace): While llama.cpp is powerful, interacting with it solely through API calls isn't very user-friendly. That's where Chat UI comes in - it's a sleek, web-based interface from HuggingFace that makes interacting with your LLM as simple as using a website. It gives you a ChatGPT-like experience but powered by your chosen models running on our infrastructure.

MongoDB: The Chat UI requires a database to store conversation history and user settings. We're using MongoDB for this purpose.

How They Work Together: When you run our script, it sets up all these components on the same DGX node. llama.cpp runs as a server in the background handling all the AI processing, Chat UI provides a user-friendly way to chat with the AI, MongoDB stores your conversations, and JupyterLab gives you access to the system for maintenance or development (but we won't focus much on that part).

4. The Script Explained

The script is available here: . Please go ahead and download it to your computer and open it. Let's demystify what our script is doing! While it might look complex at first glance, it's essentially just automating all the setup steps that would be tedious to do manually.

#!/bin/bash

#SBATCH --job-name=llamacpp
#SBATCH --output=llamacpp-%j.out
#SBATCH --error=llamacpp-%j.err
#SBATCH --time=8:00:00
#SBATCH --gpus=1
#SBATCH --container-image='docker://nvcr.io#nvidia/pytorch:25.03-py3'
#SBATCH --container-mounts=/network/rit/dgx/dgx_vieirasobrinho_lab:/mnt/dgx_lab,/network/rit/lab/vieirasobrinho_lab:/mnt/lab

This top section is telling SLURM (our job scheduler) how to set up the environment.

We're naming our job "llamacpp"
We're requesting one GPU for 8 hours
We're using NVIDIA's PyTorch container (which has all the ML libraries pre-installed)
We're connecting our lab's storage so files persist after the job ends

The middle section includes several clever functions that check if components already exist.

# Function to check if demo folder already exists
check_demo_folder() {
    if [ -d "${demo_location}" ]; then
        return 0
    fi
    return 1
}

# Function to check if llama.cpp is already installed
check_llamacpp() {
    if [ -f "${demo_location}/llama.cpp/build/bin/llama-server" ]; then
        return 0
    fi
    return 1
}

These functions save time by not reinstalling software that's already there. It's like checking if you already have ingredients before going shopping. The main part of the script is comprised of the following steps.

Sets up random ports for the services.
Creates a demo folder if needed.
Downloads and builds llama.cpp if it's not already installed.
Starts the llama.cpp server with the Phi-3-mini-4k-instruct model.
Sets up MongoDB for conversation history.
Downloads and configures Chat UI from HuggingFace.
Configures JupyterLab as an additional tool.

When the script finishes running, it displays URLs and login information for accessing your services.

5. Step-by-Step Tutorial

Now let's put this into action! By the way, if you need some visual assistance, take a look at the video below and see how to get this tutorial working.

As you may notice, the first time you run the script might take a while as there is a ton of dependencies do be downloaded.
This is specially because it takes usually half an hour to compile the llama.cpp binaries.
When you run the script a second time, it will run much faster as the dependencies will be already in place.

a. Access the DGX System

First, connect to the DGX On-Prem cluster.

ssh your_netid@dgx-head01.its.albany.edu

You'll need to enter your NetID password. If you're off-campus, make sure you're connected to the VPN first.

b. Create and Save the Script

Once logged in, navigate to your lab folder and create a new file for our script.

vim llamacpp.slurm

This opens the vim text editor. Press i to enter insert mode, then copy-paste the entire script content (provided at the end of this wiki) into the editor.

Press Esc to exit insert mode, then type :wq and press Enter to save and exit.

c. Customize the Script

Before running the script, you need to modify it for your specific lab directories.

Look for the following line (near the top).

#SBATCH --container-mounts=/network/rit/dgx/dgx_vieirasobrinho_lab:/mnt/dgx_lab,/network/rit/lab/vieirasobrinho_lab:/mnt/lab

Change it to your lab's paths as follows.

#SBATCH --container-mounts=/network/rit/dgx/dgx_YOUR_LAB_NAME:/mnt/dgx_lab,/network/rit/lab/YOUR_LAB_NAME:/mnt/lab

Replace YOUR_LAB_NAME with your actual lab name.

d. Submit the Job

Now, submit your job to SLURM using the following command.

sbatch llamacpp.slurm

SLURM will assign your job a number and start it when resources are available. You can check its status with the following command.

squeue -u your_netid

e. Check the Output

Once your job starts running, you can monitor its progress by different ways (such as the following).

tail -f llamacpp-*.out

This shows the output in real-time. You'll see messages as llama.cpp, MongoDB, and Chat UI are set up.

f. Get Connection Information

When setup is complete (usually takes 5-10 minutes depending on download speeds), the output file will contain URLs and login information.

cat llamacpp-*.out

Look for a section that looks like the following.

================================================================================

llama.cpp is available at: http://dgx05.its.albany.edu:8369

Chat UI is available at: http://dgx05.its.albany.edu:8903

================================================================================

Note: Your URLs will have different port numbers since they're randomly generated.

6. Accessing and Using Your Services

Now that everything is set up, it's time to start using your local LLM!

Using Chat UI

Copy the Chat UI URL from your output file and paste it into your browser (don't panic if the first time you open take a bit longer).
You'll see a user interface similar to ChatGPT.
Start chatting! The model (Phi-3-mini-4k-instruct by default) is already configured and ready to use.

Making API Calls Directly to llama.cpp

If you prefer programmatic access, you can interact with llama.cpp directly - remember to use your own URLs and port numbers.

Using curl

curl --request POST \
    --url http://dgx06.its.albany.edu:8369/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Explain quantum computing in simple terms:","n_predict": 128}'

Using Python

import requests

def ask_llamacpp(prompt, n_predict=128):
    url = "http://dgx06.its.albany.edu:8369/completion"
    headers = {"Content-Type": "application/json"}
    payload = {
        "prompt": prompt,
        "n_predict": n_predict
    }
    
    response = requests.post(url, json=payload, headers=headers)
    return response.json()["content"]

# Example usage
answer = ask_llamacpp("Explain quantum computing in simple terms:")  # Changed prompt to match curl example
print(answer)

JupyterLab Access (For Maintenance)

You can access JupyterLab using the URL and password provided in the output file. This is primarily useful if you need to troubleshoot or modify files directly.

7. Advanced Customization

Using Different Models

Loading Different Models from HuggingFace

The script currently loads Phi-3-mini-4k-instruct, but you can easily modify it to use other models. Edit the script and find the following line.

nohup ${demo_location}/llama.cpp/build/bin/llama-server --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf --hf-file Phi-3-mini-4k-instruct-q4.gguf --n-gpu-layers 32 --port ${llamacpp_port} > /dev/null 2>&1 &

Replace the --hf-repo and --hf-file parameters with any model from HuggingFace that has GGUF versions available - e.g., the TheBloke/Mistral-7B-Instruct-v0.2-GGUF.

# For example, to use Mistral-7B
nohup ${demo_location}/llama.cpp/build/bin/llama-server --hf-repo TheBloke/Mistral-7B-Instruct-v0.2-GGUF --hf-file mistral-7b-instruct-v0.2.Q4_K_M.gguf --n-gpu-layers 32 --port ${llamacpp_port} > /dev/null 2>&1 &

You can see the full list of available GGUF models at HuggingFace. If you want to run multiple models at once, you can modify the script to start multiple llama.cpp instances (one for each model, therefore you’ll also need to generate a port number for each one of them) and tweak the .env.local file creation to include these different endpoints.

Adjusting Resource Requests

Requesting Multiple GPUs

For larger models or better performance, modify the following line in the script.

#SBATCH --gpus=1

Change it as needed.

#SBATCH --gpus=2 # or 4, 8, etc.

Note that requesting more GPUs means you might wait longer for your job to start.

Extending Runtime

Need more than 8 hours? Modify the following line.

#SBATCH --time=8:00:00

Change it as needed.

#SBATCH --time=24:00:00 # for 24 hours

Persistent Storage Strategies

The script saves downloaded models to the following path.

${demo_location}/models

This folder persists between sessions, so models are only downloaded once. You can manage storage in different ways.

To check model sizes, run the following command.

du -h ${demo_location}/models

To Remove unused models, simply delete them from the models directory.

If you're working with multiple large models, be mindful of your lab storage quota.

8. Cleanup and Next Steps

Ending Your Session

Your SLURM job will automatically terminate after the time specified (default: 8 hours). If you want to end it early, just try the following steps.

Find your job ID.

squeue -u your_netid

Cancel the job.

scancel job_id

Saving Your Work

Conversations in Chat UI although stored in the MongoDB database are not persistent between jobs - they only persist for the time the script runs. This is because we’re saving all the data to the /tmp directory of the container (--dbpath /tmp/mongodb), so every time a container gets spawned a new database is created.

Next Steps for Exploration

Once you're comfortable with the basic setup, consider the following steps.

Experimenting with different models to compare performance.
Integrating llama.cpp into your research workflow.
Exploring the different parameters of llama.cpp for optimizing inference.
Contributing to the llama.cpp project on GitHub.

9. External Resources

10. Full Script Reference

Here's the complete script () for easy copying.

#!/bin/bash

#SBATCH --job-name=llamacpp
#SBATCH --output=llamacpp-%j.out
#SBATCH --error=llamacpp-%j.err
#SBATCH --time=8:00:00
#SBATCH --gpus=1
#SBATCH --container-image='docker://nvcr.io#nvidia/pytorch:25.03-py3'
#SBATCH --container-mounts=/network/rit/dgx/dgx_vieirasobrinho_lab:/mnt/dgx_lab,/network/rit/lab/vieirasobrinho_lab:/mnt/lab

# Function to check if demo folder already exists
check_demo_folder() {
    if [ -d "${demo_location}" ]; then
        return 0
    fi
    return 1
}

# Function to check if llama.cpp is already installed
check_llamacpp() {
    if [ -f "${demo_location}/llama.cpp/build/bin/llama-server" ]; then
        return 0
    fi
    return 1
}

# Function to check if Node.js is already installed
check_nodejs() {
  if command -v node >/dev/null 2>&1; then
    return 0
  fi
  return 1
}

# Function to check if MongoDB is already installed
check_mongodb() {
  if command -v ${demo_location}/mongodb/bin/mongod >/dev/null 2>&1; then
    return 0
  fi
  return 1
}

# Function to check if Chat UI is already installed
check_chatui() {
    if [ -d "${demo_location}/chat-ui" ]; then
        return 0
    fi
    return 1
}

# Get the DGX node name
node_name="$SLURMD_NODENAME"

echo -e "\nThe llama.cpp + Chat UI (HuggingFace) Demo is starting..."

# Generate random port numbers between 8000 and 8999
llamacpp_port=$((RANDOM % 1000 + 8000))
mongodb_port=$((RANDOM % 1000 + 8000))
chatui_port=$((RANDOM % 1000 + 8000))
jupyter_port=$((RANDOM % 1000 + 8000))

# Build the URLs
llamacpp_url="http://${node_name}.its.albany.edu:${llamacpp_port}"
chatui_url="http://${node_name}.its.albany.edu:${chatui_port}"
jupyter_url="http://${node_name}.its.albany.edu:${jupyter_port}"

# Setup demo location
demo_location="/mnt/dgx_lab/llamacpp-demo"

# Check and create demo folder if needed
echo -e "\nChecking for demo folder..."
if ! check_demo_folder; then
    echo -e "\nNot found: creating demo folder..."
    mkdir -p ${demo_location}
    echo -e "\n🔵 Demo folder created at ${demo_location}."
else
     echo -e "\n⚪ Demo folder already exists at ${demo_location}."
fi

# Setup llama.cpp models directory
mkdir -p ${demo_location}/models
export HF_HOME=${demo_location}/models

# Download llama.cpp
echo -e "\nChecking for llama.cpp installation..."
if ! check_llamacpp; then
    echo -e "\nNot found: downloading llama.cpp..." 
    git clone https://github.com/ggml-org/llama.cpp ${demo_location}/llama.cpp
    git config --global --add safe.directory ${demo_location}/llama.cpp
    cd ${demo_location}/llama.cpp
    cmake -B build -DGGML_CUDA=ON
    cmake --build build --config Release
    echo -e "\n🔵 llama.cpp downloaded, extracted and built to ${demo_location}/llama.cpp."
else
    echo -e "\n⚪ llama.cpp already installed at ${demo_location}/llama.cpp."
fi

# Start llama.cpp
echo -e "\nStarting llama.cpp server..."
nohup ${demo_location}/llama.cpp/build/bin/llama-server --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf --hf-file Phi-3-mini-4k-instruct-q4.gguf --n-gpu-layers 32 --host 0.0.0.0 --port ${llamacpp_port} > /dev/null 2>&1 &
echo -e "\n🟢 llama.cpp server started on ${llamacpp_url}."

# Download Node.js
echo -e "\nChecking for Node.js installation..."
if ! check_nodejs; then
    # echo -e "\nNot found: downloading Node.js..." 
    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.2/install.sh | bash
    export NVM_DIR="$HOME/.nvm"
    [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
    [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion
    nvm install node
    echo -e "\n🔵 Node.js downloaded and installed."
else
    echo -e "\n⚪ Node.js already installed."
fi

# Download MongoDB
echo -e "\nChecking for MongoDB installation..."
if ! check_mongodb; then
    echo -e "\nNot found: downloading MongoDB..." 
    mkdir -p ${demo_location}/mongodb && \
    curl -s https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.6.0.tgz | \
    tar -xz --strip-components=1 -C ${demo_location}/mongodb
    echo -e "\n🔵 MongoDB downloaded, extracted and built to ${demo_location}/mongodb."
else
    echo -e "\n⚪ MongoDB already installed."
fi

# Start MongoDB
echo -e "\nStarting MongoDB server..."
mkdir -p /tmp/mongodb
nohup ${demo_location}/mongodb/bin/mongod --port ${mongodb_port} --dbpath /tmp/mongodb > /dev/null 2>&1 &
echo -e "\n🟢 MongoDB server listening on port ${mongodb_port}."

# Download Chat UI
echo -e "\nChecking for Chat UI installation..."
if ! check_chatui; then
    echo -e "\nNot found: downloading Chat UI..." 
    git clone https://github.com/huggingface/chat-ui ${demo_location}/chat-ui
    npm install --prefix ${demo_location}/chat-ui
    # Fix package.json
    sed -i 's/"dev": "vite dev"/"dev": "vite dev --host 0.0.0.0 --no-open"/' ${demo_location}/chat-ui/package.json
    echo -e "\n🔵 Chat UI downloaded and extracted to ${demo_location}/chat-ui."
else
    echo -e "\n⚪ Chat UI already installed."
fi

# Create .env.local
rm -f ${demo_location}/chat-ui/.env.local
cat > ${demo_location}/chat-ui/.env.local << EOF
MONGODB_URL=mongodb://localhost:${mongodb_port}
MODELS=\`[
{
    "name": "microsoft/Phi-3-mini-4k-instruct",
    "endpoints": [{
    "type" : "llamacpp",
    "baseURL": "http://localhost:${llamacpp_port}"
    }],
},
]\`
EOF

# Fix vite.config.ts
temp_file=$(mktemp)
awk -v port="$chatui_port" '
BEGIN { 
  in_server_block = 0; 
  port_found = 0;
  allowedHosts_found = 0;
}
{
  if ($0 ~ /server: {/) {
    in_server_block = 1;
    print $0;
  } 
  else if (in_server_block && $0 ~ /port:/) {
    # Replace existing port line
    print "\tport: " port;
    port_found = 1;
  }
  else if (in_server_block && $0 ~ /allowedHosts:/) {
    # Keep the existing allowedHosts line
    print $0;
    allowedHosts_found = 1;
  }
  else if (in_server_block && $0 ~ /},/) {
    # End of server block, add port and allowedHosts if needed
    if (!allowedHosts_found) {
      print "\tallowedHosts: true,";
    }
    if (!port_found) {
      print "\tport: " port;
    }
    in_server_block = 0;
    print $0;
  }
  else {
    print $0;
  }
}' ${demo_location}/chat-ui/vite.config.ts > "$temp_file"
mv "$temp_file" ${demo_location}/chat-ui/vite.config.ts

# Start Chat UI
echo -e "\nStarting Chat UI server..."
nohup npm run dev --prefix ${demo_location}/chat-ui > /dev/null 2>&1 &
sleep 30
echo -e "\n🟢 Chat UI server started on ${chatui_url}."

# Generate a random password for JupyterLab (alphanumeric, 6 characters)
jupyter_password=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 6)

# Print session details
echo -e "\n================================================================================\n"
echo -e "llama.cpp is available at: ${llamacpp_url}\n"
echo -e "Chat UI is available at: ${chatui_url}"
echo -e "\n================================================================================\n"
echo -e "JupyterLab is available at: ${jupyter_url}\n"
echo -e "Your password is: ${jupyter_password}\n"
echo -e "Please copy and paste the link into your browser and use the password to log in."
echo -e "\n================================================================================\n"

# Start JupyterLab session
jupyter lab --allow-root --no-browser --NotebookApp.token="${jupyter_password}" --NotebookApp.allow_origin='*' --NotebookApp.log_level='CRITICAL' --notebook-dir=/mnt --port=$jupyter_port

Remember to replace YOUR_LAB_NAME with your actual lab name before using the script.

Happy LLM-ing!

llama.cpp + Chat UI (HF) on DGX On-Prem: Another Personal Local LLM Experience

Table of Contents

1. Introduction

2. Prerequisites

3. Understanding What We're Building

4. The Script Explained

5. Step-by-Step Tutorial

a. Access the DGX System

b. Create and Save the Script

c. Customize the Script

d. Submit the Job

e. Check the Output

f. Get Connection Information

6. Accessing and Using Your Services

Using Chat UI

Making API Calls Directly to llama.cpp

Using curl

Using Python

JupyterLab Access (For Maintenance)

7. Advanced Customization

Using Different Models

Loading Different Models from HuggingFace

Adjusting Resource Requests

Requesting Multiple GPUs

Extending Runtime

Persistent Storage Strategies

8. Cleanup and Next Steps

Ending Your Session

Saving Your Work

Next Steps for Exploration

9. External Resources

10. Full Script Reference