Monitoring and Tracking

“Is the GPU busy right now?” and “which run produced this checkpoint?” are answered by different tools, and mixing them up is why people end up with neither.

• Resource monitoring: live utilization, VRAM, RAM, and disk. Answers whether the hardware you are paying for is actually working. Nothing to install.
• Experiment tracking: loss curves, hyperparameters, and artifacts, kept after the instance is gone. That is what Weights & Biases is for.
• Model and dataset management: pulling weights in and pushing results out, with the cache somewhere that survives the instance. That is Hugging Face plus /vault.

Every running instance reports live utilization on the Instances page — no agent, no setup. The same numbers are on the API, which is the honest source for how much of your rented allocation is in use:

A cell prefixed with ! runs a shell command, so the standard tools work unchanged. nvidia-smi is present on every GPU image:

!nvidia-smi

For something readable while a job runs, nvitop gives a live view with per-process attribution — useful when you want to know whether it is your training loop or a forgotten notebook kernel holding the VRAM. Run it in a terminal tab rather than a cell, since it does not exit:

!pip install -q nvitop
# then, in a Jupyter terminal:
nvitop

Inside the training loop, ask the framework instead — it reports what your process holds rather than what the driver sees:

import torch

free, total = torch.cuda.mem_get_info()
print(f"in use: {(total - free) / 1e9:.1f} / {total / 1e9:.1f} GB")
print(f"peak this process: {torch.cuda.max_memory_allocated() / 1e9:.1f} GB")

W&B records hyperparameters, metrics, and system stats for each run and keeps them on their servers, so the history outlives the instance. Log in once with a token from wandb.ai/authorize:

%pip install -q wandb

import os, wandb
# WANDB_DIR is a ROOT — wandb creates its own wandb/ inside it, so runs
# land in /vault/wandb/. Default is ./wandb next to the script, which dies
# with the instance.
os.environ["WANDB_DIR"] = "/vault"
wandb.login()  # or set WANDB_API_KEY as an instance environment variable

Then wrap the run. Logging GPU memory alongside your loss is worth the one extra line — it is what tells you afterwards whether a batch size was actually safe:

run = wandb.init(project="my-project", config={"lr": 3e-4, "batch_size": 32})

for epoch in range(epochs):
    loss = train_one_epoch()
    run.log({
        "loss": loss,
        "gpu_mem_gb": torch.cuda.max_memory_allocated() / 1e9,
    })

run.finish()  # flushes the last metrics — a killed kernel loses them

The setting that matters most on GPUtw is where the cache lives. By default huggingface_hub caches under ~/.cache/huggingface, which is inside the container — so every new instance re-downloads the same weights. Point HF_HOME at /vault and you download once, ever:

%pip install -q -U huggingface_hub

import os
os.environ["HF_HOME"] = "/vault/hf"                 # cache survives the instance
os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"         # saturate the link on big pulls
os.environ.setdefault("HF_TOKEN", "")               # set at deploy time, not here

Pull a model, and push results back:

from huggingface_hub import snapshot_download

path = snapshot_download("Qwen/Qwen2.5-7B-Instruct")

# after training
model.push_to_hub("my-org/my-model", private=True)

/vault is network storage. That is what makes it persistent and shared across your instances, and it is also why the first read of a large file is slower than local disk. The rule that follows:

• Weights you load once: keep on /vault. Loading a checkpoint at startup is a one-time cost, and never re-downloading it is worth far more.
• Data read every epoch: copy to /workspace first. Local disk, and the copy pays for itself within an epoch or two.

A WANDB_API_KEY or HF_TOKEN pasted into a cell is saved into the .ipynb file, and travels with it into /vault, into any repo you push, and into anything you share. Pass them as environment variables at deploy time instead — the Environment variables field on the deploy page, or env on the create API.

Know what that means: those values are kept with the instance record so the deploy form can prefill them next time, and they are not encrypted at rest. It is better than a notebook — nothing copies them onward — but it is not a secret manager. Use a scoped, short-lived token where the provider offers one (Hugging Face fine-grained tokens, a W&B service account) rather than an account-wide key.