Instance states
An instance moves through DEPLOYING (machine being prepared — image pull, storage, GPU attach), RUNNING (billing applies), STOPPED (kept for restart, not billed), FAILED (the deploy never came up, or the host failed — see recovery below), and TERMINATED (deleted). INSUFFICIENT_FUNDS means the wallet could not cover the next hour; top up and restart.
Out-of-memory (OOM) behavior
When a process exhausts the instance's RAM limit, only that process is killed — not the whole container. Your SSH session, Jupyter server, and other processes keep running; the killed job simply exits. If the main process itself is the one killed, the container restarts in place: /workspace and /vault are untouched, and anything the previous container had written elsewhere (installed packages, /home, /root) is recovered automatically into /workspace/.recovered/ within seconds.
Keep long-running checkpoints in /workspace or /vault anyway — they are the storage that is contractually yours.
Data recovery after a host outage
If the machine hosting your instance loses power or goes down, your data is preserved on the machine itself — /workspace, installed packages, and any files you kept elsewhere in the container (a /training-data, /opt, home directories): everything except /proc-style runtime paths. Recovered non-workspace files appear under the rootfs/ (or layer/) subfolder of the recovery folder. What happens next depends on how long the outage lasts:
- Under 24 hours: when the machine returns, your instance is rebooted on it automatically and your files appear under /workspace/.recovered — billing resumes when the instance starts
- 24 hours or longer: your preserved files are moved into your vault under recovered/<machine>-<date>/ and are FREE of storage charges for 7 days; after that the folder counts as normal vault usage
Files already in /vault are never affected by a host outage — vault storage does not live on the GPU machine.
Recovered folders are a courtesy copy with a 7-day free window — review them promptly, keep what you need, and delete the rest to avoid storage charges.