GPU Node: Rules of Use
Why these rules
The node has 96 CPU cores and 2 GPUs shared by everyone in the group. One job that grabs all cores or stacks processes on a busy GPU blocks the whole node for everyone.
Per-user CPU, memory and GPU limits are enforced by the system. Code that ignores these rules gets throttled, fails, or is killed. Questions go to the node admin before you start, not after.
Before you start a job
Check what is already running, every time:
nvidia-smi # which GPU is free, who is on it, how much memory
btop # current CPU and RAM load
Start your job only on a GPU with no other process on it. If both GPUs are busy, wait or ask the people using them. Never start on a busy GPU because it “still has free memory”.
GPU rules
Never hardcode a GPU (
cuda:1,device=1,gpu_id=0). Choose the free GPU when you launch, and let your code usecuda:0:CUDA_VISIBLE_DEVICES=1 python train.py # runs on physical GPU 1, appears as cuda:0One process per GPU. Do not start a process on a GPU that already runs someone else’s job, and do not stack several of your own processes on one card without asking first.
CPU-only jobs must not touch the GPU. PyTorch, TensorFlow and JAX create a GPU context (and TF/JAX reserve memory) as soon as they see a GPU, including in every joblib or multiprocessing worker. Hide the GPUs:
CUDA_VISIBLE_DEVICES="" python my_cpu_job.pyMulti-GPU jobs (both cards) only after agreeing on a time slot with the admin.
CPU and memory rules
At most 32 cores per user, all your jobs together. Need more for a short run? Ask first.
Never use “all cores”. No
n_jobs=-1, noos.cpu_count(), nonum_workers=96. Read the cores you are actually allowed to use:import os n_cores = len(os.sched_getaffinity(0))Limit library threads. Every worker process of joblib, multiprocessing or a DataLoader starts its own BLAS/OpenMP threads, which multiplies quickly. Set this before starting parallel jobs:
export OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1More threads than cores does not make your job faster. It makes everyone’s job slower, including yours.
Memory: watch your RAM in
btop. If your job needs more than 100 GB, tell the admin before starting it.
Clean up after yourself
Shut down Jupyter kernels you are not using (Kernel → Shut Down, not just closing the tab). An idle kernel keeps its GPU memory until it is stopped.
No idle GPU processes. A process holding GPU memory without computing for more than a few hours will be killed without notice.
Check before you log off:
nvidia-smi --query-compute-apps=pid,used_memory --format=csv # compare PIDs with the list below ps -u $USER -o pid,etime,pcpu,args | head -20Crashed or finished runs that left workers behind:
pkill -u $USER -f your_script_name