I had one of those homelab maintenance tasks that started as a simple disk space check and turned into a useful reminder about how virtual disks actually behave.
The first VM was a self-hosted GitHub Actions runner on Ubuntu. It had accumulated a pile of CI debris: old workspaces, pip caches, Docker BuildKit state, CUDA installers, stale runner update folders, and multiple CUDA toolkit versions. After cleaning it up, Ubuntu showed only about 24 GB used, but the storage backend still needed discard and TRIM before it could reclaim the unused blocks.
Then I repeated the same process on my GLaDOS-01 VM, which runs several Dockerized media and AI workloads. That one was even more satisfying. TrueNAS showed the zvol using 440.09 GiB before TRIM. After enabling discard and running fstrim, it dropped to 125.23 GiB.
That is over 300 GiB reclaimed without moving data, rebuilding the VM, or resizing the guest disk.
Start with ncdu
When a Linux VM starts filling up, I usually start with ncdu:
sudo ncdu -x /
The -x flag matters because it keeps the scan on the same filesystem. That helps avoid accidentally walking into mounted shares, backup repositories, NFS mounts, or other filesystems that are not actually part of the root disk.
On the GitHub Actions runner, ncdu quickly pointed to the usual CI suspects:
/home/administrator/actions-runner/_work
/home/administrator/.cache/pip
/root/.cache/pip
/var/lib/docker/volumes
/usr/local/cuda-*
On GLaDOS-01, it pointed at the usual long-running Docker host suspects:
/var/lib/docker
/root/.cache
/usr/local/cuda-*
/home/administrator/obico-server
/docker/tdarr
That was the important first step. The VMs were not mysteriously wasting space. They were accumulating exactly the kind of debris you expect from self-hosted runners, Docker workloads, local AI experiments, CUDA installs, and media services.
GitHub Actions Runners Need Their Own Cleanup Strategy
Self-hosted GitHub Actions runners are not disposable unless you make them disposable.
GitHub-hosted runners start fresh for each job. A self-hosted runner keeps its workspace, downloaded actions, tool cache, temporary files, and anything else the workflow leaves behind.
On this VM, the runner work directory had grown dramatically:
/home/administrator/actions-runner/_work
Inside it were folders like:
_work/_tool
_work/_actions
_work/_temp
The _tool directory had a cached CUDA installer. _actions had downloaded actions. The regular workspace directories had old checkout and build data.
I added a post-job cleanup hook using ACTIONS_RUNNER_HOOK_JOB_COMPLETED so the runner cleans itself after each job.
The cleanup script ended up looking like this:
#!/usr/bin/env bash
set -euo pipefail
RUNNER_WORK="/home/administrator/actions-runner/_work"
if [[ ! -d "$RUNNER_WORK" || "$RUNNER_WORK" != */actions-runner/_work ]]; then
echo "Invalid runner work path: $RUNNER_WORK"
exit 1
fi
echo "Cleaning GitHub Actions runner workspace: $RUNNER_WORK"
find "$RUNNER_WORK" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
mkdir -p "$RUNNER_WORK/_temp" "$RUNNER_WORK/_actions" "$RUNNER_WORK/_tool"
python3 -m pip cache purge >/dev/null 2>&1 || true
rm -rf /root/.cache/pip 2>/dev/null || true
sudo -u administrator python3 -m pip cache purge >/dev/null 2>&1 || true
rm -rf /home/administrator/.cache/pip 2>/dev/null || true
docker buildx prune -af >/dev/null 2>&1 || true
docker builder prune -af >/dev/null 2>&1 || true
docker container prune -f >/dev/null 2>&1 || true
docker volume prune -f >/dev/null 2>&1 || true
echo "Cleanup complete."
Then I wired it into the runner’s systemd service with a drop-in override:
[Service]
Environment="ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/usr/local/bin/gh-runner-cleanup.sh"
After reloading systemd and restarting the runner, I confirmed the environment variable was present:
systemctl show actions.runner.Jonah-May-OSS.githubrunner01.service -p Environment
That changed the runner from something I had to manually clean into something that cleans itself after every job.
Old Runner Versions and CUDA Installers Were Hanging Around
The runner directory also had old versioned folders:
bin.2.334.0
bin.2.335.1
externals.2.334.0
externals.2.335.1
The active symlinks pointed to the newer version:
readlink -f /home/administrator/actions-runner/bin
readlink -f /home/administrator/actions-runner/externals
Since bin and externals pointed to 2.335.1, the older 2.334.0 folders were safe to remove.
I also found a large CUDA repo installer sitting directly in the runner directory:
cuda-repo-ubuntu2404-13-3-local_13.3.0-610.43.02-1_amd64.deb
That was just a downloaded installer package. Once CUDA was installed, it did not need to stay there.
Those were easy wins:
sudo rm -rf /home/administrator/actions-runner/bin.2.334.0
sudo rm -rf /home/administrator/actions-runner/externals.2.334.0
sudo rm -f /home/administrator/actions-runner/cuda-repo-ubuntu2404-13-3-local_13.3.0-610.43.02-1_amd64.deb
Docker BuildKit Was Holding Space Too
ncdu also showed large Docker volumes under:
/var/lib/docker/volumes
They were Buildx/BuildKit state volumes:
buildx_buildkit_builder-..._state
buildx_buildkit_builder-..._state
Checking Buildx showed multiple persistent builder containers:
docker buildx ls
docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' | grep -i buildkit
There was already a working default builder, so the old-named builders could be removed:
docker buildx rm builder-66b0e136-1469-40c5-a618-fbc807720004
docker buildx rm builder-f5be2147-158f-432e-89a8-f7caac45bdd6
Then I pruned the BuildKit and Docker leftovers:
docker buildx prune -af
docker builder prune -af
docker volume prune -f
For a CI runner, that is a reasonable tradeoff. Builds may lose some cache, but the runner stops accumulating hidden Docker state forever.
CUDA Cleanup
Both VMs had collected old CUDA-related files over time.
On the runner, I found multiple CUDA versions:
/usr/local/cuda-13.2
/usr/local/cuda-13.3
/usr/local/cuda
/usr/local/cuda-13
The symlinks were inconsistent:
/usr/local/cuda -> /usr/local/cuda-13.2
/usr/local/cuda-13 -> /usr/local/cuda-13.3
But the installed packages were CUDA 13.3, and nvidia-smi showed CUDA 13.3 on the driver/runtime side. The old CUDA 13.2 tree was not owned by dpkg, which made it stale local filesystem debris rather than a package-managed install.
I switched the default CUDA symlink to 13.3:
sudo rm -f /usr/local/cuda
sudo ln -s /usr/local/cuda-13.3 /usr/local/cuda
Then verified:
hash -r
which nvcc
readlink -f "$(which nvcc)"
nvcc --version
readlink -f /usr/local/cuda
After that, the old unowned CUDA 13.2 tree could be removed:
sudo rm -rf /usr/local/cuda-13.2
On GLaDOS-01, I did the same kind of cleanup for older CUDA folders, old PyTorch source/build directories, and unused NVIDIA developer tools. The useful pattern was to check package ownership before deleting:
dpkg -S /usr/local/cuda-13.2 2>/dev/null || echo "not owned by dpkg"
If a directory was package-managed, I used apt purge. If it was clearly an old, unowned source or toolkit directory, I removed it directly.
Docker Logs and Caches Add Up on Long-Running Hosts
On GLaDOS-01, Docker was the other major space user. Some of that was expected. Large GPU-enabled containers are not small. Images for Whisper, GLaDOS TTS, llama.cpp, Tdarr, Jellyfin, and Obico can easily add up.
The important part was separating normal image size from avoidable growth.
I checked Docker’s view first:
docker system df
docker system df -v
Then compared that to the filesystem:
sudo du -sh /var/lib/docker/* | sort -hr
For container logs, I added Docker log rotation in /etc/docker/daemon.json:
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"args": [],
"path": "nvidia-container-runtime"
}
},
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3"
}
}
After validating the JSON and restarting Docker:
sudo python3 -m json.tool /etc/docker/daemon.json
sudo systemctl restart docker
Newly created containers inherit the log rotation settings. Existing containers need to be recreated for the default logging options to apply which is easy enough to do since I use Portainer.
Application-Level Cleanup: Tdarr
Some of the cleanup was application-specific.
Tdarr had accumulated job reports and backups:
/docker/tdarr/server/Tdarr/DB2/JobReports
/docker/tdarr/server/Tdarr/Backups
I reduced Tdarr’s job report cap and pruned old reports. Since the VM itself is backed up, I also kept only a short local backup history.
Cleanup Helped, But the Storage Backend Still Needed TRIM
After cleanup, the GitHub runner looked great from inside Ubuntu:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 220G 24G 187G 12% /
GLaDOS-01 also had much less used space after cleanup.
But in a VM, deleting files inside the guest is only half the story. The guest filesystem knows the blocks are free, but the storage backend may not know those blocks can be unmapped.
That matters when the VM disk is backed by sparse or thin-provisioned storage. In my case, the disks lived on TrueNAS-backed zvols presented to Proxmox over NVMe/TCP.
The Proxmox disk config originally lacked discard:
scsi0: truenas-ssd01:vol-vm-116-disk-0...,iothread=1,size=225G
I enabled discard while preserving the existing disk options:
qm set 116 -scsi0 truenas-ssd01:vol-vm-116-disk-0-...,iothread=1,discard=on,size=225G
For GLaDOS-01, the same idea applied, but the disk had a few more options:
scsi0: truenas-ssd01:vol-vm-110-disk-1-...,aio=native,iothread=1,size=500G
So I enabled discard without dropping the existing options:
qm set 110 -scsi0 truenas-ssd01:vol-vm-110-disk-1-...,aio=native,iothread=1,discard=on,size=500G
After rebooting the VM, I ran:
sudo fstrim -av
On the GitHub runner, the output included:
/: 196.2 GiB trimmed on /dev/mapper/ubuntu--vg-ubuntu--lv
On GLaDOS-01, the impact on TrueNAS was dramatic. Before TRIM, TrueNAS showed the zvol using 440.09 GiB. After enabling discard and running fstrim, it dropped to 125.23 GiB.
That reclaimed roughly 315 GiB on the storage backend.
Enable fstrim Going Forward
Once the discard worked, I enabled the periodic trim timer inside Ubuntu:
sudo systemctl enable --now fstrim.timer
systemctl status fstrim.timer --no-pager
That keeps the guest filesystem and storage backend in sync over time, instead of letting deleted blocks remain allocated forever.
Final Result
On the GitHub runner, the root filesystem ended up around:
24G used inside Ubuntu
16.55 GiB used on TrueNAS after trim
On GLaDOS-01, TrueNAS zvol usage dropped from:
440.09 GiB before TRIM
125.23 GiB after TRIM
That is the difference between cleaning files inside the guest and actually reclaiming storage on the backend.
The useful pattern was:
Use ncdu to find the real offenders.
Confirm with du and find before deleting.
Clean the obvious junk.
Automate cleanup for recurring CI debris.
Check package ownership before deleting old toolkits.
Prune Docker and BuildKit caches intentionally.
Enable discard in Proxmox.
Run fstrim inside the guest.
Verify the storage backend actually reclaimed the space.
For long-running Ubuntu VMs, especially self-hosted GitHub Actions runners and Docker-heavy AI/media hosts, that workflow is far more useful than just looking at df -h and guessing.
