I noticed the root drive of my home server (Debian) is at 99% capacity, which was odd to be because I don’t store anything on the root ssd. sudo df -h confirms that 99% of my 256gb drive is full. But sudo du -sh * all added up, only comes to about 30gb.
This is a pc that only runs docker containers and one virtual machine for home assistant. And yes I have restarted, Any ideas as to how to find the missing 200+ gigabytes?
EDIT: sudo ncdu allowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.
If you’re using BTRFS, check if you don’t have any stray snapshots.
What’s your filesystem?
If you are using something like btrfs, for example, the usage reports can be misleading, and getting the exact size can be complicated… I’d recommend using more fs-specific tools, such as btdu.
Would different filesystems really report an over 200gb difference?
Generally no but in realitly it could contribute. some have weird behaviours in how they allocate space so knowing can be useful to rule things out our suggesting gotchas to look for.
In theory, it can. One possible reason with Btrfs might be that you are only mounting a subvolume even though there might be other files in the same filesystem (such as snapshots/copies of the subvolume for backup) but that are not being mounted.
Also some tools like gparted do not handle btrfs disk usage very well and will display it as if the whole partition is 100% full.
One more puzzle piece here is that
duwon’t report on files that have been marked for deletion but are still held on to by some process. There’s anlsofincantation to list those, but I can’t recall it off the top of my head.It used to be part of sysadmin work to detect the processes that held on to large files if
dfreports that you’re running out of space, and restart them to make them let go of the file. But I haven’t done that in ages. And if you restarted the host OS that should have taken care of that.I assume you also know how to prune container resources.
Good call on the docker prune, I didn’t think about that. That accounted for about 25GB. Still not enough but at least I’m not at 99% anymore.
Wouldn’t
du -hs *only check the space used inside the folder you are in?I’d check with
sudo du -hs /*myself if I wanted. Or vedncduto get a visual representation.sudo du -hs /*looked like it began started listing every file on the entire server.
dfreports on filesystems, not drives.du -hs *won’t find “hidden” (.) files and folders, you’ll need a slightly sifferent regx (which I will leave as an exercise for you / I don’t have that info here)And also both
duanddfshow different results depending on the underlying filesystem, ie btrfs (and maybe ZFS?) won’t show how much deduplication is happening.Also, you might be looking at sparse files too, and from memory, you’ll need another option for
duordfto report those correctly.sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information or what’s safe to delete but I’m getting somewhere.Nice. Glad you’re on to something.
I can’t help you with docker files as I don’t use it, but, there’s usually a way to find out who / what owns a file, so I hope docker utils can tell you if it’s safe to delete
If you have enough room to install the
ncducommand, it’s super helpful!Actually running
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.Isn’t that tool freaking nifty? I love ncdu. BTW the
-xflag is useful to make sure you stay on the same file system (useful if you have network shares, extra disks, and to avoid digging stuff like /proc and /sys)
This is a cool tool thanks, unfortunately it is reporting the same (far, far below 99%) number.
$ man fstrimAs @Strit wrote, use sudo, as docker keeps its container, image and volume files under /var/lib/docker, and that folder is not readable without sudo (or root).
I did use sudo
What folder did you run it in?
Sorry I believe I wasn’t actually in the root of the drive!
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.deleted by creator
Docker containers can eat a lot of space over time. When’s the last time you did a
docker system prune? Be sure to read up on what it does before you try it.Thanks that allowed me to clear up about 20GB! Also
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information or if it’s safe to delete but at least I’m getting somewhere.
Check for mounts hiding the underlying drive?
Sometimes
du . -xwill help, too. (-x doesn’t cross mount points).Did du give any permission errors? It can’t count the size of directories that it doesn’t have permission to access.
It did yes, a few “invalid argument” a few “permission denied” and a few “no such file or directory”
UPDATE:
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.
To help you we need to understand your setup
- show us the output of ‘mount’
- show us the output of ‘lsblk’
- show us the output of ‘fdisk -l /dev/sda’
- Do you run snapshots?
EDIT:
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.mountoutputted a lot of text. I suspect this may be getting somewhere but I’m not too experienced. There is a lot ofoverlay on /var/lib/docker/overlay2/[long string of numbers and letters]. I also see a lot (~25) of nsfs on/run/docker/netns/[string of letters and numbers] type nsfs (rw)lsblkgame me my drives as expected, and fdisk gavefdisk: command not found.The point to show the output is to help us understand your system. Not to point directly at the issue. Like for mount this will show what partition is mount where.
You system don’t have fdisk is installed.
I’d be curious to see
du -ito see what’s going on with inodes. Alternatively I did have an issue long time ago with docker containers, sparse files and dirty disk. Force-runningfsckresolved my issues in the past.du -igaveinvalid optionand fsck gavecommand not foundEDIT:
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.that’ll teach me to type in a hurry. I meant
df, notdu. Lookup man page for options
Where are you running
du -sh *? (I.e. what directory, are you definitely scanning the whole file system?) I’m sure it’s obvious, but can never hurt to check!What does
du -sh /show? (Generally, the*glob pattern in the shell will not match hidden dot-files, so is it possible they are being excluded?)EDIT
sudo ncduallowed me to find a 72gb [long string of characterless]-json.log file in /var/lib/docker/containers and many 1gb+ files in /var/lib/docker/overlay2. I’m not sure what to do with this information (or what’s safe to delete) but I’m getting somewhere.~~
sudo du -sh /shows a list of threedu: cannot access '/run/user/1000/gvfs' Permission deniedanddu: cannot access No such file or directoryand a fewcannot read directory invalid argumentand at the end it shows5.4T /which I assume is my root drive combined with what’s inmnt. ~~
