Small percentage of clients losing free space and having boot issues

[prodservices]

I've got a very strange issue none of us have seen, so I'm posting to see if it's unique to our environment or if anyone else has seen it.

-All clients are 10.12.6, hardware is Mini's, Retina's (13 & 15) and current MBP 13" No touchbar, as well as an iMac...so no commonalities so far as hardware...
-Some clients shipped with Sierra, others were upgraded through self-service using the technique for offering Sierra featured on DerFlounder's page (thanks again Rich, you're awesome).
-Affected clients (relatively small number, less than 10% of our fleet) have issues booting - they hang during boot (progress bar gets about 1/3 of the way) although I've had success getting them to boot more reliably with a PRAM reset first. If left alone, some clients will eventually boot after several hours.
-Some clients are hard freezing and need a reboot where things will work okay for 5-15 minutes, then freeze again.
-Here's where it get's weird - the remaining space on the disk you can watch disappear as the client sits idle. It looks like it's being used by the system, but there's nothing there. Honestly it looks like a massive swapfile but I cannot actually locate where it's at. But you can literally look at a finder window with status enabled and watch the free space tick away.
-Re-image and things (so far) are working as expected on a test client.

Has anyone run into something like this? it's very strange and we're all kind of flummoxed since there's been no new policies rolled out except some very basic security remediation, which would have affected at least 125+ more clients if that was the rogue policy causing this strangeness...

Thanks all if anyone is able to offer any insight.

[Sonic84]

if you can get your hands on a system with the issue, attach it to a Mac workstation via Target Disk Mode, then try to find out what is using all the space. You can do so visually with programs like Grand Perspective or by revealing hidden items and enabling "calculate all sizes" in Finder. When you find the source, you can hopefully determine the culprit by its location, file name, or contents.

[AVmcclint]

Back in my Mac Genius days, I encountered a user whose iBook had 50MB free on its 80GB hard drive. After purging 1GB of cache via Single User Mode so it would boot up again, I installed DiskInventoryX (the predecessor to Grand Perspective) and found a couple 30GB log files. Apparently a runaway process just barfed all over her hard drive until it wouldn't boot up any more. The process went unchecked for weeks because she only put her computer to sleep. She never shut it down or rebooted - which would have reset the process and stopped the excessive logging.

Fast forward to today, and I have an advanced search saved in JSS that displays the free disk space so I can prevent things like that from happening again. I also have a smart group set to email me when someone's drive gets over a certain percentage.

[AVmcclint]

addendum: If you find that the culprit is some process that's filling your drives with log files, the next step will be to identify the process and figure out why it's gone crazy. With the 30GB log files, I had to use the tail command to read the last 100 lines or so since they were too large to open in any kind of text editor.

[dan-snelson]

@prodservices FWIW, the following post includes a script you can run via Self Service to determine disk usage: Disk Usage Report Self Service Policy

[bentoms]

@prodservices Are these Macs multi-user & running Google Drive, per chance?

[seann]

sudo du -chd 1 /

[prodservices]

Hi Everyone,

Thanks for the comments and suggestions - we've figured out what's happening but not why:

Machines that are off the network for an extended period 8+ weeks are having the directory /Library/Application Support/JAMF/tmp/ filled up with .tmp files, some as large as 700 GB! This is having an impact as we have a number of professors that are taking laptops down from the shelf to start classes.

If we look at he polices ran since coming back online and the only one that is different is the CIS audit/remediation set, and it looks like they only ran 4 or 5 times. Are these known for creating lots of data?

[KatieE]

@prodservices I've heard a couple reports of those policies running away, but have never had luck tracking that down. In a recent update I actually added local logging to help trace if something went wrong. Would love to gather more data on that if you're able to provide it.

[prodservices]

Hi Katie,

Thanks for the offer - I'm afraid that we've suspended those polices for now, just to get us through the first few weeks of classes, and then plan a controlled reintroduction.

Where/what should we be looking for?