Policy Inadvertently Erasing Significant Data

[donkruse]

We've experienced a serious issue where nearly 90 out of about 100 Mac computers have had enough data erased from their volumes, including any attached or mounted network volumes, that they are non-functional. It wasn't a complete volume wipe but a random (seemingly) brute force file deletion all across the local volume(s). macOS version or format did not seem to matter. Additionally, the time zone gets set to UTC instead of what it should be and passwords stop working (we're assuming this is because of the file deletions).

This does seem to tie in with a specific policy running, but its a policy that installs an app via a pkg. Its an ARD modified installer pkg for FileMaker Pro 18 Advanced using the—flawed—instructions from Claris/FileMaker.

We have since enlisted the help of a Cybersecurity team to help us investigate but we are wondering if anyone else has experienced something similar? We don't yet know if we screwed something up, if our Jamf Pro cloud server messed up, if Jamf messed up their backend, or if this was something malicious.

In the attached screenshot note that it isn't the FileMaker policy that is failing its because most the data on the local drive has been deleted, including Jamf.

We

[adamcodega]

It's probably all frozen now but how was the policy configured? Run pkg, submit inventory, that it? What instructions did you follow to create the FileMaker Pro Advanced 18 pkg?

[mark_buffington]

Is it all Catalina-specific like in the screenshots? Any difference when installing the custom package by double-clicking in the macOS GUI?

[GabeShack]

That "removing local copy" line is interesting. I am wondering if because its Catalina, there is something not quite right with how you are setting up the image originally possibly? Write access is blocked for Catalina library and system files, but perhaps the pkg installers are breaking one of the symlinks somehow which then looks like things are missing?

Can you open the pkg and look at any scripts that run during the install?

Gabe Shackney
Princeton Public Schools

Sidenote, we are staying away from Catalina for now probably at least until next year, or we may just skip it altogether. We did that with High Sierra as well.

[mm2270]

Can you deconstruct that FileMaker Pro pkg using something like Suspicious Package, or the Terminal and pkgutil --expand maybe? I'm guessing there is some postinstall or preinstall script in the package you can examine that may be doing something unintended. I'm not sure what you mean exactly by an "ARD modified installer pkg", but maybe some type of script was embedded in it that doesn't work right when pushed via a Jamf Pro policy.

[donkruse]

We used the instructions provided by Filemaker to create a network distributable installer.

link text following the steps for the ARD install. This is how we've integrated this with Jamf Pro in the past. The FileMaker instructions are obviously copied-and-pasted from earlier versions so there are some things that are simply incorrect but otherwise this creates a package that can be distributed via ARD and this does not cause an issue.

The issue is only caused after it has been uploaded to our Jamf cloud and then gets installed via a policy.

The policy does nothing other than install. Thanks for the tip mm2270 about checking the jamf downloaded package.

Also this affected all the macOS version we have running: 10.13, 10.14, and 10.15.

[donkruse]

The only script is a post-install script:

#!/bin/sh
PACKAGE_PATH=${1}
TARGET_LOCATION=${2}
if [ -e "$TARGET_LOCATION/FileMaker Pro 18 Advanced.pkg" ]
then
installer -package "$TARGET_LOCATION/FileMaker Pro 18 Advanced.pkg" -target /
fi
if [ -e "$TARGET_LOCATION" ]
then
rm -dfR "$TARGET_LOCATION"
fi

Here's the rest....

PackageInfo:

```
<?xml version="1.0" encoding="utf-8"?>
<pkg-info overwrite-permissions="true" relocatable="false" identifier="com.filemaker.ardwrapper.FileMakerPro18Advanced.pkg" postinstall-action="none" version="0" format-version="2" generator-version="InstallCmds-714 (19C57)" install-location="/tmp/fmp_kNENYefmxh" auth="root"> <payload numberOfFiles="9" installKBytes="285074"/> <bundle-version/> <upgrade-bundle/> <update-bundle/> <atomic-update-bundle/> <strict-identifier/> <relocate/> <scripts> <postinstall file="./postinstall"/> </scripts>
</pkg-info>
I can't open the Bom or Payload with the tools I have (fresh post-disaster laptop missing all my previous tools).

[mm2270]

I think that script is the problem. It's using a $TARGET_LOCATION that is defined as ${2}, but in a Jamf policy, $2 becomes Computer Name. Parameters 1-3 are predefined as mount point, computer name and user name. The package/script seems like it was written to be run by hand, or some method where you can pass parameters 1 and 2 to the script. Are those being defined anywhere in another location that you can find? If not, Jamf Pro already has those script parameters defined for itself.
Since the script is running a rm -dfR on $2 or $TARGET_LOCATION, it seems likely that it's deleting files on the Mac somehow instead of the script's intended target.

This is just an educated guess. I can't be sure, and I'm surely not going to test it out to see. But it does seem like that's your problem.

[GabeShack]

And the policy history on the affected computers show no other policies being run that could affect it? (Also nothing else in the policy that is being triggered?).

Can you look at the Install.log in the log reports section in console(if 10.15) for the same specific time as the data loss occurs?

I've not seen anything that specifically erases data in a jamf policy, except at one point Google File Stream was deleting data on our macs so we disabled the install for that software as Google had acknowledged the issue.

Gabe Shackney
Princeton Public Schools

[GabeShack]

Just saw @mm2270 's reply, that looks like he may have nailed it down. Funny considering FileMaker is owned by Apple, you'd think they have more up to date install instructions for this aside from using Remote Desktop.

Gabe Shackney
Princeton Public Schools

[andrew_nicholas]

I'm with @gshackney and @mm2270, judging by the variable names, whoever wrote the script flipped PACKAGE_PATH and TARGET_LOCATION (never even calling PACKAGE_PATH), and so when they did the "clean up" they did everything at / as root recursively, and if it wasn't SIP protected it was napalmed. Screen cap of a test I did with a simple package to spit out that if you used Composer to make the package, $2 is target path which seems to default to "/", which would include connected volumes. I think this might be an RGE more than a targeted attack.

[donkruse]

@mm2270 I think you're onto something. I don't have time to test this today but we will try testing it with the script as is (on systems we can wipe) and modify it to just hard-code in the variables. In our environment they don't need to be variables because every system is set up the same.

Thanks @gshackneyt and @andrew-nicholas for testing this. We just didn't catch this in testing so we've changed our testing process (mostly to slow it down and require more documentation).

I think that script is the problem. It's using a $TARGET_LOCATION that is defined as ${2}, but in a Jamf policy, $2 becomes Computer Name. Parameters 1-3 are predefined as mount point, computer name and user name. The package/script seems like it was written to be run by hand, or some method where you can pass parameters 1 and 2 to the script. Are those being defined anywhere in another location that you can find? If not, Jamf Pro already has those script parameters defined for itself. Since the script is running a rm -dfR on $2 or $TARGET_LOCATION, it seems likely that it's deleting files on the Mac somehow instead of the script's intended target.

[donkruse]

FYI there weren't any logs remaining locally on any of the affected computers we inspected. On the few computers we started into Recovery Mode to examine log files all the usual log locations were either missing or the entire path was gone. Logs on our Jamf server showed the FileMaker policy as being the last thing run but with a 'failure'.

[bpavlov]

Postinstall scripts inside a pkg are not effected by Jamf script parameters.

[mm2270]

@bpavlov You may very well be right about that. I had some second thoughts after I posted what I did above, on whether the Jamf script parameters would be overriding anything in the postinstall script. But even if that is true, I still think the script isn't properly defining what the target of the rm command should be. I don't see how $TARGET_LOCATION is being properly defined anywhere. It may just be defaulting to / as the location to start recursively rm'ing files, which would explain why some important files are being nuked on the device.

I guess what I'm saying is, a packages payload by itself can't really delete files. Payloads are for dropping files into place, not removing them. So it stands to reason something in the script may be causing files to be deleted. Would you agree?

[bpavlov]

@mm2270 The logic is well thought out and I'd agree that there's gotta be something in there that might be causing the issue. My comment was short and specific since I didn't want the idea that Jamf script parameters applied to a package's postinstall script to continue propagating.

I imagine if it were indeed the postinstall script that was causing the issue that others would be seeing the same. I know FileMaker Pro isn't as popular as something like Microsoft Office, but it obviously gets used enough that we would have heard of other similar reports I think. So I'm not entirely convinced that this is the cause of the @donkruse 's issue. But I also don't have all the information they have to work with. For what it's worth, we deploy FileMaker Pro 18 and have not seen this issue.

One potential scenario though: most users aren't seeing this because SIP is enabled by default and thereby preventing the data loss. Something similar played out last year with Google's keystone auto updater software that seemed to impact lots of people in the AV field because they had disabled SIP due to AV-related software requirements.

That said, Claris definitely has a lot to fix with their packaging. See this thread: https://www.jamf.com/jamf-nation/discussions/32103/packaging-license-for-filemaker-18