Natick school district scapegoating Casper Suite

[hkim]

Sounds more like someone misconfigured Casper more than anything else. WIth power comes responsibility.

http://www.boston.com/yourtown/news/natick/2013/01/natick_student_computers_fried.html

[donmontalvo]

I emailed the author asking for a follow up article *after* they find the root cause.

jaclyn.reiss@globe.com

Sounds like someone cobbled together a script or policy without testing.

What's so hard about finding/killing a specific process and rm'ing the app? :)

Hand a loaded gun to a child, why don't they...LOL

Don

[mm2270]

This story is just dumb. I confess I'm a self proclaimed Casper fanboy, but looking at this as objectively as possible, it seems highly implausible that this was caused by a bug in Casper. From the description, it sounds like they were using Restricted Software's function to delete offending apps, but misconfigured it, perhaps putting in some kind of wildcard that caused it to see all processes as something to delete. If they don't know what they're doing, they need to let someone else manage their computers so they won't do stupid things anymore.

And this is particularly frustrating for me to read because I live the next town over from Natick. Makes me want to drive over there and slap them all upside the head, LOL.

[mcrispin]

I'm not a fan of the poor grammar:

"Since every student could be doing something different, the bug removed different parts for everybody," he said, noting that Casper Suite company employees flew in from Minnesota to assist Natick officials. "It made the recovery process very difficult."

Did the (alleged) bug, Casper or the employees, make, "the recovery process very difficult"?

It's a shame we don't have real libel laws in the US.

[acdesigntech]

LOL! I interviewed with them some years back! Obviously didn't get the job (I probably wanted too much money).

Sounds like a case of an overzealous scripter not testing his/her script before unleashing it. I have casper do all manner of search and disable and/or destroy almost monthly at my company (almost entirely dependent on one or more buzzwords someone has read in a magazine), and never even once came close to this kind of disaster.

Of course I always test, test, and retest, then deploy to my friends/family circle, then my second pilot group, then require 100% feedback from all piloters that everything is working (or not) before deploying to the masses.

[jarednichols]

Well this should make the Boston JUG interesting :)

[Kev]

FYI: I just tested this. I setup Restricted Software to kill the process "Tor" and in fact in does alert users who do not have this process running or the software installed. However, I did not have it delete anything, so all we have in my environment is a nuisance. There definitely seems to be a bug and their administrators simply trusted their tools. To my knowledge, there is no way to scope restricted software without elaborate use of exempting all computers other than test machines, this should probably be fine tuned at some point.

It would appear that when you specify a process to look for such as "tor", it actually searches for "tor" meaning anything such as opendirectoryd, activity monitor, etc all would get killed, an alert sent and the process would be deleted if you have that option selected.

[nkalister]

yeah, restricted software as currently implemented by the casper suite is a flaming sledgehammer wielded by a blind, half-deaf meth addict.
The best policy is to stay the hell out of it's way and just ignore it until it dies in a fire.

[nkalister]

@Kev:
Restricted software just greps running process names (or does something else that produces results remarkably similar to what you'd get by grepping running processes . . .) There's probably some other process with 'tor' in the name on those machines . . . .

[Kev]

Exactly...

It wil kill the following so far:

/System/Library/PrivateFrameworks/GenerationalStorage.framework/Versions/A/Support/revisiond
/usr/libexec/opendirectoryd
/usr/sbin/DirectoryService
/Library/Application Support/VMware Fusion/vmware-usbArbitratorTool

I can see where this could cause some major problems. JAMF documentation does not explain what this feature does in great enough detail.

However, if you are looking to kill TorBrowser, just look for "TorBrowser" as the process to kill, "tor" and "vadalia" will be killed with it and the intended result is accomplished.

[mm2270]

Yeah, that explanation makes perfect sense to me. Picked up lots of other stuff with "tor" in the name. Maybe if they had put in tor.app instead? I don't know, maybe they tried that and it wasn't working and so they switched it. I would agree that not having an easy way to apply "scope" to restricted software to test it out is a big flaw that needs to be corrected. In their case it was destructive, in other cases its a nuisance, as Kev suggested.

[mm2270]

Meh. Double post.

[kitzy]

It is possible to spin up a "testing" instance for testing features like this before deploying to the entire install base. I realize that's not always an ideal situation, but especially in a case like this I feel like having an isolated test environment would be a really good idea.

That said, being able to scope this feature would also be a big help.

[jarednichols]

external image link

[mm2270]

Yeah, I'm really on the fence about whether we can call this a "bug". If the sitch outlined above is what happened, (and it does seem likely) there is blame on both sides as I see it. JAMF could certainly make the Restricted Software process much more kind or more robust in how it works. But the Natick school techs really should have tested this out before deploying to their masses. I mean, you're deleting something and relying on automation to do it. To just trust it blindly is foolish.

That said, I can see an easy way (on paper at least) to make this less destructive. Make the restricted software matching simply exclude anything from system level paths, like from /System/Library/ and /Library/ and only match processes originating from /Applications/, /Users/ and other paths. If someone really wants to match processes from the OS, place a checkbox within each Restricted Software item you create like "Match system level processes" with a big fat warning about how destructive it can be if not implemented with care.

[jarednichols]

A "quarantine" function would also be useful. Instead of deleting it, move it to an area where it's not allowed to be executed from.

[Kev]

I think they are using JAMF as a scapegoat for their lack of testing. However, there is no easy way to test the method they use. That being said, maybe they should have used a different method to accomplish this task if they truly wanted to delete the processes.

I would think of it as a bug or a flaw because of one reason: If I specify I want you to kill processA and you kill processAB, processAC and so on, without warning me that you will do so or documenting that you are actually searching for anything containing processA, then something is flawed.

[dgreening]

I got several emails from my department head about this over the weekend. I responded that the likely cause was that the person administering Casper did not properly test their policy or restricted software task before releasing the hounds. We test EVERYTHING internally before releasing it, so this will not be a problem for us.

Just for giggles I checked to see if anyone had Tor on their computers, and only two people did, both in our department.

[Paul_Price1]

My approach to this problem has been as follows; Identify the worst offenders. You could probably use Caspers application usage abilities. Bring them into the Dean's office and use the current processes and procedures for student discipline. Keep busting kids until the data shows results and you see the usage disappear. It is amazing how quickly word will spread through the student population. Not only will you clear up the Tor issue you will also plant a seed of caution in the student body without having divulged how you caught offenders. Did a teacher see something, did another student turn them in or was it the IT department and a tech solution.
I have found that the paranoid speculation a student has of a School's IT department capabilities can far exceed the reality of what we can do. Let their active imaginations ponder on your abilities.

Interesting story, (poor reporting) thanks for posting

[dgreening]

Additionally, we blocked the IPs which Tor uses on our firewall. Good luck using it now!

[wudi]

Hello fellow admins -

A defect has been confirmed with Restricted Software in the 8.6x series of the Casper Suite. The defect is related to using non-ASCII characters in conjunction with having the Restricted Software set to be deleted when detected. Non-ASCII characters are treated as a wildcard character, and therefore have the potential to match incorrect processes. For instance, "????.app" will match all running applications since all of the characters are treated as wildcards. An entry for "MyApp?.app" would do a wild card search on "MyApp*.app".

At this time, we are recommending that you do disable the "Delete" option for Restricted Software if you have any non-ASCII characters.

A patched version of the Casper Suite (8.63) is being tested and readied to prevent this from happening in the future. If there are immediate questions, please feel free to reach out to myself, or your account manager here at JAMF, and look for an updated version of the Casper Suite to be available shortly.

Thanks -

[mm2270]

Thanks for officially confirming the issue Wudi and for letting us know a fix is in the works.
I'm glad to hear that at least this wasn't as seemingly innocuous as putting in a simple string and it clobbered system files. Had a feeling there was some wildcard matching at play here but didn't know what exactly.

It does beg the question though of why they needed to use non ASCII characters in Restricted Software to catch Tor though.

[chris_kemp]

Cut and paste from a Word doc, maybe? That wreaks havoc on certain software around here. Formatting codes sometimes turn into odd characters in ASCII.

[donmontalvo]

@chris.kemp "The old copy and paste terminal commands out of word kablammo trick!"- Maxwell Smart

Wudi, your candor is appreciated, and testiment to the integrity of the JAMF team. I hope the Natick School District understands that the reason JAMF Nation is here and he/she should join and expect a warm welcome. I doubt there is a single person here who hasn't dropped the ball, but I'm sure we all learned from it and we have LABs to make sure it doesn't happen again. :)

Don

[acdesigntech]

+1 don :)

Ahhh the upgrade from 7.31 to 8.1.... Memories :P

[jarednichols]

I hope the Natick School District understands that the reason JAMF Nation is here and he/she should join and expect a warm welcome. I doubt there is a single person here who hasn't dropped the ball

Yep. We've all had an "oh fuck" moment when we feel our bodies chill and then we scramble to set a policy to 'disabled.'

[franton]

That's usually about the time you've just received 50+ odd emails with the same fault for all ;)

[Chris_Hafner]

This is always good to know. However, it's still ont heir hands for not testing first. Glad we found a bug and it's being worked on. Bad publicity stinks, but hey.

[Chris_Hafner]

This is always good to know. However, it's still ont heir hands for not testing first. Glad we found a bug and it's being worked on. Bad publicity stinks, but hey.

[chris_kemp]

@donmontalvo - lol, pretty much!

Also guilty here, when I first started implementing Policies I created one (via JSS Mobile, no less...WWIT??) that was supposed to be scoped to a single computer, but accidentally I set it to global and installed an update to every machine in the system in the middle of the afternoon! I got lucky, no one had any issues and the users just shrugged it off and rebooted when the message came up, but I thought I was going to have heart failure when I checked the Policy logs expecting to see one entry... :D

[lisacherie]

Although unfortunate the issues experienced at Natick, the following feature request may assist junior administrators with a need to block subsets of applications, and could be considered as templates for restricted software.

eg. Torrenting application tomato - should the string "tomato" be entered as the process to match, automator.app is also matched in the regular expression, a string of tor would have the same result.

https://jamfnation.jamfsoftware.com/featureRequest.html?id=21

[donmontalvo]

(oops, this one was a little harsh - my apologies to the Natick administrator. Don)

[Chris_Hafner]

Ouch... two in one Don. In any event, I find it a bit hard handed to simply use the "Delete" command when you can kill the process and have notifications sent. OK, I can certainly see the draw, especially if you've got thousands upon thousands of users. However, just nuking an app usually end up making the user think they did something wrong. Because of this they end up re-downloading the app and then "tinkering" with it, if even possible. Working in education I can certainly say that a little knowledge on the users side goes a long way. If you don't take the time to educate them, they will keep you busy!

P.S. OK, so the admin at Natick made a huge mistake, but comparing him/her/them to the adobe update team is a bit below the belt!

[donmontalvo]

@Chris_Hafner Point taken, I removed the comment. I'm guessing the Natick admin learned a lesson and this probably won't happen again (test, test, test). Can't say the same for Adobe's installer develpment team (excuse, excuse, excuse). :(

[Sandy]

@jarednichols "Yep. We've all had an "oh fuck" moment when we feel our bodies chill and then we scramble to set a policy to 'disabled.'"

Where's my "like" button when I need it?

Sandy

[Chris_Hafner]

@Don Nope, Adobe's install team is fair game ;-)

[technicholas]

Any good web filter should of blocked Tor iBoss blocks Tor just fine.

Darn... they should of tested the script before pushing it to all the computers.

[jarednichols]

Any good web filter should of blocked Tor iBoss blocks Tor just fine.

Yeah, I'm a big proponent of blocking a lot of this stuff at the firewall. I know in a lot of cases it's not going to happen, but I do think it's a best practice. If you're blocking stuff upstream from your endpoints, you're not going to go playing whack-a-mole trying to stop it. Kids will try it, see it doesn't work, give up for a bit, maybe try it again, see it still doesn't work, then give up.

[llitz123]

@jarednichols "Yep. We've all had an "oh fuck" moment when we feel our bodies chill and then we scramble to set a policy to 'disabled.'" Where's my "like" button when I need it? Sandy

"Like"