Jamf Pro 10.15.1 FAQ

I'd encourage you all to email security@jamf.com. They eventually did respond to me with enough detail that I was able to move things forward in my organization.

I tried posting the email here because I believe every Jamf customer has the right to the information without spending half a day chasing it down, but my post was deleted almost immediately. I'm not surprised, but I am disappointed.

@kitzy While i can understand the frustration, I don't think posting that information here when Jamf doesn't feel ready is the right move either. You've posted it elsewhere and well now it probably screws it up for other potential situations in the future where Jamf may now have to re-think how they go about communicating issues like this. That is, they may have to get NDAs and what not signed before agreeing to communicate any further on security issues. This essentially makes it even tougher for admins who work at organizations where taking action on a security issue like this may already be tough without the proper "paperwork".

As far as I am considered almost everyone who posted here owes Jamf and Aaron an apology.

I am more than happy with the way Jamf handles these types of issue. I think this is only the 2 or 3 one that I can reminder in the past 11+ years something this happened. I like this approach and I want Jamf to continue it without any changes.

Here, I have a CVE code for you "CVE-because Jamf said so" if you don't trust your vendors then that is on you!!

"Your" internal BS process shouldn’t have any impact on how Jamf address this, as admins we are creating bureaucracy loops and it's everyone's fault. Keep it up and we will be in “Metropolis”. As this thread proves we have started the downward spiral.

Cutting and pasting my post from two years ago..

Posted: 3/3/17 at 1:30 PM by gachowski I think you guys might be looking at this the wrong way... While I agree transparency is important. You guys are trying get Jamf to disclose something that they feel they shouldn't. Do your change control recommend the change and say that the vendor recommended the upgrade ASAP for not made public security reasons. If your change control approval group denies the change then that is on them. C PS The more I think about it, it's not really very cool for anyone to push Jamf to release details about the vulnerability.

This isn't the first time jamf has been under scrutiny for this. The API exploit that allowed an attacker unauthorized access to your scripts was also not disclosed at all. We have two choices here, one assume all notifications like this are really really bad, or two just wait it out.

I am in the crowd, where when I see jamf post stuff like this and basing it off of past experiences, this means that it is bad. Also, jamf saying they don't think anyone is using it in the wild is not really an excuse. This also makes jamf look a bit inept, as they cannot build a proper process to release this information and instead want us to use jamf nation. Why would a jamf nation post be the best place to have this discussion or disclose this info? Now I have to go to jamf nation and search for this article and sift through all the comments to get a glimpse of the info we need?

I propose these topic points as discussion points for both jamf and the community:

• An actual vulnerability process be documented and implemented on jamf's end
• A method and process for customers or third parties to disclose security issues to jamf
• a SLA built upon delivery of said fix and disclosure of what the vulnerabilities were
• A centralized place to find security patches and updates (I don't think searching jamf nation discussions is a good UX)
• Allow for additional disclosures like: email, phone call, etc. If people have to sign NDAs that is fine too

What I am seeing here, is yet jamf dropping the ball again, on what every modern tech company should have. This isn't the first time this has happened, and it seems we haven't learned or grown from the previous times this has happened.

At this point jamf is really telling us to just reverse engineer the product and diff the installers to find the bugs ourselves since it won't be disclosed.

Just like my opinions man.

100% agreed with @tlarkin. Jamf's response to this incident has been abysmal, and their efforts to prevent the admin community from helping each other only makes it worse.

We should expect better from our vendors. We deserve better from our vendors. Jamf isn't the only MDM out there, and events like this make each renewal decision just a bit more difficult.

Dear Jamf: less gobbling up talent in the admin community, more fixing PIs and learning to communicate effectively with your users.

Edit: A proper response would have been something like: "Hey Jamf users, we found a really bad vulnerability involving DoS/RCE/bbq/etc and we have a patch available. Please update as soon as possible. We will be making a full disclosure on x date."

Going to third @tlarkin's post. In order for tightly change-controlled organizations to update out-of-band, particularly when they may be in an end-of-quarter change freeze, they're going to need a LOT more than "Vendor says it's bad and that we should update immediately". CVE score and ID (if one has been issued) are the minimum baseline. Also, what's the risk? Compromised server? Unauthorized access to client data? Unauthorized access to endpoints?

It would seem to me that this information could be disclosed without detailing the vulnerability, but I'm not a dedicated security professional, do not play one on TV, and did not stay at a Holiday Inn Express last night, so...

So, I have seen the note on this exploit and I won't disclose anything about it, because that is the ethical thing to do. I can tell you, it is indeed bad enough to warrant any emergency changes you may have to go through to get it done immediately. I will say, I am disappointed that the info I have seen on this makes either jamf look inept, or it assumes the customer base is dumb, which is to say the least sort of insulting. Also, reading the wording of this note, it is completely misleading and you can tell it was written so for damage control from a marketing perspective.

In the end, if a vuln has the ability for RCE, then just state that. How you respond to vulns speaks more volumes than this cloak and dagger stuff. Also, the vague wording in the note I saw, did not really instill confidence in the spirit of transparency to paying customers.

In the end, this is really just disappointing, and if I really want to know the risks I am put in a spot where I have to start diffing installers and RE the product since the vendor won't disclose it.

The question is not whether I trust Jamf or not. I agree that this is extremely rare for a Jamf product, and if they weren't operating within verticals where audit and compliance is a focus, I can see where this would look like a responsible way to go about handling this sort of situation. The issue is that no one else really cares who I trust, and they shouldn't. My change control team (and the governance teams auditing them, and the bank examiners/industry auditors reviewing them, etc.) work in a vast sea of documentation, and if I'm not contributing what they need to make the step above them happy, then I'm not getting approval to change anything.

I'm a little confused by the accusation that by participating in reasonable change control, I'm somehow the creator of bureaucracy. I'm certainly not apologizing for working somewhere with audit and security requirements, because that's insane.

I didn't get any sort of reply from security@jamf.com, so it seems that the "responsible disclosure be damned" side has won the day, and perhaps this whole conversation is moot.
In the interest of fairness, I did receive a reply about the same time that I posted, with a CVSS score and sufficient information to submit an emergency change request. What I received is not enough to provide any insight on how to exploit anything (nor should it be), and I'm honestly surprised that what I received is being hidden from customers. It's completely within the norm for a responsible vulnerability disclosure, since a patch is already available.

@tlarkin <<sighs>> in terms of dealing with change control, I think I can get this pushed through on iOS 13 compatibility grounds alone though I hate not being able to fully brief the bosses. If those were the only grounds I was going to hold off for a couple more weeks. Is there anyone at Jamf I can talk to on NDA basis that will reveal what you saw?

That being said, I now shift to the technical side...When I perform the manual upgrade to our clustered environment, does said vulnerability target anything related to the Tomcat version itself? Does it target any specific build of MySQL or is it strictly an issue with the web application that Jamf provides from the download portal? Should I be tightening up my server.xml beyond their current advice?

Guessing you can’t answer any of that due to ethics grounds...Jamf support says I should be fine doing a “vanilla” upgrade to 10.15.1... That being said I find myself wanting to make sure that what I do truly patches whatever this issue is.

@blackholemac

The thing is, many Orgs have their own Compliance and Governance, Security Engineering, Blue/Red Teams, Change Approval Board (CAB), and they have a process they have defined internally to accomplish these goals. All the major vendors I deal with disclose Vulns, and we actively pull data from Vuln Scanners to correlate them with data from our data collection tools. jamf is one of those data collection tools for macOS and iOS endpoints. It is up to those teams at each Org to accept the risks, or to approve and push forward patches.

jamf's role in this is simply information. There is a vuln, jamf is aware of it, there is an existing patch for it, but what are the risks? How do we communicate to the people for changes, and how do we go through our defined processes. I agree, I won't apologize or make excuses about change boards and compliance and governance policies, as well as security patching SLAs.

See here is the thing, I have a 14 day SLA to patch anything critical and is available via network access for any RCE. This is a policy set by my company and it doesn't matter if I agree or disagree with it, it is a policy that was put in place by people above my pay grade. So, I must abide by it. This means I have to patch everything with in 14 days, unless I can otherwise prove an exception for. Of course we have emergency patching that is different than the standard SLA.

I cannot raise an exception if I don't know what it is. I cannot weigh the risks if I don't know what it is. God forbid we get audited, but if we did we cannot explain to our auditors anything about our endpoint management solution with this vuln because the vendor will not disclose the info.

I get it, security disclosures are difficult, and jamf wants to protect all those customers who just don't patch their environments. However, jamf needs to build a policy and process for disclosures. I am not going to tell jamf how to do that, because jamf needs to do it so it benefits them and all customers, not just me. It is very clear, at this point in time, none of this exists.

So, jamf needs to engage with the community and build a proper process for disclosing vulns to customers, so customers can leverage their internal processes and do what is best for them. I work for a data company, so we get audited on a regular basis and I am used to it at this point and I am very aggressive with all my patching (OS, third party apps, configs, infra, etc) and the thing we cannot control is our SaaS, which we are jamf cloud. So, for me to not have this disclosure I simply go back to what I stated previously. Which is, I guess if I really want to know the severity of this vuln I will go back to running diffs against the ROOT.war files and start to RE the code myself. Which is something I should never have to do to get a security disclosure.

Also, to address this:

As far as I am considered almost everyone who posted here owes Jamf and Aaron an apology. I am more than happy with the way Jamf handles these types of issue. I think this is only the 2 or 3 one that I can reminder in the past 11+ years something this happened. I like this approach and I want Jamf to continue it without any changes. Here, I have a CVE code for you "CVE-because Jamf said so" if you don't trust your vendors then that is on you!! "Your" internal BS process shouldn’t have any impact on how Jamf address this, as admins we are creating bureaucracy loops and it's everyone's fault. Keep it up and we will be in “Metropolis”. As this thread proves we have started the downward spiral.

I will not apologize for having expectations that every modern tech company follows or should follow. I will not apologize for my Org being security conscious and putting SLAs on everything. We are well with in our abilities from a technology standpoint to actively patch and update our tech, so we should strive to do so. Also, to accept any risk, one must understand what that risk is. If you cannot understand the risk, you just have to assume the worst. Which is not a great way to do business.

Also, if you haven't ever worked for an Org that has standard processes for these things, and you want to call them, "BS," I think you should do a bit of light research. A lot of Orgs right now are probably on end of Q3 change freezes right now, and to emergency patch jamf they might have to get an exception. This is very common for EOQ freezes to allow finance to close the books out on everything and not have any chance of anything external impacting them. Of course, there is always an exception to every process, and jamf is not disclosing enough to make that exception really.

@tlarkin first off I’m sorry if I came off as ANYTHING negative to you. I wrote that post mainly looking up to you.

I do have a basic understanding of what major corporate change control boards are like...Via horror stories.

I agree that Jamf needs to find a way to responsibly disclose this sort of thing to their customers in a way that we can take it to whomever the powers that be are. In my case, I’m fortunate enough that my change control board is a small meeting with 6-7 colleagues once a week where we discuss and schedule any infrastructure changes in my org. Like you I hate going in with minimal info because it then looks like I didn’t prepare enough.

Before I knew of this vulnerability, I was already going to put a 10.15 upgrade on the docket but was going to defer a week or two due to personal scheduling issues. Alas given this vulnerability and given my nature as a custodian to my orgs equipment, I’m gonna have to get it on this weeks docket for 10.15.1.

I do fully support your views on finding a way for Jamf to disclose this sort of thing.

Thank you for your time.

@blackholemac all good homie I did not take it personally at all, and some red tape at Orgs is ridiculous, but the process exists for reasons.

:-)

@gachowski

: )

It's pretty obvious it's something bad. Honestly for as long as I've been a Jamf admin (ten years), a lot of places take longer to upgrade. Be it resources or time, I'm sure there are customers holding out on older platforms. You also have customers who are not active in the Jamf Nation community. So the message probably won't get to them.

It's most likely that Jamf knows not everyone is itching to upgrade to the latest like myself or other customers. They know disclosing this puts those hold outs at risk.

If your org can't make an exception to this non CVE disclosure, then I would suggest carving out an exception for this. I've worked at places where you have to put in a change request and list your reasons why and disclosing the CVE is needed. We would make a note of the exception of the CVE not being disclosed if we had to. Just like it seems the same here with their disclosure.

So it's: 1) put older installs at risk of being compromised and Jamf getting a black eye for not remediating for these users, or 2) live with their decision to not disclose and patch if you can.

The flip side of the coin is, I would like to know what was at risk and if the bad guys exploited that CVE in my org.

I agree that jamf could be doing a better job of disclosure here. However, the jamf-nation forum, which is publicly accessible, is not be the place for details beyond what they've posted here already.

Perhaps a combination of the email blast that alerted us over the weekend, plus a secure site to log into as a customer with appropriate (NDA) access? Details could be posted and updated as they become available, and people would have a better idea of what they're dealing with without coaxing their contacts at jamf for info they're not allowed to discuss, and making inferences in order to judge whether it is safe to wait or not.

Now: "Scuse me, while I patch this server..."

Jamf's CISO @Aaron.Kiemele says "We continue to recommend all customers upgrade to Jamf Pro 10.15.1 immediately to mitigate this issue."

Are emails from jamf not getting through to me again, or have customers not been sent an email telling them to upgrade? I only found out about all of this thanks to slack and twitter. I gave up on the jamfnation fire hose long ago.

Edit: I see someone above mentioned an email blast this weekend. Weird I get all sorts of emails via our premium support contract, I believe this is not the first time a critical email went out that we never got

The subject of the email that I got announcing 10.15.1 was "Jamf Pro Maintenance Window Update". Wonder if that only went Cloud customers? Or if it went to everyone, I wonder how many on-prem customers ignored it because it sounded like a cloud thing?

I for one didn't ignore it, I didn't get it. Could have been an issue on my end, who knows. I put a ticket in and hope jamf can find details of email being send to me if so so I can look into it further on my side.

@egsjlusky : Correct: that email went to Cloud customers only. Jamf also posts automated RSS-style notifications in the #JamfNation channel on Slack. Those automated alerts about the cloud upgrade to 10.15.1—on a Sunday morning, no less—are what tipped me off that something was very, very wrong.

There are some non cloud customers that got the email, so it's not cloud only.

Jamf knows my email when it's time for renewal, sending jamfnation alerts, premium account updates, etc. I suggest they figure out why all customers didn't get this email. The same exact same thing happened the last time there was a critical upgrade, I was part of a group that didn't get it.

@bradtchapman Not a cloud customer here, and I received the email.

Mirroring what @CasperSally said, I did not receive anything by email. It was by chance that I logged onto JamfNation to search for something else, and saw the announcement at that time. I do have an email from 9/10 telling me that Jamf Pro 10.15 is available, and I would have assumed that being signed up for those notifications might also include me in important security patch announcements.

@macn00b there is a very good chance that will be deleted. As others have posted that and had it deleted as well.

Edit: and it's gone.

In general to everyone: If you have need for more info and haven't done so yet. Email Security@jamf.com it may take a bit to reply but someone on the InfoSec team will send out the Canned Response with CVSS and generic details. they made my InfoSec team and Change Management team more at ease for scheduling the update out of cycle.

@gachowski Your line of reasoning seems to me to be a bit out of touch. Not everyone administering Jamf has final say on when and what they update. Each organization's IT dept has their own set of priorities besides Jamf security issues and updates so those might need reworked.

Communication is key to making this happen and there is only so much time and so many resources. It's things like this that can make other platforms for managing Apple devices a more alluring option. I've been under pressure in the past to consider SCCM. While it might not be up to the task of matching what Jamf can do, that's not always the only variable.

The last couple times we've updated Jamf we've had issues that we had to contact Jamf about and work through for hours. Last Friday we lost a whole afternoon trying to update from 10.12 to 10.15 and some of that was due to Jamf not communicating effectively with our app support guy the process we needed to take for our system. We had to roll back and now have to deal with this security update as well as those issues without a solid message from Jamf other than update now. Jamf not communicating effectively does not help their cause of keeping us as a customer. I don't expect technical details on this security issue but I do expect knowing how severe it is in a professional manner that respects their customers.