Jamf Pro 10.15.1 FAQ

[Garrett_Denney]

Hi Jamf Nation,

We recently became aware of a security vulnerability that impacts versions of Jamf Pro 9.4 and later. To mitigate the issue, we are making Jamf Pro 10.15.1 available today.

This vulnerability does not pose a risk to private data or managed devices. It does have the potential to impact the integrity and availability of your web server.

Cloud customers will be automatically upgraded during the upgrade window outlined below. Premium and Custom customers can contact their Customer Success representative to schedule an upgrade. On-premise customers can download the installer via the My Assets page on Jamf Nation.

Details we are able to provide at this time are below. If you have additional questions, please contact your Jamf representative or leave a comment below.

Thank you,
Garrett

---

Update #1: Sept 29, 2019
Jamf Pro 10.13.1 - Now Available

Yesterday we disclosed a critical security vulnerability that impacts all Jamf Pro instances from 9.4.0 through 10.15.0 and made Jamf Pro 10.15.1 available to mitigate the issue. Today we are making an additional build available for customers that are unable to upgrade directly to the latest Jamf Pro release.

We recognize that some customers might have specific constraints that prevent them from immediately upgrading to 10.15.1. To give those customers an immediate path to mitigation, we’re making 10.13.1 generally available today.

Because all standard cloud customers are already upgraded to 10.15.1 (and protected from the known vulnerability), 10.13.1 is only available to customers that control their instance version, such as On-Premise and Premium Cloud.

To upgrade to Jamf Pro 10.13.1, please contact our Customer Success team at success@jamf.com. We have the capacity this weekend should you want to upgrade immediately.

---

Frequently Asked Questions

What is the issue?
We recently became aware of a critical security vulnerability that could potentially impact any Jamf Pro instance. Jamf Pro 10.15.1 mitigates this issue. This issue does not impact any other Jamf products.


Why is this important?
We take security very seriously and want to move quickly to give you every option to upgrade and stay secure. This vulnerability does not pose a risk to private data or managed devices. It does have the potential to impact the integrity and availability of your web server.

Is my instance impacted?
All Jamf Pro instances running version 9.4 or later are impacted and should be upgraded to 10.15.1 as soon as possible.

When will my standard cloud instance be upgraded?
Cloud upgrades began during a global cloud maintenance window today (Sept 28) at 1700 UTC and will continue through 0500 UTC on Sept 29.

When will my Premium Cloud instance be upgraded?
Please contact success@jamf.com to schedule an upgrade for your environment.

How can I secure my on-premise instance?
An installer is available now in the My Assets page on Jamf Nation.

[dmarcnw]

It's pretty obvious it's something bad. Honestly for as long as I've been a Jamf admin (ten years), a lot of places take longer to upgrade. Be it resources or time, I'm sure there are customers holding out on older platforms. You also have customers who are not active in the Jamf Nation community. So the message probably won't get to them.

It's most likely that Jamf knows not everyone is itching to upgrade to the latest like myself or other customers. They know disclosing this puts those hold outs at risk.

If your org can't make an exception to this non CVE disclosure, then I would suggest carving out an exception for this. I've worked at places where you have to put in a change request and list your reasons why and disclosing the CVE is needed. We would make a note of the exception of the CVE not being disclosed if we had to. Just like it seems the same here with their disclosure.

So it's: 1) put older installs at risk of being compromised and Jamf getting a black eye for not remediating for these users, or 2) live with their decision to not disclose and patch if you can.

The flip side of the coin is, I would like to know what was at risk and if the bad guys exploited that CVE in my org.

[chris_kemp]

I agree that jamf could be doing a better job of disclosure here. However, the jamf-nation forum, which is publicly accessible, is not be the place for details beyond what they've posted here already.

Perhaps a combination of the email blast that alerted us over the weekend, plus a secure site to log into as a customer with appropriate (NDA) access? Details could be posted and updated as they become available, and people would have a better idea of what they're dealing with without coaxing their contacts at jamf for info they're not allowed to discuss, and making inferences in order to judge whether it is safe to wait or not.

Now: "Scuse me, while I patch this server..."

[CasperSally]

Jamf's CISO @Aaron.Kiemele says "We continue to recommend all customers upgrade to Jamf Pro 10.15.1 immediately to mitigate this issue."

Are emails from jamf not getting through to me again, or have customers not been sent an email telling them to upgrade? I only found out about all of this thanks to slack and twitter. I gave up on the jamfnation fire hose long ago.

Edit: I see someone above mentioned an email blast this weekend. Weird I get all sorts of emails via our premium support contract, I believe this is not the first time a critical email went out that we never got

[egsjlusky]

The subject of the email that I got announcing 10.15.1 was "Jamf Pro Maintenance Window Update". Wonder if that only went Cloud customers? Or if it went to everyone, I wonder how many on-prem customers ignored it because it sounded like a cloud thing?

[CasperSally]

I for one didn't ignore it, I didn't get it. Could have been an issue on my end, who knows. I put a ticket in and hope jamf can find details of email being send to me if so so I can look into it further on my side.

[bradtchapman]

@egsjlusky : Correct: that email went to Cloud customers only. Jamf also posts automated RSS-style notifications in the #JamfNation channel on Slack. Those automated alerts about the cloud upgrade to 10.15.1—on a Sunday morning, no less—are what tipped me off that something was very, very wrong.

[CasperSally]

There are some non cloud customers that got the email, so it's not cloud only.

Jamf knows my email when it's time for renewal, sending jamfnation alerts, premium account updates, etc. I suggest they figure out why all customers didn't get this email. The same exact same thing happened the last time there was a critical upgrade, I was part of a group that didn't get it.

[RobertHammen]

@bradtchapman Not a cloud customer here, and I received the email.

[bvrooman]

Mirroring what @CasperSally said, I did not receive anything by email. It was by chance that I logged onto JamfNation to search for something else, and saw the announcement at that time. I do have an email from 9/10 telling me that Jamf Pro 10.15 is available, and I would have assumed that being signed up for those notifications might also include me in important security patch announcements.

[exno]

@macn00b there is a very good chance that will be deleted. As others have posted that and had it deleted as well.

Edit: and it's gone.

In general to everyone: If you have need for more info and haven't done so yet. Email Security@jamf.com it may take a bit to reply but someone on the InfoSec team will send out the Canned Response with CVSS and generic details. they made my InfoSec team and Change Management team more at ease for scheduling the update out of cycle.

[jhuls]

@gachowski Your line of reasoning seems to me to be a bit out of touch. Not everyone administering Jamf has final say on when and what they update. Each organization's IT dept has their own set of priorities besides Jamf security issues and updates so those might need reworked.

Communication is key to making this happen and there is only so much time and so many resources. It's things like this that can make other platforms for managing Apple devices a more alluring option. I've been under pressure in the past to consider SCCM. While it might not be up to the task of matching what Jamf can do, that's not always the only variable.

The last couple times we've updated Jamf we've had issues that we had to contact Jamf about and work through for hours. Last Friday we lost a whole afternoon trying to update from 10.12 to 10.15 and some of that was due to Jamf not communicating effectively with our app support guy the process we needed to take for our system. We had to roll back and now have to deal with this security update as well as those issues without a solid message from Jamf other than update now. Jamf not communicating effectively does not help their cause of keeping us as a customer. I don't expect technical details on this security issue but I do expect knowing how severe it is in a professional manner that respects their customers.

[tlarkin]

Communication is key to making this happen and there is only so much time and so many resources. It's things like this that can make other platforms for managing Apple devices a more alluring option. I've been under pressure in the past to consider SCCM. While it might not be up to the task of matching what Jamf can do, that's not always the only variable.

I am all for holding jamf accountable and asking they build an actual disclosure process where they can communicate the necessary data to customers. However, this is no way makes me even want to touch SCCM with a 5 million foot pole.

There are other MDMs, but lets be honest here, does anyone want to use the MSFT stack to manage macOS or iOS devices?

[jhuls]

@tlarkin I don't want to turn this into a comparison of management solutions but will say that there are reasons it didn't happen. That doesn't mean though that I won't be pressured to look at something other than Jamf.

[brunerd]

If you are on 10.13 for one another reason, you may get a WAR file from Jamf for 10.13.1
(we are moving to Jamfcloud soon, so a Java upgrade just to get to 10.15.1 is not where I want my efforts and testing to go)

Upgrading using a WAR file, it is an "artisinal process" where you can miss putting back some of your old config files, unless you've cobbled together your own script to do this, it's best to let the already tested install scripts from Jamf do the heavy lifting. To do that we'll use the guts of the 10.13.0 .run file and put the new 10.13.1 WAR inside

#make a work folder and go into it
mkdir JamfPro10.13.1
cd JamfPro10.13.1

#download makeself and expand in work folder
curl -L https://github.com/megastep/makeself/archive/master.zip -o ./master.zip
unzip ./master.zip

#download the Jamf 10.13.0 installer for Linux, move it to your work folder
open https://www.jamf.com/jamf-nation/my/products

#expand the .run file
./jamfproinstaller.run --target ./expanded

#remove the old 10.13.0 ROOT.war
rm ./expanded/resources/ROOT.war

#download the 10.13.1 WAR from the link your jamf buddy gives you
#copy the new WAR where the one was
cp ~/Downloads/ROOT.WAR ./expanded/resources

#make a new .run file with makeself
./makeself-master/makeself.sh --gzip ./expanded/ ./jamfproinstaller-10.13.1.run "Jamf Pro Installer" ./install.sh

#for goodness sakes test it on a dev server first

[Taylor_Armstron]

I'll jump through the hoops and email security, but WHY is Jamf posting this on the forum, and not reaching out to customers via email themselves? "Criticial" security issue, and they're hoping customers stumble onto the forum and find out!?

Count me in as one who cannot move forward without formal documentation, I need to have my CTO sign off on out-of-band maintenance, and to further complicate things, I'm a contractor whose contract ends in under two weeks, before the next scheduled maintenance window. Zero wiggle room for weekend or after-hours work right now, so this means coordinating on multiple levels.... and not a hint of any outreach from Jamf's security team to their customers proactively.

This is not how you treat enterprise customers.

[Hugonaut]

https://resources.jamf.com/documents/products/security-disclosure-notice-jamf-pro-10.15.1.pdf

[Taylor_Armstron]

from link above:

Now that all customers have had three days to upgrade, we are able to share additional information about the vulnerability.

This IS a communications failure.

"All customers who browsed the web forums of Jamf Nation, because we did not reach out to them to notify them".

This isn't how you deal with things at this level. Glad to have some progress, but still an utter fail in my eyes. All other companies we deal with in my office would have reached out through our account managers to notify us proactively.

[exno]

Emails were sent out over the weekend. If you didn't receive it you may need to check your spam filtering or check with jamf support on why you may not have received it.

There was an email to cloud users were they got an update schedule.

Then there was an email to customers with the subject line "Critical Security Upgrade: Jamf Pro 10.15.1" from notifications@jamf.com
containing an acknowledgment of a vulnerability then a link to "more info" in the release notes. as well as a request to reach out to Success@jamf.com with any question we may have.

this forum post was by far the most info available at first but still not enough. Security@jamf.com filled in the many blanks so that I can get changes approved. and the announcement today seems to be closer to what we all wanted originally

[taugust04]

Just took a few minutes to read through this thread. Great read and good discussion. I think if Jamf had added this line to their initial communication from 9/28, this would have satisfied most customers who have to deal with change control:

"To protect our customers while preparations to upgrade are made, Jamf will not be immediately releasing public details of this vulnerability. If you require more technical details of this vulnerability for beginning the remediation and change control process at your organization, please contact security@jamf.com."

Google did something similar with Chrome a week or two ago when 77.0.3865.90 was released to patch a huge security vulnerability.

As others have posted, some organizations have more rigid change control policies and procedures than others. Thankfully I do not, but I can see why the initial obfuscation of communication would make some customers angry.

That being said, I can understand how Jamf wanted to hold some data back so that malicious actors didn't start port scanning public IP ranges looking for hosts affected by this vulnerability, and then having hosts ending up in Shodan (or something worse) before admins had time to patch. Hopefully Jamf is listening for the next time this occurs.*

*Hopefully there won't be a next time.

[Sonic84]

Has the CVE number been posted yet? I can't override my companies Change Freeze window without it.

[mhrono]

I still take issue with one line in the official disclosure.

"This issue only impacts the Jamf Pro server and does not impact managed clients or client management functions."

This is 100% false and Jamf should stop propagating this. According to the disclosure, exploitation of the vulnerability could result in remote code execution as the Tomcat user on the server. If an attacker can get RCE with all the rights of the user running Tomcat, this means the attacker could read the plaintext database credentials from the DataBase.xml file. With those credentials, an attacker could:

• Delete, modify, or add new scripts to the database, which run as root on the client
• Send MDM commands to enrolled devices
• Delete, modify, or add new packages
• And any other action possible with full access to the database, which is anything

An attacker could reach out and impact all managed clients and client management functions.

[Taylor_Armstron]

Emails were sent out over the weekend. If you didn't receive it you may need to check your spam filtering or check with jamf support on why you may not have received it.

Thank you for that. For what it is worth, Jamf did reach out this afternoon after seeing the post. They apparently sent an email, but to a contact I've never heard of. Now going through the list of contacts they have for us and cleaning it up - somehow have 9 names on the list, and mine is the last one. So I can say that at least they made an attempt, but looks like the person contacted may be someone in a procurement office. (we have 20,000+ employees, purchasing doesn't even happen in the same State where I sit).

[jonathan_rudge]

Has anyone elses pre-stage scope gone "blank" ?

[bvrooman]

I did get an email about an hour ago notifying me of the update, but it makes reference to "last weekend, we alerted you to a critical vulnerability..." which did not seem to occur.

I also received a call from Jamf, following up on the apparently-missing email. They seem motivated to get to the bottom of why I didn't receive the notice, but have received similar (non-security) update emails, which is appreciated.

[iviemeister]

We did not receive (and still have not received) any emails from JAMF about this.

If not for seeing the Register article this evening, we still wouldn't have heard about it...

I also don't see any guidance on determining if the issue has been exploited, and any recommended remediation if it has been.

[tlarkin]

@iviemeister

The TL;DR is that the vulnerability is bad enough you'll want to update as soon as you can. The rest of the ramblings in this thread is about proper disclosure process, and that is the gist of it.

[iviemeister]

@tlarkin

Yeah, we updated ASAP after seeing the article, THEN came here to find out why we were not notified.

Guidelines for detecting prior exploitation would be good to have...

[blackholemac]

Now that we have it patched, I second everyone on here's posts about how communication was not done right.

Had we been taken into confidence Monday when I first called asking questions, I could have put in an emergency change control request Monday night and had this patched already. I didn't know what was going on until AFTER our pre-scheduled change control meeting on Tuesday (early in the day). Since I had no clue what the flaw was at that point despite escalating to support twice, I had to push change control through on iOS compatibility grounds and "a flaw that Jamf considers bad". As such our change control folks and I scheduled the change for tonite on a Wednesday (our night for changes generally.) Later Tuesday when I saw how bad the problem was, I wanted to get it patched Tuesday night, but my family already had other commitments by that point.

Basically Jamf's fail on not getting legit folks informed in a timely manner cost me 48 hours of time in which I could have had our cluster patched had I known the severity of the vulnerability.

[bashhash]

When do you expect to roll out automatic Jamf Pro upgrades? It's a pain in the butt constantly having to visit the website, download the new version, replace it, and get up and running. It would be wonderful if we could tick off a check-box for automatic upgrades, or have it sync to upgrade to the latest version of whatever Jamf Pro Server is on.

Also, the download to 10.16 is broken, so I'm stuck unable to manage packages for the time being: https://www.jamf.com/jamf-nation/my/products

[Taylor_Armstron]

Automatic updating of an on-prem production server is the last thing I would ever want, FWIW.

For cloud, I get it, the main reason we run on-prem is to keep tighter control.

As for the download - give it a min and try again? I just downloaded both the Mac and the Windows installer without a problem, so may still be propagating on the CDN.

[edit. Realized I completely mis-understood you. You're talking admin packages, not auto upgrading the server. My mistake and apologies]

[donmontalvo]

@tlarkin Kudos! One of the responsibilities we have, as subject matter experts (SMEs) for the macOS platform and management tools, is to mentor our IT groups.

The more effective we are at mentoring, the more trust trust and confidence we gain, and the more success the platform will have in the environment.

(Apologies if the word "mentoring" triggers some folks...hey, its the 2020s...but can't think of a better word to describe this.)

Also, to accept any risk, one must understand what that risk is. If you cannot understand the risk, you just have to assume the worst. Which is not a great way to do business.