Jamf Pro 10.0 virtually killed my JSS

[DirkM2012]

For the last two weeks I was working with support on an issue that seemed minor at the beginning. QuickAdd failed to complete the enrollment and the logs showed that this was caused by a failed MDM profile install. On the server the logs get flodded with messages like this:

at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:229)
    at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1927)
    ... 15 more
2017-11-30 05:51:28,336 [ERROR] [Tomcat-10  ] [JAXBPlistParser          ] - Error unmarshalling
2017-11-30 05:51:28,336 [ERROR] [Tomcat-10  ] [MDMController            ] - Error processing mdm request, returning 400. Device: Null, CommandUUID: mraNull
2017-11-30 05:51:51,412 [ERROR] [Tomcat-8   ] [JAXBPlistParser          ] - Error unmarshalling
2017-11-30 05:51:51,412 [ERROR] [Tomcat-8   ] [MDMController            ] - Error processing mdm request, returning 400. Device: Null, CommandUUID: mraNull
2017-11-30 05:52:00,296 [ERROR] [Tomcat-3   ] [JAXBPlistParser          ] - Error unmarshalling

Since it took support two weeks to figure that that this is a product bug, the backups of the pre-upgrade database had been overwritten (30 day retention) and I'm basically stuck with a non working JSS and the option to wait for them to fix this or start over from scratch and re-enroll everybody.

I'm sure that they are working on this "critical" bug with high priority but it is really frustrating to hear that this issue is known for a couple of weeks and there is still no eta for a fix. What is even more frustrating is the fact that even week ago I would have had a working backup to restore 9.x.

I guess that's another chapter in my lessons learned book. I'm just glad that I only have 50 Macs and not 5000...

[scentsy]

@dmatth01 sorry for the bad situation.
could you share more details please, I'm planning on upgrading our JSS version 9.96 on a Windows 2012 R2 we have less than 100 macs with 60 policies, 10 configuration profiles, we bind to AD, its a small enviroment.

hope your issue gets resolve soon.

[DirkM2012]

All I can really say I that 9.98 on 2012 R2 was working perfectly fine. I upgraded to 10.0 on 11/2 and noticed shortly after that quickadd.pkg finishes with an "Installation failed" message.

Jamf.log show this over and over on the client:

Thu Nov 30 16:28:34 xxx jamf[1468]: Enforcing management framework...
Thu Nov 30 16:28:39 xxx jamf[1468]: Error installing the computer level mdm profile: profiles install for file:'/Library/Application Support/JAMF/tmp/mdm.mobileconfig' and user:'root' returned 400 (The operation couldn’t be completed. (MDMResponseStatus error 400.))
Thu Nov 30 16:28:39 xxx jamf[1468]: Problem installing MDM profile.
Thu Nov 30 16:28:39 xxx jamf[1468]: Problem detecting MDM profile after installation.

I have 50 Macs and about 5000 Windows computers, so the issue wasn't the highest priority for a week or two. But I knew we had some high level (CxO ranks) users with new Macs on order, so eventually that issue would need to get fixed. I opened a case with Jamf and over the last two weeks talked to various people. Last Tuesday it turned out that this is a known bug and that 10.0 fails to parse some response that 9.x had no issue processing. At least that's what I'm told.

I haven't tried pushing any new policies or packages recently, that may actually work. At least my "enrollment" packages got installed fine prior the installer saying it failed. But I cannot really tell those users that everything is fine even though the installer said it failed.

Since I don't have a 9.x database backup anymore I can't do much about this other than waiting for Jamf to provide a fix. Or to start over, re-create all policies and smart groups and send out emails to my Mac users asking them to re-enroll the assets.

[scentsy]

@dmatth01 Thank you for posting this.

[dgreening]

Can you please share the product issue number? Thanks!

[DirkM2012]

I was not told a product issue number.

[were_wulff]

@dmatth01

I apologize if you weren't given a Product Issue number by support; that is usually something that is given out to a customer if we determine the issue they're having is, in fact, a Product Issue. It's likely that whoever you were working with simply forgot.

I looked up your open case and found that it was attached to PI-004915.

I haven't done much work on that particular Product Issue, but it looks like the underlying cause is needing Akamai, which Apple sometimes redirects to for MDM communication, to be whitelisted along with Apple's IP block.

The workaround in the Product Issue is to whitelist www.apple.com on port 80 in addition to 17.0.0.0/8 on ports 2195/2196 as well as whitelisting Akamai CDN.

17.0.0.0/8 redirects to Akamai CDN need to be allowed as well, so if you have rules blocking redirects set up on your network they'll need to be eased up just a little to allow traffic related to Apple's IP block (17.0.0.0/8) or domain to happen.

It looks like the person you're working with already told you that, however, I'm including it here for anyone else who may run into the same issue.

Thanks!
Were Wulff
Jamf Customer Experience.

[DirkM2012]

I will win the lottery before my network guys are opening all those ports. And this issue didn't exist in 9.x, so from my perspective it is a 10.x issue that needs to be resolved on the JAMF server. I was also told that this may or may not fix the issue, it's rather hit or miss.

As I said before, I could start over with 9.x and everything will be fine, wait for a fix for 10.x or switch to something else that works.

[ryan_ball]

If you have one JSS, it is not a security risk or even much trouble to allow that one server access to those netblocks/ports. I'm pretty sure those requirements have been in place since 9.x.

Here is the 9.9 doc mentioning the same thing: http://docs.jamf.com/9.9/casper-suite/administrator-guide/Ports.html

[geoffreykobrien]

I noticed this as well, but it turned out our DEP macs were going in as mobile devices. Deleting them out resolved my issue.

[ajfunk]

The workaround in the Product Issue is to whitelist www.apple.com on port 80 in addition to 17.0.0.0/8 on ports 2195/2196 as well as whitelisting Akamai CDN.

My organization had this exact issue when upgrading to JAMF PRO 10, and the above quote resolved the issue for us. I had my network engineers open the aforementioned FQDN/Ports, and all is working flawlessly.

@dmatth01 please encourage your network engineers to adopt this work-around.

[DirkM2012]

I got them to open 17.0.0.0/8 on port 80 in addition to the other ports but they are now asking what exactly "whitelisting Akamai CDN" means.

[ajfunk]

They'd whitelist Akamai CDNs via DNS I'd assume:

.akadns.net
.akam.net
.akamai.com
.akamai.net
.akamaiedge.net
.akamaihd.net
.akamaistream.net
.akamaitech.net
.akamaitechnologies.com
.akamaitechnologies.fr
.akamaized.net
.edgekey.net
.edgesuite.net
.srip.net

Those are the domains that Akamai uses, not sure if all of them are used for their CDN.

[ajfunk]

@dmatth01

Before asking them to whitelist, have them set up web filtering to permit www.apple.com, as opposed to permitting Apple's owned public IP subnets (what I assume they're currently doing).

[CasperSally]

We whitelist the apple 17.x network (still not on v10), but whitelisting the whole akamai CDN seems like asking for trouble for a K12 who needs to maintain CIPA compliance. There's no way to know what else you're allowing beyond the apple traffic you're trying to allow.

[daz_wallace]

Anyone having difficulty getting their security / networking teams to open the required and documented ports to Apple, check out this presentation from JNUC 2017, and send onto your relevant teams. It explains the communications and the level of security involved (both client-side and Apple-side).

https://www.jamf.com/resources/a-push-odyssey-journey-to-the-center-of-apns

If they're still not willing to open the required ports, you're gonna have a bad time now, and in the future.

But I hope that helps,

Darren

[jhbush]

@were.wulff I would appreciate a detailed response from Jamf regarding the Akamai whitelisting not just a best guess from a fellow Jamf Nation member. Do you have this available?

[DirkM2012]

Instead of just waiting 8 weeks with nightmares about SCCM I decided to stand up a new JSS 9.101 on Windows Server 2016, after all my JSS is broken so it cannot hurt much at this point. I completed basic config and confirmed JSS working in general. I then stopped Tomcat on my DMZ server, renamed root.war and ROOT inside webapps and copied the root.war from my new master server. After starting Tomcat it prompted me to point it to the new database which I did and everything started working fine after that. I can enroll new computers, see the MDM profile on the client and get no MDM related errors in the logs anymore. This is the same Windows server with no changes other than a new root.war and pointing to a new JSS database. To me this confirms that this is not a firewall or ports issue, this is purely a bug in Jamf Pro 10 and while others may or may not experience this I would strongly recommend to not upgrade to Jamf Pro 10 any time soon. And if you do make damn sure you don't loose your 9.x database backup.

I now have to re-create all smart groups and polices and manually transfer all the other settings but at least JSS is working again. And last but not least I will need to figure out how I can reconnect my DMZ server to the old database, deploy as last task the new QuickAdd.pkg that points to my new database and point the DMZ server back to the new database once all (or most Macs) got the new config.

Most disappointing is the lack of urgency at Jamf with no eta on when this issue might be resolved in Jamf Pro 10, no workarounds, nothing. I guess I just take this too serious and Mac users and admins are usually way more relaxed than their Windows counterparts...

[kyleblanc]

Hey Everyone,

So this has happened to our JSS environment as well and we are hitting a wall with our Security team to allow all outbound traffic over port 80 to www.apple.com

Throughout all the research we have done, talked to JAMF, talked to Apple, etc and no one has been able to answer the question of what packets/payloads/data is actually being transferred over port 80? Why does this specifically have to be open for APNS?

[emily]

We're seeing this too, on our 9.101.0-t1504998263 production instance. We were informed to check on the firewall rules and haven't done so yet (holiday dead week) but will at the beginning of the year. Just wanted to bump this up and note that I don't believe it's specifically a v10 issue.

[bentoms]

@kyleblanc Port 80 is needed for VPP titles IIRC, to get the description... icon etc

[russeller]

This might not affect everyone with these issues, but we were having this error and here is what we found:

The issues was tied to a Smart Group with inaccurate criteria, of a date range that isn't valid. We determined this due to an error in the logs: [MDMController ] - org.joda.time.IllegalFieldValueException:
Cannot parse "yyyy-mm-dd": This is a product bug. We ran a JSS summary and found the smart group with the date error in the criteria and fixed it (or we could have deleted it). The issue appears to have been resolved.

Our specific issue is one of our campus techs put in Feb 29th 2017 as that date in a criteria as it wasn't valid. Hope this helps someone.

[guidotti]

@bentoms I am seeing this on version 10.1.1-t1513360285 in my TEST environment.
I cannot install MDM on any enrolled device.
Do you think opening itunes.apple.com TCP Port 80 for VPP will solve this?
I have that open as a firewall request (since it wasn't in 17.0.0.0/8, which I opened already).

I get these errors (reduced for length):

2018-01-29 15:21:33,426 [ERROR] [Tomcat-21 ] [JAXBPlistParser ] - Error unmarshalling
2018-01-29 15:21:33,426 [DEBUG] [Tomcat-21 ] [JAXBPlistParser ] - Error unmarshalling
javax.xml.bind.UnmarshalException
- with linked exception:
[com.ctc.wstx.exc.WstxIOException: Connection timed out: connect]

2018-01-29 15:21:33,426 [ERROR] [Tomcat-21 ] [MDMController ] - Error processing mdm request, returning 400. Device: Null, CommandUUID: mraNull
2018-01-29 15:21:33,426 [DEBUG] [Tomcat-21 ] [MDMController ] - com.jamfsoftware.jss.exceptions.operations.MDMActionCreationException: java.lang.NullPointerException
com.jamfsoftware.jss.exceptions.operations.MDMActionCreationException: java.lang.NullPointerException at com.jamfsoftware.jss.mdm.actions.MDMActionFactory.createActionForRequest(MDMActionFactory.java:331) at com.jamfsoftware.jss.mdm.enrollment.MDMController.process(MDMController.java:124) at com.jamfsoftware.jss.mdm.enrollment.MDMController.doPut(MDMController.java:92)

[guidotti]

I just did a support chat. The agent said they are intending to add www.apple.com TCP 80 to the official required network port document.
It looks like JSS is trying to reach out to: http://www.apple.com/DTDs/PropertyList-1.0.dtd

-Bruce

[lashomb]

The agent said they are intending to add www.apple.com TCP 80 to the official required network port document

I was told its a bug in the code where the server reaches out to http://www.apple.com/DTDs/PropertyList-1.0.dtd.

We had to create a dynamic group for Akamai addressing; even so we get the 400 errors still. So we are probably missing something somewhere.

[russeller]

Any other updates for the 400 errors?! I’m getting the same @guidotti is seeing now with 10.3.0.

[thejenbot]

@ssrussell also see https://www.jamf.com/jamf-nation/discussions/27070/error-processing-mdm-request-returning-400-floodi.... i have been keeping an eye on my logs and interestingly enough we generally only see this error during the school day - very rarely at off-peak hours. curious to know if you are (or anyone else is) seeing that as well. also, i've found that if you're in the parent node of the jss (when you would see a * after the text in the browser tab) you can't go back too far, whereas if you're in the child node (only seeing a in the browser tab) you can see back much further. in the parent i can only go back about a day, and in the child node i can see back almost a month (since it doesn't show the chatty license monitor info).

[guidotti]

I had to open the www.apple.com range as I mentioned above to get rid of these errors.

[kerouak]

I'm getting these.. [ERROR] [Thread-1773] [JAXBPlistParser ] - Error unmarshalling
2019-03-11 12:30:00,035 [ERROR] [Thread-1773] [MDMController ] - Unable to parse device from request
com.jamfsoftware.jss.mdm.actions.exceptions.PlistParsingException: Plist was null when trying to get device UDID at com.jamfsoftware.jss.mdm.actions.MDMActionFactory.getDeviceFromPlist(MDMActionFactory.java:759) at com.jamfsoftware.jss.mdm.enrollment.MDMController.getDeviceFromRequest(MDMController.java:266) at com.jamfsoftware.jss.mdm.enrollment.MDMController.process(MDMController.java:162) at com.jamfsoftware.jss.mdm.enrollment.MDMController.doPut(MDMController.java:107)

I was told it was fixed in 10.9, however, I've just seent he same on my 10.10 webapp..

[FastGM3]

@kerouak

What is the end result of getting these errors?

I'm seeing similar errors and haven't been able to enroll a macOS device. Just curious if you're seeing the same enrollment problems. iOS enrolls just fine, macOS enrollment is a no go manually, via quickadd, or DEP. Every now and then one does enroll, can't duplicate anything.

Support has been working the case for a while now. We're using 10.10.1