Downgrading JSS from 9.32 to 8.73, not too bad or a big bag of hurt?

[tomt]

I just emailed this to support but figured I should also ask the Nation. The upgrade has gone fine in my test setup but it never hurts to hear other people's experiences.

I'm planning to do this upgrade either tonight or tomorrow. My plan is to clone the JSS before upgrading (just in case) along with a manual backup of the database. JSS is on a Mini running 10.8 server with an AFP distribution point.

If everything goes sideways and I am forced to revert back to the 8.7.3 clone how will that effect the client machines? Will they just receive the older client and run happily or will I be in for a world of hurt?

Thanks,
Tom

[were_wulff]

@tomt][/url

"Big bag of hurt" would be putting it lightly.

There is no clean or easy way to go from 9 to 8 if you get to 9 and decide you want to go back to 8.

Every computer that has checked in will have the 9 binary, and will need to be re-enrolled. This isn't something we can do via policy either, as the 9 binary won't communicate well (or, really, at all) with an 8 JSS. It's also not something that can always be done via a Recon network scan, as we've found in a lot of cases we have to remove framework on the affected computers, reboot, and THEN re-enroll.
That's potentially a lot of legwork if you have several hundred computers or more.

Every case I've worked with that started with, "I want to go back to 8!" has ended with, "...I'll just get used to 9." if that's any indication of how ugly it can be to get straightened back out and get everything back on 8 again.

It would be extremely rare that anyone in support would recommend going from 9 to 8; I can't honestly think of any time I've seen or heard that being recommended.

You can minimize potential issues upgrading production by getting in touch with your TAM to schedule a health check PRIOR to the upgrade so they can take a look with you and go through any settings changes that may need to be made, spot any issues that may not be presenting symptoms yet, recommend house cleaning (mostly manual log flushing, turn off inventory options you aren't using, clean out old, unused policies and config profiles, clean up your inventory by deleting records that are no longer necessary to keep, clean up your smart and static groups, etc...) prior to the upgrade so that the production database being upgraded is as tidy as possible.

After all, upgrading a messy database can easily result in a messy upgrade.

And, as always, take a backup just prior to starting the upgrade; if something DOES have to go back to 8, it's something we'd want to do within 5 minutes of a failed upgrade, and not poke around with it for a few hours or days first, at that point, clients may have checked in and upgraded their binary, and that's when it becomes kind of a mess to undo.

Edit: To add, I took a look at your account, and we have an old JSS summary from May of this year. Based on what I see in that summary, you don't want to upgrade production without going through some pretty decent house cleaning first. The database size is far too large for the number of devices, and I see several MySQL settings that need to be changed.
Sending in a new full JSS summary will give us a more accurate picture, and we'd like to see that first since the one we have is several months old and may no longer be accurate.

Thanks!
Amanda Wulff
JAMF Software Support

[tomt]

Thanks for the quick, thoughtful and thorough (as usual) reply Amanda. I'll send a new one today.

Tom

[were_wulff]

@tomt

Sounds good!
I left some notes on the case you've got open with your TAM (who should be in in about 15 minutes) to link him to this thread and let him know what's going on.

Amanda Wulff
JAMF Software Support

[andrewseago]

"Big bag of hurt"

[tomt]

Summary and logs sent.

[were_wulff]

@tomt

Took a quick look at them this morning and left notes for Lee; there are a few clean up things we'll want to do as well as some settings tweaks for Tomcat and MySQL. Main thing I saw on the log was that port 2195 is blocked, which means you're not going to be able to send push notifications or configuration profiles through the JSS. I noted that and left a reminder to discuss that if those are features you'd been intending to use.

Your TAM, Lee, is usually in between 10:30 and 11am CDT, so he'll be able to see the notes I've left.

Amanda Wulff
JAMF Software Support

[tomt]

Thanks Amanda. On the port 2195 thing, that's outbound to Apple, correct?

[were_wulff]

@tomt

Yep, that's outbound to Apple and allows the JSS to actually get push notifications out.

We also typically want the following ports open, in addition to 2195:

2196 - This is not strictly necessary, as it's just a feedback port, but it's incredibly useful to have open for troubleshooting purposes.

5223 - This is the port clients use to communicate with Apple.

443 - This is a fallback for wireless devices if they can't get through on 5223.

The 17.0.0.0/8 IP address range (or 17.0.0.0 - 17.255.255.255 if your firewall/proxy/content filter doesn't allow the other format) should also be whitelisted; the whole block is owned by Apple, and the APNS uses DHCP, so there isn't a specific little block within the range that it uses.

If you're using Lightspeed, you may need to give them a call and ask them to turn off SSL packet inspection if it's not something you can turn off in the GUI as we have seen SSL packet inspection cause the APN to reject those packets and cause things to just not go through or to only go through intermittently. I've not super familiar with Lightspeed's interfaces, but notes on past cases have indicated that it's something that may not be an option in the GUI and will require a Lightspeed technician to go in and turn off; that may or may not be the case, as many of those notes were 6 or so months old, so, just putting that one out there just in case.

These are Apple's direct recommendations and apply to any MDM provider, so it's not something that is a specific JAMF requirement.

Apple's documentation:
http://support.apple.com/kb/TS4264
https://developer.apple.com/library/ios/technotes/tn2265/_index.html

Thanks!

Amanda Wulff
JAMF Software Support

[tomt]

Upgrade went smoothly and everything seems to be working as expected. So I guess there's nothing left for me to do except bitch about the new interface. :-)

[tomt]

May have spoken too soon. For some reason 9.32 will not send email notifications after the upgrade. The 8.73 was working just fine.

Edit: This is the error that shows up in the JSS web interface when I test.
"Error sending message: javax.mail.MessagingException: Exception reading response; nested exception is: java.net.SocketTimeoutException: Read timed out"

[tomt]

Posting again to include the edit in the JamfNation email.

May have spoken too soon. For some reason 9.32 will not send email notifications after the upgrade. The 8.73 was working just fine.

Edit: This is the error that shows up in the JSS web interface when I test.
"Error sending message: javax.mail.MessagingException: Exception reading response; nested exception is: java.net.SocketTimeoutException: Read timed out"

[were_wulff]

@tomt

I've seen things like that happen now and again after upgrades; usually if we just go in and re-enter the information and re-save it takes care of it.

When it doesn't, there isn't a GUI way (currently) to delete SMTP server settings and re-enter them, so we have to go into MySQL and do the following:

use jamfsoftware;
truncate smtp_server;

It will say 0 rows affected, that's normal, and it's a lie, it's affected all the data in the table.

Exit MySQL, restart Tomcat, then go back to the SMTP Server page in the JSS and re-enter the information.

Amanda Wulff
JAMF Software Support

[tomt]

Thanks Amanda, we tried your suggestion with no change. I've just updated the ticket with an email to Lee.

[tomt]

After some more fiddling with the SMTP settings we've got the emails sending happily.

I am seeing sporadic policy failures where the client can't mount the DP and it fails over to our secondary DP. Not sure yet what's up with that.

[tkimpton]

It's always good practice to upgrade in a development environment first with test machines and all your policies first, then test in a test environment a mirror clone of your live.... Otherwise....big bag of hurt

[rtrouton]

Indeed. I went from 9.x to 8.73 in my test environment a few times and I had to uninstall both the Casper agents on the test machines and the JSS software on the test server in order to get back to where 8.73 was running normally.

For details on the JSS-side of the uninstall process, I wrote them up as part of this post:

http://derflounder.wordpress.com/2013/09/24/uninstalling-casper-on-red-hat-enterprise-linux/

Doing that in a test environment was fine. Doing it in production would have been ugly.