Well, my database is a lot smaller (1.1GB compressed) and it took me a little less than two hours to upgrade, in my test Windows 2008 R2 VM.

In one of my tests the progress bar stopped updating, but refreshing my browser later fixed that. Everything was fine.

With that size DB it will take quite a while, I think. The conversion steps seem to be the bulk of it.

A database that size will take a VERY long time (many hours) to upgrade. I ran into this awhile back - you can clear out space by flushing old logs and computer check-in reports. That brought the size way down for me, and made the upgrade much faster.

I whittled our database down to about 2GB before upgrading from 8.73 to 9.22. We're running the JSS on CentOS 6.4 with mySQL living on a different server. The total time for us to upgrade was roughly 30-35 minutes. Like others have said, the upgrade will take quite awhile to upgrade with a DB that large.

So, how did it go? Getting ready to do this too so this is good info to digest before taking the plunge.

@boettchs I'd upgrade a test environment if at all possible before taking the plunge.

With support from Jamf software we decided on cutting down the size of the MySQL DB by
- flushing the log files and setting up auto flush log files
- the size of one of the tables in the DB "unixapps" was at 16GB, jamfsoftware recommended that we truncate the table as that table will be dropped in the upgrade process. So truncating it before the upgrade would speed up the entire process.

The DB size right now is around 11GB

Based on what we've been told the upgrade should take between anywhere between 15-20 hrs (this is an estimate)

I'm currently in the progress of upgrading the server now.

I'll reply back with how long it took to upgraded our server when it's done.

That is very similar what happened to us when we upgraded, ours was the plugins table and it was over 20GB total. First, we turned off the collection of plugins in the inventory, then, we worked with support and truncated the table, as well as running the optimize and repair commands with mysqlcheck. And flushing the old logs.

We also adjusted the mysql and tomcat settings in the JAMF Database utility, increasing the packet size and increasing Tomcat's maximum memory (based on the machine's abilities).

*If mysql settings are greyed out there is a command you can use to fix it, I wrote it somewhere. If you need it, let me know and I'll find it.

It helped a lot, got the database around 1GB and the JSS startup screen ended up completing in about 2 minutes.

I ended up running it two or three times in the test environment (highly recommend!), being a little extra cautious, but everything went smoothly on production.

Well, our attempts to first install 9.31 a while back and 9.32 this past weekend have been a disaster.
The server (JSS) is a Wintel 2008 R2 box, and although it appeared that we got it sorted this past weekend, post testing has resulted in all kinds of issues for us. We are currently working with JAMF support, but are down.
When imaging, the Self Service app is installing corrupted and can't be used as well as assorted other issues. I have to say that this has been a very disappointing experience for us. We only have 700 Macs at this point, and I honestly am not sure about how to cope with any future upgrades - especially major upgrades down the road.
Even if we tested on another box, it would have to replicate the prod box to be of any value. We have some other boxes in a lab that were fine - but a lab is not a prod environment with all of it's own challenges...
Good luck to anyone making a major jump.

My shop upgraded from 8.73 to 9.32 about a month or so back. I posted about the general experience here:

http://derflounder.wordpress.com/2014/06/28/upgrading-from-casper-8-73-to-9-32/

In our case, it went smoothly.

Is a server 2008 VM an option for testing?

We run our JSS in a windows VM, so getting a 2nd one set up for testing upgrades was trivial.

I wouldn't suggest any major upgrade without a lot of testing in a test environment if something breaking is a big deal in your environment. Being down as a result of an upgrade is enough of a deal to warrant a 2nd box (if VMs aren't an option), or it isn't. We struggled with support for weeks on issues we found in testing... but the upgrade to production (when we could eventually do it), went smoothly.

Ideally JAMF would find all the bugs and waiting until .3 release you'd hope major bugs are worked out, but if your environment is anything like mine - we always seem to find bugs others haven't ran into.

Thanks @CasperSally][/url and @rtrouton][/url. We have one JSS instance - not a VM as this point.
I agree we have to come up with something better to avoid (as much as possible) such problems.

Since we (Mac guys) have no control over the Wintel boxes, we're sorta hobbled by this and have to engage others for assistance when working on them. We can't even disable the AV stuff without a member of the server team and the AV team on hand. A very tough field to negotiate with one hand tied behind your back...

Of course, nothing is worse than problems on a system that is new and unfamiliar. That made it much worse to be under the gun and basically be struggling to know where everything is and how much it's changed. I read the docs, and even looked at the test servers a while back, but it just wasn't enough to be confident handling this fire on day-one. Live and learn. Hoping this experience gives us more ammo for the next round.

Appreciate the feedback though.

@boettchs

Hope you don't mind me jumping in; I took a quick look over your open case, and wonder if the issue with the JSS installer not thinking Java is properly installed might just be system variables needing to be set/readded. I've seen those variables occasionally get wiped out when the installer fails.

Awhile back, I'd posted the instructions for it in this thread.

Since they're pretty short, I'll just repost here:

Open up Control Panel >> All Control Panel Items >> System >> Advanced System Settings >> Environment Variables

Under System Variables, we should see something similar to the following:

Variable: JAVA_HOME Value: C:Path oJavajdk.1.x.x_xx
Variable: JRE_HOME Value: C:Path oJavajre7
Variable: JSS_HOME Value: C:Path oJSS


Change the drive letter and paths to match what the actual drive letter and paths are for your system.

If we don’t see these variables under System Variables, we’ll want to create them.

After they're created, the server will need to be rebooted.

As for Windows installs in general, coming from a support standpoint, Windows servers currently need a few extra steps to make sure everything goes smoothly when running the JSS installer:

Snatching from one of my own posts in a thread about servers in general to copy-paste the relevant parts to Windows servers:

The issues we in Support tend to see the most often both with new installs, re-installs on a new VM, or upgrades/updates to the JSS:

- Our installer conflicts with most Antivirus/Internet Security software out there.
By conflicts, I mean the installer sometimes will completely fail until the AV software is either completely disabled or temporarily uninstalled.
When it rolls back, it breaks Tomcat in the process, and then it's usually a call to support to get it untangled and running again.
The issue seems to be with the AV software flagging the ROOT.war file as malware.
In some cases, whitelisting .war files can get you around it, but that leaves the door open for any malware that’s using .war and is probably not the best of ideas.


- Sometimes, the installer doesn’t play nice with domain admin accounts vs. local admin accounts. Not a huge deal, just switch over to a local account and it tends to work fine.
When that happens, it’s usually because, while the domain admin account does have domain admin rights it doesn’t always have a full set of local admin rights.
Even then, we find we have the best luck by running the installer at an elevated privileges command prompt (right click, run as administrator) using msiexec /i.


- UAC can occasionally cause the installer to fail, and we sometimes find we need to disable it for the duration of the install.


The errors the installer gives won’t be that exact, they’re kind of vague: “A script required by the installation has failed to run.” or “Error changing log paths.”, but those of us in support have seen both messages often enough to know it means one or more of the above three things.
The most common cause out of the three is anti-virus software: Sophos, Symantec Endpoint, McAfee, and Cymphonix (which is usually domain based, and does NOT show a tray icon, it just shows up as cymdir in task manager) are four that I’ve run into frequently that tend to kill it.

They're also issues that development is aware of, mostly because I never stop talking about it when I run into one of them. :)

All of the above can be avoided entirely by going the Manual Install route instead of using our Windows installer, however, so it’s not too difficult to avoid those issues.

Initial setup with a Manual Install involves a little more legwork, but on Windows servers, it’s typically easier in the long run, especially if your server is running AV software.


- Paths for Java need to initially be set in the system variables, or the installer may throw an error about java not being installed.
This thread has more instructions on what needs to be done to avoid/fix that.

- On that note, sometimes we need to add the path to the MySQL bin directory in the PATH variable for the JSS Database Utility to work.


- Currently, a JDS will not run directly on Windows.
Not a deal breaker, necessarily, as if you want to run a JDS in your environment you can do so by using a Mac or a Linux box/VM to set it up instead.
However, if you’d planned on having a JDS on the same server as the JSS, Windows wouldn’t work out for that.


- Minor and cosmetic, but still worrisome when noticed: There is a cosmetic defect (D-005575) in the JSS Database Utility for Windows. No matter what you set Tomcat's maximum memory allocation to, it will always appear to reset to 512MB. This is not accurate and is a cosmetic/visual only. If you run tomcat7w.exe and look at the Java tab, you'll see the correct amount of memory allocated.

- When you first upgrade/install a JSS, use any browser other than IE to open it.
Sometimes, IE will appear to hang on "initializing database" and it will stay there forever.
It isn't actually locked up in most cases, it's something to do with IE; open it up in any other browser on any computer that is able to reach the JSS, it will go through the initialization, and you'll be able to use it in IE again.

Most of these things aren't necessarily deal breakers in terms of Windows server vs. Other OSes, or impossible to get around, but it's nice to know about them. It looks like you'd commented in the thread with the above info, but since it may also be helpful for someone else, I figured it was worth re-copying here.

I've also let your TAM know about this thread so he can keep an eye on it as well.

Thanks!
Amanda Wulff
JAMF Software Support

@ boettchs Up until 9.3.2 I saw issues in upgrading Casper on our Windows 2008 server in our test environment, specifically around Tomcat's webapps folder and keystores. In those cases I had to actually clear out the Tomcat folders entirely before running the upgrade, so the Windows installer would not get caught up and fail.

Since 9.3.2 I'm seeing great success in our test environment, with zero of the issues of past.

Production upgrade scheduled for tomorrow so keeping my fingers crossed all goes well as what I'm seeing in our test VM.

@clifhirtle

Just because I'm curious: Had you manually stopped Tomcat prior to running the installer at all?

I've seen issues, prior to 9.31, where the Windows installer didn't seem to accurately check or didn't wait long enough for Tomcat to properly stop, and would try to go ahead with moving the usual files it backs up (server.xml, TomcatSSLKeystore, and a .p12 file if one exists for a third party SSL cert) while Tomcat was still running.

What would happen, at that point, is that it couldn't really move OR overwrite the files needed to let the installer go through, and it would error out and fail, then leave everything in a broken state when it did its rollback.

What we found was that stopping Tomcat manually, prior to running the installer seemed to stop that from happening since the service was already fully stopped and the installer had no issues moving/copying the files it needed to.

If I recall, in 9.31, development changed a few of the timeout & polling thresholds for the installer and how it checked to see if Tomcat had stopped and checked to see if Tomcat had properly started again, which took care of a lot of the issues.

Initially, I know the hard timeout limit for checking to see if Tomcat had restarted successfully was 30 seconds, and if Windows took longer, the installer would assume that the Tomcat installation had failed, the installer would fail, it would roll back, and everything would fall apart.

In 9.31, the timeout was changed from 30 seconds to 5 minutes (it polls every 3 seconds until it gets a 'yep, Tomcat is running!' back from the OS or until the 5 minutes runs out, at which point it reports a failure, if memory serves) based on the fact that the test VM (2008R2) I was using to demonstrate the problem to one of our developers took a whopping two minutes and thirty seconds to get Tomcat restarted to the point that the installer would accept the service as 'running'.
Granted, my often abused test 2008R2 VM probably isn't the best comparison to a real world production server, but I'd rather have the hard timeout limit on the high end than on the low end.

I think they also raised the time limit at the start of everything, where the installer checks to make sure the Tomcat service has stopped from 30 seconds to around three minutes as well.

I'm in the "bitten dozens of times, still shy" category where Tomcat, the JSS Installer & Windows are concerned, and still recommend manually stopping the Tomcat service AND backing up the Tomcat directory on Windows servers prior to running the installer just in case.

Amanda Wulff
JAMF Software Support

Thanks for the detailed response @amanda.wulff! I gave up testing point releases after multiple failed upgrades in 9.2x so it may well have been the timeout value. I do recall preceding upgrades with every possible system-side tweak I could manage beforehand, unchecking the read-only attribute on JSS folder, import DB vs upgrade, backing up JSS folder, stopping Tomcat and MySQL services, etc. My support profile is a graveyard of various failed upgrade cases which (although we worked around by dumping entire folder contents) I ultimately did not feel comfortable going ahead with on our non-replicated, single prod server.

9.32 has been flawless so far, other than a minor error after importing the prod DB into our test VM indicating an import error using cleartext password on the command line (or something to that effect). I had assumed that was the result of having dropped and added the jamfsoftware DB in the MySQL console too many times in the test VM. Ultimately it did not seem to effect the usability of the DB in our test environment so I just kept going with the upgrade and have seen no other issues. Hoping for the same in prod tomorrow!

@clifhirtle][/url

"Warning: Using a password at the command line can be insecure", by any chance?

If that's a yes, then we're using MySQL 5.6 and will want to make sure, both in dev and production, that we've got strict mode turned off.

Turning strict mode off usually makes it stop complaining about that (oddly, not using a password at all does as well), and stops anything over 40 characters from being truncated.

The warning itself is something Oracle considers to be expected behavior and it normally doesn't cause any major problems, however, if turning strict mode off doesn't stop it entirely, going back to 5.5 certainly does as the feature that causes that warning to happen isn't present in 5.5.

Turning off strict mode. In 5.6 it is on by default, and is not supposed to cause issues, but in cases where we're finding it is, the following KB has steps to turn it off: https://jamfnation.jamfsoftware.com/article.html?id=318 While you're in there editing things to turn strict mode off, be sure to up your max_allowed_packet to at least 512M under both mysqld and mysqldump, and up your max_connections to something higher than the default unless you've got a small--under 300--amount of devices.

The long, sometimes angry, occasionally NSFW due to profanity in the comments bug report about that warning: http://bugs.mysql.com/bug.php?id=66546

Keep in mind that, sometimes, that error is misleading and the real error follows the warning about passing the password at the command line being insecure.

I suppose it's obvious by this point that I may have run into this many times before, isn't it? :)

Amanda Wulff
JAMF Software Support

Interesting @amanda.wulff! That would explain it. Prod on MySQL 5.5, test on 5.6.

Do not have my.ini but doe have "C:ProgramDataMySQLMySQL Server 5.6my-default.ini" and of course it does identify as having strict mode enabled.

As you mentioned, it does not seem to effect anything with the DB import, more an annoyance.

EDIT: This post was 2 years old, and the information in the comment is no longer recommended or safe to use on the JSS.

Therefore, I am removing it, as using it can cause adverse performance in a JSS that is 9.8 or higher.

@Everyone that has replied here since yesterday - Wow - I thank you. I have been working with our rep (Dan the Man!) non-stop and didn't even see this until this AM.
I have to spend some time to look over what's been posted here.

What we are seeing in a nutshell after an uninstall of JSS and reinstall of 9.32 is that after imaging, some Macs have a corrupted Self Service.app. It comes across coincidentally at 2.2MB which is the same size as the Self Service.app before it's unzipped. So we are wondering if it's not being unzipped correctly at install, or if the install/download is somehow truncated. Oddly, the icon on the SS app shows normal, but we know right off if it's bad due to the file size. Normal is ~5.2MB.

If we install a QuickAdd.pkg or enroll again, it installs fine and works fine. If you download the QuickAdd.pkg from the JSS it comes down OK every time. I'm more unsettled by the inconsistency than anything here. I don't know if there's other issues we haven't found yet or if this is the only one.

I will read all that's here and see if there is anything we haven't tried yet. I reallyl appreciate the assistance offered up here. Great community!

Scott

@amanda.wulff][/url: Thanks for the suggestions. Going through one at a time, and I did find that one variable in the below is missing:

Variable: JAVA_HOME Value: C:Path oJavajdk.1.x.x_xx
Variable: JRE_HOME Value: C:Path oJavajre7
Variable: JSS_HOME Value: C:Path oJSS

Also want to tell you that we did indeed disable the SEP software by putting the server into a disabled group and stopping all SEP services. That was a chore as we have to get about 6 different humans from different groups together as everyone has specific and limited rights.

Dan K has be doing some other things as well, but I thought Id report that the variable above is not on the server. We upped the Tomcat max memory during the install to 6GB as the server has 32GB and is it's sole purpose.

Cheers, Scott

@amanda.wulff thanks a million for the detailed info here, invaluable as a reference guide for anyone making the upgrade.

Our long-awaited upgrade from 8.73 > 9.32 went off seamlessly. FWIW, we did reduce DB size by flushing all logs > month back, removing stale devices, and cleaning up any old policies, packages, etc that were still hanging around on the CasperCorner up to no good.

Since upgrading I did see Tomcat stop 2x, but since working with our local TAM to ensure the various Tomcat/MySQL connections/memory settings are updated to 1/2 device RAM on Tomcat memory, 512MB on max allowed packets, and bumping MySQL DB connections from 100 > 601, as recommended here all is well again in the Land of Casperism!

@clifhirtle

+1 for seamless upgrades!

+1,000,000 for house cleaning prior to said upgrades! If more people did that, there would be many smoother upgrades. :)

Glad to hear it went smoothly!

Amanda Wulff
JAMF Software Support

Our database is always lean. Pre-upgrade, it was ~30MB. We did everything possible to try to get that seamless experience, but alas on Windows, it was not to be with us. Most of the issues seem to have resolved, but the oddities we saw have me still concerned as there really are no sound root causes for things we experienced.

It's almost as if it took a week for the JSS to get caught up and function properly after we finally got a clean install of 9.32.
The hardest part was trying to troubleshoot an all-new interface, etc. while putting out the fires. But in the end, it's the best way to learn...
JAMF was a lot of help during it all, and while I'm still smarting from it, I can't wait to install 9.4 ;)

@boettchs

Ah, Windows.
Windows can be kind of squirrelly with the JSS installer.

Most commonly, it's anti-virus that ruins the party, and if we can temporarily disable its on access scanning things usually go through pretty smoothly.

I also tend to recommend stopping Tomcat manually prior to running the installer on Windows servers, just to make sure we don't run into the weird situation where Tomcat hasn't fully stopped and our installer, instead of pausing like it's meant to, just decides to plow ahead anyway (which causes the install to fail when it can't move certain Tomcat config files due to them being in use).

Really, with Windows servers I err on the far side of caution when running upgrades and do the following:

- Antivirus off. Make sure any domain based AV is temporarily disabled as well.
- UAC off.
- Logged in as a local admin, not a domain admin. I've seen problems that arise when the domain admin doesn't have a proper set of local admin rights.
- Manually stop Tomcat.
- Manually backup your Tomcat folder to the desktop, just in case.
- Run the installer from a Command Prompt opened as Administrator with: msiexec /i /drag/the/installer.msi /lv c:JSSinstall.log (That's a lower case L in that command). That gets us a debug log of the installer if something fails, so we can look at it and figure out exactly what tripped it up. If it goes fine, the JSSinstall.log it makes can be removed.
- IF it fails, don't click OK, go to the Windows emp directory and copy the JSS installer log file that gets auto-deleted when OK is clicked.
- Send those to your TAM to look at if it has failed.
- Celebrate if it went smoothly, manually restart Tomcat, and continue on with your day.

The 9.3 installer took care of a few of the timing issues that caused a lot of headaches, which is nice, but we do still see a lot of...arguments...between antivirus software and the installer; when the installs fail, it's usually that.

Amanda Wulff
JAMF Software Support

FWIW, our server is Windows Server 2008 R2