Thanks for the detailed responses from everyone especially @amanda.wulff and @clifhirtle

I apologise for not having responded back earlier with the times it took to upgrade but the upgrade only took a few hours once flushed the log files, cleaned up the DB a bit as well as any stale clients.

I hope everyone has good success upgrading from JSS 8.xx to 9.xx!

Just to add on to what Amanda said:
1) To emphasize, on a Windows server, make sure you are running the installer as a local admin, not a domain admin.

2) Unless you have a strong reason not to, I would suggest turning off plug-in inventory collection (Settings -> Computer Management -> Inventory Collection -> Software -> Plug-In). I think everyone on this forum would agree that the plugin table can grow astronomically. If you need collection about a certain plug-in, chances are there is an existing Extension Attribute to collect the data you need.

3) For new server setups, the first thing you should do after your JSS comes up is replace the SSL certificate, either with a valid signed cert, or with a self-signed certificate from the JSS's built-in CA. I wish the JSS setup assistant would run that as a last step.

-Hank

Good points, @hzimmerman !

Just to expand on #2, because it's something I see frequently enough:

In 8.x specifically, the tables that tend to grow to massive sizes are:

plugins - If you're not using that data for anything and don't need to collect it, turn it off in the inventory collection preferences and truncate the table to clear out the unneeded data. If you are using it, typically a good manual log flush will trim it. If it doesn't, we usually truncate (after making a backup) and let it rebuild as devices check back in.

unixapps - If you're not using that data, turn it off in the inventory collection preferences and truncate the table to clear out the unneeded data. I'm not sure I've ever run into someone who was actively using that data, but I'm sure there are a few people out there! If you are using it, we do the same as we do for plugins/fonts: Take a backup, try some manual inventory log flushing, and if it still doesn't come down in size, truncate and let it rebuild as devices check back in normally.

unix_executables - Same as unixapps.

fonts - Same as plugins. If you are using font data for reporting, typically a good manual log flush of inventory related logs will trim it. If it doesn't, we usually truncate (after making a backup) and let it rebuild as devices check back in.

Fonts and Plugins can grow pretty large in 9.x as well, but with regular log flushing, they shouldn't ever get to an out of control size.

Applications can get fairly large as well, but that can be taken care of by manual log flushing and by going in and making sure we don't have more policies updating inventory than are absolutely necessary.

When you're looking at your 8.x JSS summary, and are looking at the list of tables, make certain that your list of tables ends with whitelist.
As a side note, a 9.x database's last table in the list is vpp_user_accounts, so if your full summary ends on something that isn't vpp_user_accounts, please get in touch with your TAM ASAP.

If it doesn't end with whitelist, it means one of two things:

1) The summary didn't generate properly. It happens sometimes; try it again and see if we see whitelist as the last table.

2) There is table corruption somewhere. Most commonly, applications, plugins, fonts, unix_executables or unixapps; when those tables get overly large, they sometimes fall over on themselves and crash.
In that case, get in touch with your TAM; we can try a mysqlcheck -u root -p --auto-repair -o jamfsoftware to see if it helps, but if we know which table it is, and we know it's large, we typically truncate, and then run the repair-optimize.

If you see tables that show NULL sizes or have trouble viewing a summary if you select table row counts, that is usually caused by having third party custom views; off the top of my head, the only one I can recall offhand that does that is Web Help Desk.
The custom views are to allow integration between their software and the data in the JSS, and since the JSS can't always parse the data from the views, we sometimes see that a table row count being selected will cause a blank page to show up when generating a summary, and the sizes for those views show up as having a NULL size.
It's harmless and doesn't affect anything, upgrades will go through just fine with the custom views as the JSS won't try to do any updating to those, we just have to get the data a different way if we need it.

Amanda Wulff
JAMF Software Support

At this stage I would like to suggest augmenting the JAMF Docs with some of this info above. Obviously, it's very useful and certainly will save many people headaches if they know these things before they begin upgrades/setup. This site and the people are great, but either offer a secondary document or update the current ones to reflect this stuff. My 2¢ after having been in a fire for a week :)

@boettchs][/url

Man, if I could get into the Tech Writing department...;)

We do pass these sorts of things along to them for consideration, though!

I have a lot of Evernotes that I frequently use for common things that pop up, they're usually what I reference for threads like these if I don't remember everything off the top of my head:

Windows install prep.
Windows install that didn't go quite as intended and now the JSS is down.
Guidelines for Tomcat & MySQL settings broken down by amount of devices.
Guidelines on what to look for in a JSS summary in terms of potential future (or current) problems.
Lists of log files, their default locations, and what they're useful for.
How to put applications into debug mode.
How to put the JSS into full debug mode and get a clean, debug only log.
How to figure out what's causing that dang "Problem detecting MDM profile after installation" error in 9.3x.

I have a few documents that touch on what to look for in log files, but those are a bit harder to make useful to other people, as I'm not the strange log file genius that some of my colleagues think I am (shh!).

I look at text block shapes, which are pretty much uniform in console and adjust to however large the window is made, and pause when I see something that doesn't 'match' with what I know good, 'nothing is wrong, everything is great, here are some statuses' text to look like, and I know very well what the 'shape' of a stack trace blog looks like so I always pause on those.

In debug logs and system.log files, I can scan and filter out WARN vs. ERROR vs. DEBUG and know when to pause based on that as well--and when I find something, as embarrassing as it may be to admit, I don't just know the answer sometimes and toss the error into Google to find it!
It's a bit tricky to document that in a way that's helpful to other people, though, "Look for shapes, then use Google!" isn't super helpful.

I've also got a messy, long, disorganized (in the sense that everything in there is 'how recently I decided to add it' and not in any sort of category order) document of useful MySQL commands that touch on everything from finding bad profiles to clearing APN queues (for everything or for single devices), manual log flushing, changing JSS account privilege levels, resetting a forgotten local JSS user password (we use that to avoid having to truncate the users table if AD is down and nobody remembers the local JSS admin account password), undeleting deleted config profiles (Spoiler: They aren't actually deleted when you delete from the webapp, they just get a switch flipped in the database to make them not show up and not be in use anymore! If you accidentally delete one that you want to get back without re-creating, contact your TAM, we can usually still get them.), and things of that nature, but since a lot of the commands in that document can be pretty destructive (if they aren't just select statements) in the wrong hands, those typically are only sent out on a case by case basis and nearly always sent out over a webex in a troubleshooting situation.

I do try to keep our internal things up to date, and I'm fairly certain that most of my own notes that I mentioned above are on our internal sites, since others in Support can search those and use them, but official documentation and JAMF Nation KBs go through an approval process (which is a good thing!), so it sometimes takes a little bit longer to get those up and going.
I've also been working on cleaning my Evernotes mentioned above up a bit so they can be sent to our Tech Writing department for tweaking and, hopefully, approval to go up somewhere public facing.

TL;DR: I agree, and I like documentation.
The more documentation, the better, especially for things like this where we're aware that Windows tends to require a few extra steps to make sure updates to the JSS go smoothly in addition to just the general 'prep list' sort of things that should be done before major upgrades and, in my 'I like my databases tidy' view, before any upgrade at all.

Amanda Wulff
JAMF Software Support

@amanda.wulff: Of course we probably all have our own notes and as you said, things are available here. However, I still feel that (based on my own experiences and reading others) that some of these tips should be in the (as it stands today) very lean upgrade instructions. Thinking one has followed the process only to find that there were many other tidbits to try not only wastes valuable time, but can unnecessarily lead to mistakes and problems that could have been avoided.

Learning from trial by fire can be good - lord knows we have learned a lot that way recently, but some things I'd prefer not to learn when clients on a global scale are affected.

I and others appreciate the efforts put in by volunteers here as well as JAMF employees. None of this is meant to knock anyone - merely a request to make such info more obvious to those of us who are new or fairly new to Casper. This thread alone is a good stand-alone PDF in my docs. Hopefully others will benefit from it as well.
While I like good documentation, there is not much I like less than creating and maintaining it

Scott

@amanda.wulff - I second that your post on MySQL and Tomcat settings NEEDS TO become a knowledge base article.

I would actually be in favor of JAMF flushing and re-writing all of their Kbase articles for Casper 9 (or setting up a new Kbase for 9 and later). They aren't organized well, don't search well, and there are some workflows/processes that require going through a couple of articles and taking snippets from both.

Totally revamping and expanding technical documentation should be high on JAMF's list. And the support folks should be eating the dogfood. If you have tons of info stored outside of the knowledge base, you're doing it wrong.

Should I submit this as a Feature Request, in order to get some traction?

Feature request is now in to have the Tomcat and MySQL optimization information added to the JAMF Nation KnowledgeBase. Please vote it up.

https://jamfnation.jamfsoftware.com/featureRequest.html?id=2494

@rtrouton nice feature request it's already got my vote thanks for putting that in!
It's going to help loads of people upgrading from 8.xx to 9.xx

@amanda.wulff Wow, your posts are incredible!!!