How long does it take to upgrade JSS 8.73 to 9.22

[bmak]

Hi Everyone,

I am currently running a upgrade of our Production server after taking the necessary steps to clone the VM, backup the DB and also snapshot the Virtual Linux server before running the upgrade.

The JSS is running on a Redhat Linux Server 6.x and the MySQL DB is roughly 31GB located on a seperate drive.

I've since run the linux upgrade at 6:30pm it is now 10pm (sydney time)
I've looked at the JAMFSoftwareServer.log file and there is no entry after 18:32
The last entry of the log is

2014-01-11 18:32:47,569 [INFO ] [Table ] - Altering column purchasing_information.enrollment_profile_id...

When accessing the web url it's been stuck on

Updating encoding for applications (Step 2 of 3)...
Converting 11.61 GB... Do not restart your JSS.

The progress bar looks like it's stuck but the animation of the progress bar is still moving.

How have other fellow system administrators faired with there upgrades?
How long did the entire upgrade take?

Thanks for reading and responding.

[alexjdale]

Well, my database is a lot smaller (1.1GB compressed) and it took me a little less than two hours to upgrade, in my test Windows 2008 R2 VM.

In one of my tests the progress bar stopped updating, but refreshing my browser later fixed that. Everything was fine.

With that size DB it will take quite a while, I think. The conversion steps seem to be the bulk of it.

[chris_kemp]

A database that size will take a VERY long time (many hours) to upgrade. I ran into this awhile back - you can clear out space by flushing old logs and computer check-in reports. That brought the size way down for me, and made the upgrade much faster.

[mml7]

I whittled our database down to about 2GB before upgrading from 8.73 to 9.22. We're running the JSS on CentOS 6.4 with mySQL living on a different server. The total time for us to upgrade was roughly 30-35 minutes. Like others have said, the upgrade will take quite awhile to upgrade with a DB that large.

[scottb]

So, how did it go? Getting ready to do this too so this is good info to digest before taking the plunge.

[CasperSally]

@boettchs I'd upgrade a test environment if at all possible before taking the plunge.

[bmak]

With support from Jamf software we decided on cutting down the size of the MySQL DB by
- flushing the log files and setting up auto flush log files
- the size of one of the tables in the DB "unixapps" was at 16GB, jamfsoftware recommended that we truncate the table as that table will be dropped in the upgrade process. So truncating it before the upgrade would speed up the entire process.

The DB size right now is around 11GB

Based on what we've been told the upgrade should take between anywhere between 15-20 hrs (this is an estimate)

I'm currently in the progress of upgrading the server now.

I'll reply back with how long it took to upgraded our server when it's done.

[jennifer]

That is very similar what happened to us when we upgraded, ours was the plugins table and it was over 20GB total. First, we turned off the collection of plugins in the inventory, then, we worked with support and truncated the table, as well as running the optimize and repair commands with mysqlcheck. And flushing the old logs.

We also adjusted the mysql and tomcat settings in the JAMF Database utility, increasing the packet size and increasing Tomcat's maximum memory (based on the machine's abilities).

*If mysql settings are greyed out there is a command you can use to fix it, I wrote it somewhere. If you need it, let me know and I'll find it.

It helped a lot, got the database around 1GB and the JSS startup screen ended up completing in about 2 minutes.

I ended up running it two or three times in the test environment (highly recommend!), being a little extra cautious, but everything went smoothly on production.

[scottb]

Well, our attempts to first install 9.31 a while back and 9.32 this past weekend have been a disaster. The server (JSS) is a Wintel 2008 R2 box, and although it appeared that we got it sorted this past weekend, post testing has resulted in all kinds of issues for us. We are currently working with JAMF support, but are down.
When imaging, the Self Service app is installing corrupted and can't be used as well as assorted other issues. I have to say that this has been a very disappointing experience for us. We only have 700 Macs at this point, and I honestly am not sure about how to cope with any future upgrades - especially major upgrades down the road.
Even if we tested on another box, it would have to replicate the prod box to be of any value. We have some other boxes in a lab that were fine - but a lab is not a prod environment with all of it's own challenges...
Good luck to anyone making a major jump.

[rtrouton]

My shop upgraded from 8.73 to 9.32 about a month or so back. I posted about the general experience here:

http://derflounder.wordpress.com/2014/06/28/upgrading-from-casper-8-73-to-9-32/

In our case, it went smoothly.

[CasperSally]

Is a server 2008 VM an option for testing?

We run our JSS in a windows VM, so getting a 2nd one set up for testing upgrades was trivial.

I wouldn't suggest any major upgrade without a lot of testing in a test environment if something breaking is a big deal in your environment. Being down as a result of an upgrade is enough of a deal to warrant a 2nd box (if VMs aren't an option), or it isn't. We struggled with support for weeks on issues we found in testing... but the upgrade to production (when we could eventually do it), went smoothly.

Ideally JAMF would find all the bugs and waiting until .3 release you'd hope major bugs are worked out, but if your environment is anything like mine - we always seem to find bugs others haven't ran into.

[scottb]

Thanks @CasperSally][/url and @rtrouton][/url. We have one JSS instance - not a VM as this point.
I agree we have to come up with something better to avoid (as much as possible) such problems.

Since we (Mac guys) have no control over the Wintel boxes, we're sorta hobbled by this and have to engage others for assistance when working on them. We can't even disable the AV stuff without a member of the server team and the AV team on hand. A very tough field to negotiate with one hand tied behind your back...

Of course, nothing is worse than problems on a system that is new and unfamiliar. That made it much worse to be under the gun and basically be struggling to know where everything is and how much it's changed. I read the docs, and even looked at the test servers a while back, but it just wasn't enough to be confident handling this fire on day-one. Live and learn. Hoping this experience gives us more ammo for the next round.

Appreciate the feedback though.

[were_wulff]

@boettchs

Hope you don't mind me jumping in; I took a quick look over your open case, and wonder if the issue with the JSS installer not thinking Java is properly installed might just be system variables needing to be set/readded. I've seen those variables occasionally get wiped out when the installer fails.

Awhile back, I'd posted the instructions for it in this thread.

Since they're pretty short, I'll just repost here:

Open up Control Panel >> All Control Panel Items >> System >> Advanced System Settings >> Environment Variables Under System Variables, we should see something similar to the following: Variable: JAVA_HOME Value: C:Path oJavajdk.1.x.x_xx Variable: JRE_HOME Value: C:Path oJavajre7 Variable: JSS_HOME Value: C:Path oJSS Change the drive letter and paths to match what the actual drive letter and paths are for your system. If we don’t see these variables under System Variables, we’ll want to create them. After they're created, the server will need to be rebooted.

As for Windows installs in general, coming from a support standpoint, Windows servers currently need a few extra steps to make sure everything goes smoothly when running the JSS installer:

Snatching from one of my own posts in a thread about servers in general to copy-paste the relevant parts to Windows servers:

The issues we in Support tend to see the most often both with new installs, re-installs on a new VM, or upgrades/updates to the JSS: - Our installer conflicts with most Antivirus/Internet Security software out there. By conflicts, I mean the installer sometimes will completely fail until the AV software is either completely disabled or temporarily uninstalled. When it rolls back, it breaks Tomcat in the process, and then it's usually a call to support to get it untangled and running again. The issue seems to be with the AV software flagging the ROOT.war file as malware. In some cases, whitelisting .war files can get you around it, but that leaves the door open for any malware that’s using .war and is probably not the best of ideas. - Sometimes, the installer doesn’t play nice with domain admin accounts vs. local admin accounts. Not a huge deal, just switch over to a local account and it tends to work fine. When that happens, it’s usually because, while the domain admin account does have domain admin rights it doesn’t always have a full set of local admin rights. Even then, we find we have the best luck by running the installer at an elevated privileges command prompt (right click, run as administrator) using msiexec /i. - UAC can occasionally cause the installer to fail, and we sometimes find we need to disable it for the duration of the install. The errors the installer gives won’t be that exact, they’re kind of vague: “A script required by the installation has failed to run.” or “Error changing log paths.”, but those of us in support have seen both messages often enough to know it means one or more of the above three things. The most common cause out of the three is anti-virus software: Sophos, Symantec Endpoint, McAfee, and Cymphonix (which is usually domain based, and does NOT show a tray icon, it just shows up as cymdir in task manager) are four that I’ve run into frequently that tend to kill it. They're also issues that development is aware of, mostly because I never stop talking about it when I run into one of them. :) All of the above can be avoided entirely by going the Manual Install route instead of using our Windows installer, however, so it’s not too difficult to avoid those issues. Initial setup with a Manual Install involves a little more legwork, but on Windows servers, it’s typically easier in the long run, especially if your server is running AV software. - Paths for Java need to initially be set in the system variables, or the installer may throw an error about java not being installed. This thread has more instructions on what needs to be done to avoid/fix that. - On that note, sometimes we need to add the path to the MySQL bin directory in the PATH variable for the JSS Database Utility to work. - Currently, a JDS will not run directly on Windows. Not a deal breaker, necessarily, as if you want to run a JDS in your environment you can do so by using a Mac or a Linux box/VM to set it up instead. However, if you’d planned on having a JDS on the same server as the JSS, Windows wouldn’t work out for that. - Minor and cosmetic, but still worrisome when noticed: There is a cosmetic defect (D-005575) in the JSS Database Utility for Windows. No matter what you set Tomcat's maximum memory allocation to, it will always appear to reset to 512MB. This is not accurate and is a cosmetic/visual only. If you run tomcat7w.exe and look at the Java tab, you'll see the correct amount of memory allocated. - When you first upgrade/install a JSS, use any browser other than IE to open it. Sometimes, IE will appear to hang on "initializing database" and it will stay there forever. It isn't actually locked up in most cases, it's something to do with IE; open it up in any other browser on any computer that is able to reach the JSS, it will go through the initialization, and you'll be able to use it in IE again.

Most of these things aren't necessarily deal breakers in terms of Windows server vs. Other OSes, or impossible to get around, but it's nice to know about them. It looks like you'd commented in the thread with the above info, but since it may also be helpful for someone else, I figured it was worth re-copying here.

I've also let your TAM know about this thread so he can keep an eye on it as well.

Thanks!
Amanda Wulff
JAMF Software Support

[clifhirtle]

@ boettchs Up until 9.3.2 I saw issues in upgrading Casper on our Windows 2008 server in our test environment, specifically around Tomcat's webapps folder and keystores. In those cases I had to actually clear out the Tomcat folders entirely before running the upgrade, so the Windows installer would not get caught up and fail.

Since 9.3.2 I'm seeing great success in our test environment, with zero of the issues of past.

Production upgrade scheduled for tomorrow so keeping my fingers crossed all goes well as what I'm seeing in our test VM.

[were_wulff]

@clifhirtle

Just because I'm curious: Had you manually stopped Tomcat prior to running the installer at all?

I've seen issues, prior to 9.31, where the Windows installer didn't seem to accurately check or didn't wait long enough for Tomcat to properly stop, and would try to go ahead with moving the usual files it backs up (server.xml, TomcatSSLKeystore, and a .p12 file if one exists for a third party SSL cert) while Tomcat was still running.

What would happen, at that point, is that it couldn't really move OR overwrite the files needed to let the installer go through, and it would error out and fail, then leave everything in a broken state when it did its rollback.

What we found was that stopping Tomcat manually, prior to running the installer seemed to stop that from happening since the service was already fully stopped and the installer had no issues moving/copying the files it needed to.

If I recall, in 9.31, development changed a few of the timeout & polling thresholds for the installer and how it checked to see if Tomcat had stopped and checked to see if Tomcat had properly started again, which took care of a lot of the issues.

Initially, I know the hard timeout limit for checking to see if Tomcat had restarted successfully was 30 seconds, and if Windows took longer, the installer would assume that the Tomcat installation had failed, the installer would fail, it would roll back, and everything would fall apart.

In 9.31, the timeout was changed from 30 seconds to 5 minutes (it polls every 3 seconds until it gets a 'yep, Tomcat is running!' back from the OS or until the 5 minutes runs out, at which point it reports a failure, if memory serves) based on the fact that the test VM (2008R2) I was using to demonstrate the problem to one of our developers took a whopping two minutes and thirty seconds to get Tomcat restarted to the point that the installer would accept the service as 'running'.
Granted, my often abused test 2008R2 VM probably isn't the best comparison to a real world production server, but I'd rather have the hard timeout limit on the high end than on the low end.

I think they also raised the time limit at the start of everything, where the installer checks to make sure the Tomcat service has stopped from 30 seconds to around three minutes as well.

I'm in the "bitten dozens of times, still shy" category where Tomcat, the JSS Installer & Windows are concerned, and still recommend manually stopping the Tomcat service AND backing up the Tomcat directory on Windows servers prior to running the installer just in case.

Amanda Wulff
JAMF Software Support

[clifhirtle]

Thanks for the detailed response @amanda.wulff! I gave up testing point releases after multiple failed upgrades in 9.2x so it may well have been the timeout value. I do recall preceding upgrades with every possible system-side tweak I could manage beforehand, unchecking the read-only attribute on JSS folder, import DB vs upgrade, backing up JSS folder, stopping Tomcat and MySQL services, etc. My support profile is a graveyard of various failed upgrade cases which (although we worked around by dumping entire folder contents) I ultimately did not feel comfortable going ahead with on our non-replicated, single prod server.

9.32 has been flawless so far, other than a minor error after importing the prod DB into our test VM indicating an import error using cleartext password on the command line (or something to that effect). I had assumed that was the result of having dropped and added the jamfsoftware DB in the MySQL console too many times in the test VM. Ultimately it did not seem to effect the usability of the DB in our test environment so I just kept going with the upgrade and have seen no other issues. Hoping for the same in prod tomorrow!

[were_wulff]

@clifhirtle][/url

"Warning: Using a password at the command line can be insecure", by any chance?

If that's a yes, then we're using MySQL 5.6 and will want to make sure, both in dev and production, that we've got strict mode turned off.

Turning strict mode off usually makes it stop complaining about that (oddly, not using a password at all does as well), and stops anything over 40 characters from being truncated.

The warning itself is something Oracle considers to be expected behavior and it normally doesn't cause any major problems, however, if turning strict mode off doesn't stop it entirely, going back to 5.5 certainly does as the feature that causes that warning to happen isn't present in 5.5.

Turning off strict mode. In 5.6 it is on by default, and is not supposed to cause issues, but in cases where we're finding it is, the following KB has steps to turn it off: https://jamfnation.jamfsoftware.com/article.html?id=318 While you're in there editing things to turn strict mode off, be sure to up your max_allowed_packet to at least 512M under both mysqld and mysqldump, and up your max_connections to something higher than the default unless you've got a small--under 300--amount of devices.

The long, sometimes angry, occasionally NSFW due to profanity in the comments bug report about that warning: http://bugs.mysql.com/bug.php?id=66546

Keep in mind that, sometimes, that error is misleading and the real error follows the warning about passing the password at the command line being insecure.

I suppose it's obvious by this point that I may have run into this many times before, isn't it? :)

Amanda Wulff
JAMF Software Support

[clifhirtle]

Interesting @amanda.wulff! That would explain it. Prod on MySQL 5.5, test on 5.6.

Do not have my.ini but doe have "C:ProgramDataMySQLMySQL Server 5.6my-default.ini" and of course it does identify as having strict mode enabled.

As you mentioned, it does not seem to effect anything with the DB import, more an annoyance.

[were_wulff · Accepted solution]

EDIT: This post was 2 years old, and the information in the comment is no longer recommended or safe to use on the JSS.

Therefore, I am removing it, as using it can cause adverse performance in a JSS that is 9.8 or higher.

[scottb]

@Everyone that has replied here since yesterday - Wow - I thank you. I have been working with our rep (Dan the Man!) non-stop and didn't even see this until this AM.
I have to spend some time to look over what's been posted here.

What we are seeing in a nutshell after an uninstall of JSS and reinstall of 9.32 is that after imaging, some Macs have a corrupted Self Service.app. It comes across coincidentally at 2.2MB which is the same size as the Self Service.app before it's unzipped. So we are wondering if it's not being unzipped correctly at install, or if the install/download is somehow truncated. Oddly, the icon on the SS app shows normal, but we know right off if it's bad due to the file size. Normal is ~5.2MB.

If we install a QuickAdd.pkg or enroll again, it installs fine and works fine. If you download the QuickAdd.pkg from the JSS it comes down OK every time. I'm more unsettled by the inconsistency than anything here. I don't know if there's other issues we haven't found yet or if this is the only one.

I will read all that's here and see if there is anything we haven't tried yet. I reallyl appreciate the assistance offered up here. Great community!

Scott

[scottb]

@amanda.wulff][/url: Thanks for the suggestions. Going through one at a time, and I did find that one variable in the below is missing:

Variable: JAVA_HOME Value: C:Path oJavajdk.1.x.x_xx
Variable: JRE_HOME Value: C:Path oJavajre7
Variable: JSS_HOME Value: C:Path oJSS

Also want to tell you that we did indeed disable the SEP software by putting the server into a disabled group and stopping all SEP services. That was a chore as we have to get about 6 different humans from different groups together as everyone has specific and limited rights.

Dan K has be doing some other things as well, but I thought Id report that the variable above is not on the server. We upped the Tomcat max memory during the install to 6GB as the server has 32GB and is it's sole purpose.

Cheers, Scott

[clifhirtle]

@amanda.wulff thanks a million for the detailed info here, invaluable as a reference guide for anyone making the upgrade.

Our long-awaited upgrade from 8.73 > 9.32 went off seamlessly. FWIW, we did reduce DB size by flushing all logs > month back, removing stale devices, and cleaning up any old policies, packages, etc that were still hanging around on the CasperCorner up to no good.

Since upgrading I did see Tomcat stop 2x, but since working with our local TAM to ensure the various Tomcat/MySQL connections/memory settings are updated to 1/2 device RAM on Tomcat memory, 512MB on max allowed packets, and bumping MySQL DB connections from 100 > 601, as recommended here all is well again in the Land of Casperism!

[were_wulff]

@clifhirtle

+1 for seamless upgrades!

+1,000,000 for house cleaning prior to said upgrades! If more people did that, there would be many smoother upgrades. :)

Glad to hear it went smoothly!

Amanda Wulff
JAMF Software Support

[scottb]

Our database is always lean. Pre-upgrade, it was ~30MB. We did everything possible to try to get that seamless experience, but alas on Windows, it was not to be with us. Most of the issues seem to have resolved, but the oddities we saw have me still concerned as there really are no sound root causes for things we experienced.

It's almost as if it took a week for the JSS to get caught up and function properly after we finally got a clean install of 9.32. The hardest part was trying to troubleshoot an all-new interface, etc. while putting out the fires. But in the end, it's the best way to learn...
JAMF was a lot of help during it all, and while I'm still smarting from it, I can't wait to install 9.4 ;)

[were_wulff]

@boettchs

Ah, Windows.
Windows can be kind of squirrelly with the JSS installer.

Most commonly, it's anti-virus that ruins the party, and if we can temporarily disable its on access scanning things usually go through pretty smoothly.

I also tend to recommend stopping Tomcat manually prior to running the installer on Windows servers, just to make sure we don't run into the weird situation where Tomcat hasn't fully stopped and our installer, instead of pausing like it's meant to, just decides to plow ahead anyway (which causes the install to fail when it can't move certain Tomcat config files due to them being in use).

Really, with Windows servers I err on the far side of caution when running upgrades and do the following:

- Antivirus off. Make sure any domain based AV is temporarily disabled as well.
- UAC off.
- Logged in as a local admin, not a domain admin. I've seen problems that arise when the domain admin doesn't have a proper set of local admin rights.
- Manually stop Tomcat.
- Manually backup your Tomcat folder to the desktop, just in case. - Run the installer from a Command Prompt opened as Administrator with: msiexec /i /drag/the/installer.msi /lv c:JSSinstall.log (That's a lower case L in that command). That gets us a debug log of the installer if something fails, so we can look at it and figure out exactly what tripped it up. If it goes fine, the JSSinstall.log it makes can be removed.
- IF it fails, don't click OK, go to the Windows emp directory and copy the JSS installer log file that gets auto-deleted when OK is clicked.
- Send those to your TAM to look at if it has failed.
- Celebrate if it went smoothly, manually restart Tomcat, and continue on with your day.

The 9.3 installer took care of a few of the timing issues that caused a lot of headaches, which is nice, but we do still see a lot of...arguments...between antivirus software and the installer; when the installs fail, it's usually that.

Amanda Wulff
JAMF Software Support

[clifhirtle]

FWIW, our server is Windows Server 2008 R2

[bmak]

Thanks for the detailed responses from everyone especially @amanda.wulff and @clifhirtle

I apologise for not having responded back earlier with the times it took to upgrade but the upgrade only took a few hours once flushed the log files, cleaned up the DB a bit as well as any stale clients.

I hope everyone has good success upgrading from JSS 8.xx to 9.xx!

[hzimmerman]

Just to add on to what Amanda said:
1) To emphasize, on a Windows server, make sure you are running the installer as a local admin, not a domain admin.

2) Unless you have a strong reason not to, I would suggest turning off plug-in inventory collection (Settings -> Computer Management -> Inventory Collection -> Software -> Plug-In). I think everyone on this forum would agree that the plugin table can grow astronomically. If you need collection about a certain plug-in, chances are there is an existing Extension Attribute to collect the data you need.

3) For new server setups, the first thing you should do after your JSS comes up is replace the SSL certificate, either with a valid signed cert, or with a self-signed certificate from the JSS's built-in CA. I wish the JSS setup assistant would run that as a last step.

-Hank

[were_wulff]

Good points, @hzimmerman !

Just to expand on #2, because it's something I see frequently enough:

In 8.x specifically, the tables that tend to grow to massive sizes are:

plugins - If you're not using that data for anything and don't need to collect it, turn it off in the inventory collection preferences and truncate the table to clear out the unneeded data. If you are using it, typically a good manual log flush will trim it. If it doesn't, we usually truncate (after making a backup) and let it rebuild as devices check back in.

unixapps - If you're not using that data, turn it off in the inventory collection preferences and truncate the table to clear out the unneeded data. I'm not sure I've ever run into someone who was actively using that data, but I'm sure there are a few people out there! If you are using it, we do the same as we do for plugins/fonts: Take a backup, try some manual inventory log flushing, and if it still doesn't come down in size, truncate and let it rebuild as devices check back in normally.

unix_executables - Same as unixapps.

fonts - Same as plugins. If you are using font data for reporting, typically a good manual log flush of inventory related logs will trim it. If it doesn't, we usually truncate (after making a backup) and let it rebuild as devices check back in.

Fonts and Plugins can grow pretty large in 9.x as well, but with regular log flushing, they shouldn't ever get to an out of control size.

Applications can get fairly large as well, but that can be taken care of by manual log flushing and by going in and making sure we don't have more policies updating inventory than are absolutely necessary.

When you're looking at your 8.x JSS summary, and are looking at the list of tables, make certain that your list of tables ends with whitelist.
As a side note, a 9.x database's last table in the list is vpp_user_accounts, so if your full summary ends on something that isn't vpp_user_accounts, please get in touch with your TAM ASAP.

If it doesn't end with whitelist, it means one of two things:

1) The summary didn't generate properly. It happens sometimes; try it again and see if we see whitelist as the last table.

2) There is table corruption somewhere. Most commonly, applications, plugins, fonts, unix_executables or unixapps; when those tables get overly large, they sometimes fall over on themselves and crash. In that case, get in touch with your TAM; we can try a mysqlcheck -u root -p --auto-repair -o jamfsoftware to see if it helps, but if we know which table it is, and we know it's large, we typically truncate, and then run the repair-optimize.

If you see tables that show NULL sizes or have trouble viewing a summary if you select table row counts, that is usually caused by having third party custom views; off the top of my head, the only one I can recall offhand that does that is Web Help Desk. The custom views are to allow integration between their software and the data in the JSS, and since the JSS can't always parse the data from the views, we sometimes see that a table row count being selected will cause a blank page to show up when generating a summary, and the sizes for those views show up as having a NULL size.
It's harmless and doesn't affect anything, upgrades will go through just fine with the custom views as the JSS won't try to do any updating to those, we just have to get the data a different way if we need it.

Amanda Wulff
JAMF Software Support

[scottb]

At this stage I would like to suggest augmenting the JAMF Docs with some of this info above. Obviously, it's very useful and certainly will save many people headaches if they know these things before they begin upgrades/setup. This site and the people are great, but either offer a secondary document or update the current ones to reflect this stuff. My 2¢ after having been in a fire for a week :)

[were_wulff]

@boettchs][/url

Man, if I could get into the Tech Writing department...;)

We do pass these sorts of things along to them for consideration, though!

I have a lot of Evernotes that I frequently use for common things that pop up, they're usually what I reference for threads like these if I don't remember everything off the top of my head:

Windows install prep.
Windows install that didn't go quite as intended and now the JSS is down.
Guidelines for Tomcat & MySQL settings broken down by amount of devices.
Guidelines on what to look for in a JSS summary in terms of potential future (or current) problems.
Lists of log files, their default locations, and what they're useful for.
How to put applications into debug mode.
How to put the JSS into full debug mode and get a clean, debug only log.
How to figure out what's causing that dang "Problem detecting MDM profile after installation" error in 9.3x.

I have a few documents that touch on what to look for in log files, but those are a bit harder to make useful to other people, as I'm not the strange log file genius that some of my colleagues think I am (shh!).

I look at text block shapes, which are pretty much uniform in console and adjust to however large the window is made, and pause when I see something that doesn't 'match' with what I know good, 'nothing is wrong, everything is great, here are some statuses' text to look like, and I know very well what the 'shape' of a stack trace blog looks like so I always pause on those.

In debug logs and system.log files, I can scan and filter out WARN vs. ERROR vs. DEBUG and know when to pause based on that as well--and when I find something, as embarrassing as it may be to admit, I don't just know the answer sometimes and toss the error into Google to find it!
It's a bit tricky to document that in a way that's helpful to other people, though, "Look for shapes, then use Google!" isn't super helpful.

I've also got a messy, long, disorganized (in the sense that everything in there is 'how recently I decided to add it' and not in any sort of category order) document of useful MySQL commands that touch on everything from finding bad profiles to clearing APN queues (for everything or for single devices), manual log flushing, changing JSS account privilege levels, resetting a forgotten local JSS user password (we use that to avoid having to truncate the users table if AD is down and nobody remembers the local JSS admin account password), undeleting deleted config profiles (Spoiler: They aren't actually deleted when you delete from the webapp, they just get a switch flipped in the database to make them not show up and not be in use anymore! If you accidentally delete one that you want to get back without re-creating, contact your TAM, we can usually still get them.), and things of that nature, but since a lot of the commands in that document can be pretty destructive (if they aren't just select statements) in the wrong hands, those typically are only sent out on a case by case basis and nearly always sent out over a webex in a troubleshooting situation.

I do try to keep our internal things up to date, and I'm fairly certain that most of my own notes that I mentioned above are on our internal sites, since others in Support can search those and use them, but official documentation and JAMF Nation KBs go through an approval process (which is a good thing!), so it sometimes takes a little bit longer to get those up and going. I've also been working on cleaning my Evernotes mentioned above up a bit so they can be sent to our Tech Writing department for tweaking and, hopefully, approval to go up somewhere public facing.

TL;DR: I agree, and I like documentation. The more documentation, the better, especially for things like this where we're aware that Windows tends to require a few extra steps to make sure updates to the JSS go smoothly in addition to just the general 'prep list' sort of things that should be done before major upgrades and, in my 'I like my databases tidy' view, before any upgrade at all.

Amanda Wulff
JAMF Software Support

[scottb]

@amanda.wulff: Of course we probably all have our own notes and as you said, things are available here. However, I still feel that (based on my own experiences and reading others) that some of these tips should be in the (as it stands today) very lean upgrade instructions. Thinking one has followed the process only to find that there were many other tidbits to try not only wastes valuable time, but can unnecessarily lead to mistakes and problems that could have been avoided.

Learning from trial by fire can be good - lord knows we have learned a lot that way recently, but some things I'd prefer not to learn when clients on a global scale are affected.

I and others appreciate the efforts put in by volunteers here as well as JAMF employees. None of this is meant to knock anyone - merely a request to make such info more obvious to those of us who are new or fairly new to Casper. This thread alone is a good stand-alone PDF in my docs. Hopefully others will benefit from it as well.
While I like good documentation, there is not much I like less than creating and maintaining it :)

Scott

[RobertHammen]

@amanda.wulff - I second that your post on MySQL and Tomcat settings NEEDS TO become a knowledge base article.

I would actually be in favor of JAMF flushing and re-writing all of their Kbase articles for Casper 9 (or setting up a new Kbase for 9 and later). They aren't organized well, don't search well, and there are some workflows/processes that require going through a couple of articles and taking snippets from both.

Totally revamping and expanding technical documentation should be high on JAMF's list. And the support folks should be eating the dogfood. If you have tons of info stored outside of the knowledge base, you're doing it wrong.

Should I submit this as a Feature Request, in order to get some traction?

[rtrouton]

Feature request is now in to have the Tomcat and MySQL optimization information added to the JAMF Nation KnowledgeBase. Please vote it up.

https://jamfnation.jamfsoftware.com/featureRequest.html?id=2494

[bmak]

@rtrouton nice feature request it's already got my vote thanks for putting that in!
It's going to help loads of people upgrading from 8.xx to 9.xx

[donmontalvo]

@amanda.wulff Wow, your posts are incredible!!!