JSS on Linux: Upgrading from 8.21 to 8.52 breaking inventory search

chris_kemp
Contributor III

I'm trying to set up a test server to validate migration from our out-of-warranty XServes to Linux. I get everything set up & running on Centos just fine (a Red Hat variant that we use internally), except that when I import my database backup then Inventory searches are broken, much the same as they were apparently in 8.51...

On the Inventory tab, plain text and blank searches return no results. However, Advanced Searches work normally - a search for OS is like 10.6 returns virtually all our machines, so I know the data is in there.

Searches on other tabs appear to work normally - applications, fonts, peripherals all return valid results.

Warning - tl;dr ahead...

First pass all went well, aside from the few little hoops to jump through to get Linux set up properly. Got everything up fine, running 8.52. Used Database Utility jar file to import a current backup to have some data to work with. This is when I found the search issue, and contacted JAMF Support.

JAMF Support suggested that I re-install using 8.21, import the database, and then upgrade the applications. So I blew away the database & the jss directory and installed 8.21 - all good, except the Database Utility refuses to run (Java error about needing to run as root...even though I WAS root...). So, restored the database via mysql command line. That worked fine, I was able to see my computers with a normal search.

Then I upgraded the software in-place to 8.52 - and search is still broken. >:

I'm posting this with the hope that perhaps one of you might shed a little light on where the discrepancy is. I have asked JAMF Support as well, but of course this is all still "unsupported", so I don't know if they're going to tell me much.

Anything you can offer is appreciated! :)

1 ACCEPTED SOLUTION

chris_kemp
Contributor III

The smoking gun has been found.

After exchanging emails with David Suehring, he told me that clustering was enabled on the test server. This makes sense, because we use a two-machine cluster in production. He also explained that they only do database schema updates from the master node, to prevent multiple machines from trying to update tables. This also makes some sense.

However, there are additional considerations:

  1. The database was taken from the master node.
  2. The machine which I restored the database to is not a slave; it is a standalone machine, hosting the database & essentially functioning as "master".
  3. The web interface incorrectly shows clustering as disabled, when it is in fact enabled in the database.
  4. The updates to the database, which are generally expected to run when updating the JSS, failed with minimal notification.

To make a long story short, and cut to the real FIX: 1. In mysql, update the column 'clustered' in the table 'jss_settings' from '1' to '0'
2. Restart Tomcat (I just rebooted the box)
3. Check JAMFSoftwareServer.log to see that the tables have been updated.

Hopefully, the devs will take a look at the database import/upgrade routines and improve the way this is handled. Personally, I would like to see an interactive component that checks for clustering, and then asks the user whether to run the updates or not. This is MUCH better, IMHO, than having the scripts fail in order to protect the user from shooting themselves in the foot.

**UPDATE: Apparently there is a line that says the database updates would not be run:

2012-05-10 13:59:06,657 [INFO ] [InitializeServer ] - Initializing JSS... 2012-05-10 13:59:12,752 [INFO ] [VerifyDatabaseSchema ] - Verifying Database Schema... 2012-05-10 13:59:14,114 [INFO ] [VerifyDatabaseSchema ] - Database updates will not be run from this node 2012-05-10 13:59:14,495 [INFO ] [LookupJssSettings ] - JSS Clustering enbabled.

Still, it's easy to miss that when the logs are filling up with sql errors.

View solution in original post

2 REPLIES 2

chris_kemp
Contributor III

Well...after some debugging, it seems that the problem is due to missing database fields. In particular, back in 8.21 there was no "udid" field in the computers table - it was only in the iPhones table. However, since then "udid" was added to a few tables, including computers. The lack of this table column was causing the search to fail silently. Once the column was created in the computers table searches worked again. It is noteworthy, however, that there are similar errors in the mysql logs, so there are other missing fields that may or may not break some aspect of the JSS in a similar manner. I'm working with Support at the moment to find out what all the differences in database schemas are, and will manually create these additional columns as needed.

It is also notable that this is not exclusive to Linux. The same issue manifested itself when I tried to bring our 8.21 Xserve up to 8.52.

Moral of the story: if you're not upgrading regularly, you may run into this problem when you finally try to get current.

chris_kemp
Contributor III

The smoking gun has been found.

After exchanging emails with David Suehring, he told me that clustering was enabled on the test server. This makes sense, because we use a two-machine cluster in production. He also explained that they only do database schema updates from the master node, to prevent multiple machines from trying to update tables. This also makes some sense.

However, there are additional considerations:

  1. The database was taken from the master node.
  2. The machine which I restored the database to is not a slave; it is a standalone machine, hosting the database & essentially functioning as "master".
  3. The web interface incorrectly shows clustering as disabled, when it is in fact enabled in the database.
  4. The updates to the database, which are generally expected to run when updating the JSS, failed with minimal notification.

To make a long story short, and cut to the real FIX: 1. In mysql, update the column 'clustered' in the table 'jss_settings' from '1' to '0'
2. Restart Tomcat (I just rebooted the box)
3. Check JAMFSoftwareServer.log to see that the tables have been updated.

Hopefully, the devs will take a look at the database import/upgrade routines and improve the way this is handled. Personally, I would like to see an interactive component that checks for clustering, and then asks the user whether to run the updates or not. This is MUCH better, IMHO, than having the scripts fail in order to protect the user from shooting themselves in the foot.

**UPDATE: Apparently there is a line that says the database updates would not be run:

2012-05-10 13:59:06,657 [INFO ] [InitializeServer ] - Initializing JSS... 2012-05-10 13:59:12,752 [INFO ] [VerifyDatabaseSchema ] - Verifying Database Schema... 2012-05-10 13:59:14,114 [INFO ] [VerifyDatabaseSchema ] - Database updates will not be run from this node 2012-05-10 13:59:14,495 [INFO ] [LookupJssSettings ] - JSS Clustering enbabled.

Still, it's easy to miss that when the logs are filling up with sql errors.