10.13.2 Supplemental Update Workaround

[scafide]

Hey Jamf Nation!

As some of you have noticed, if you're Collecting Available Software Updates during a recon, your 10.13.2 Macs that have not yet ran the Supplemental update released today may not be submitting inventory.

To resolve the issue either:

Update your 10.13.2 Mac to the supplemental update.

or

Turn off collect available software updates from inventory collection preferences within your Jamf Pro Server (if you are collecting them) until your 10.13.2 Macs have updated to the supplemental updates.

[haircut]

For what it's worth, I'm not seeing the reported problem on our Jamf Cloud instances. The clients are able to successfully recon, but the formatting of the available Software Updates in the computer object view has the version number incorrectly formatted.

Can we get some clarification on affected environments? Discussion in Slack seems to point to this being an environmental issue.

[emily]

I'm speculating here but it seems like from what I've seen that mostly on-prem Windows hosted JSS instances. 9 and 10 both appear to have the same issue.

[mhasman]

Is there download URL for 10.13.2 Supplemental Update, please? I can not find it on Apple Updates

[jmahlman]

@emily This also happened on our RHEL on-prem system.

[ClassicII]

@scafide Thank you for brining this issue to light!

@bentoms Thank you for pointing me to this thread from macadmins slack!!!

[tep]

I'm also seeing this on my on-prem RHEL system.

[mm2270]

So can I assume if we don't have the "Collect available software updates" option enabled in Inventory Collection Preferences, the problem is avoided? It sounds like that's the case, but I just want to be sure I'm understanding that properly. Because we don't have that option on and haven't for a while. Sounds like we won't see the issue.

[The_Lapin]

THANK YOU!

This was driving me crazy yesterday. Couldn't figure out why all of a sudden some of my test boxes were failing to recon. "An unknown error occurred"

Totally explains why a couple of them recon without issue now, they're the same ones that I applied the 10.13.2 supplemental update to.

On prem Windows 2012 R2 w/ JSS 9.101.0-t1504998263

[emily]

@mm2270 yes, based on my testing, if "Collect available software updates" is not enabled for inventory collection this will not be an issue. I had it off on a Dev box, recon was fine, but once I enabled it recon failed. I haven't tried turning it off again to see if it can revert back okay, but I'm assuming it would start working again after. I'll test that in a bit here.

[scafide]

Hey,

Yes, the issue lies in the availability of the update, not the update itself. When the recon runs and collects that available update, that’s the issue causing recon submission failure. Toggling collecting available software updates on and off will not resolve the issue - issue resolution today revolves around turning off the collection until your Macs are patched, or making manual SQL configuration changes, which we would suggest avoiding until we fix in the parsing.

[emily]

Note that if you disable the "Check for available software updates" feature, you're also losing a software update check during recon. By default macOS only checks for updates weekly, while you may have been benefitting from more regular checks for software updates due to having recon run at a higher frequency than once every 7 days. Just throwing that out there.

[donmontalvo]

Yea, disabling "Check for available software updates" is a bit of a problem, if you're scoping an update to computers that show it as available in softwareupdate -l.

FWIW...

https://www.jamf.com/jamf-nation/discussions/25648/high-sierra-supplemental-update-breaks-recon#resp...

[tcandela]

my 10.13.2 is also having an unknown error when running recon.

via terminal softwareupdate -l (supplemental update listed)
softwareupdate -ai

I am now installing the supplemental update 10.13.2 and let's see if this unknown recon error continues.

jamf version 9.101.0

I'm assuming that computers that directly install high sierra upgrade from the App Store will initially get this supplemental update included in the installation. I have unblocked High Sierra upgrade so users can upgrade on their own time.

[eric_difulvio]

FYI, I was having the same recon error. I completed the 10.13.2 supplemental update and it corrected the unknown recon error.

[tcandela]

After the 10.13.2 supplemental update installed i ran the 'sudo jamf recon' and it successfully completed !!

softwareupdate - l
softwareupdate - ia

[donmontalvo]

Same here, once the item (with its syntax wonkiness) was installed, and no longer on the softwareupdate -l list, recon ran fine on the LAB computers that had the problem.

Apple is looking into why the last couple (few?) Supplemental Updates' names are wonky (tic 100404578230).

[jamesgreenMatte]

Happening for us as well. JamfPro 9.100 using RHEL as host.

[brunerd]

FWIW It's not broken in 9.101 !?

Looking at the output of the CLI softwareupdate it appears the lack of version in the parentheses is the culprit... the dangling hyphen on "...Supplemental Update-" is weird too. Sloppy, Apple.

$ sudo softwareupdate -l
Software Update Tool

Finding available software
Software Update found the following new or updated software:
   * macOS High Sierra 10.13.2 Supplemental Update- 
    macOS High Sierra 10.13.2 Supplemental Update ( ), 138293K [recommended] [restart]

Here's what a 10.13.2 machine without patch inventory looks like in 9.101 as you can see the version column is all messed up...

As a test I went in Recovery mode and replaced /usr/sbin/softwareupdate with a script that output the same data but with "(1.0)" instead of "( )" ... but it seems the jamf binary was not having any of that and inventory now says "No Updates Available" - pfeh! Anyway I think that's it...

Perhaps a quick report to Apple could fix this up too: https://bugreport.apple.com/web/

Expected result: Version number in parentheses Actual result: Nothing Burger ( )

[emily]

@brunerd are you sure you have "check for available updates" enabled on your JSS? If you don't, it doesn't check during recon, and recon won't break. Alternately, the character size of the software updates table in your JSS may be long enough to accept the amount of characters of this long-named update.

I won't go into too much detail here, Jamf can do that, but based on the logging I'm seeing it doesn't have to do with parentheses or anything like that, and more has to do with the length of the name of the update and it being more characters than that table allows.

Contact a Jamf support partner/buddy/TAM/whatever they're called these days and they can tell you what to modify to allow more characters on the table to accept the longer name.

[scottb]

Well, if it makes anyone feel better, 10.13.3 Beta (17D34a) doesn't fail recon. Today.

[sburt]

FYI no issues seen on our on-prem 9.101.4 instance on Ubuntu.

[ThijsX]

Anyone knows if the supplemental update is already available as an package?

Thanks!

[ThijsX]

My bad, already found it!
https://support.apple.com/kb/DL1951?viewlocale=en_AU&locale=en_AU

[mhasman]

@txhaflaire Thank you!

[donmontalvo]

@emily Awesome, Jamf indeed confirmed the character limit issue. Tested the schema change in our QA environment as per Jamf fixed the issue for us.

We followed up with Apple so they can update ticket.

I won't go into too much detail here, Jamf can do that, but based on the logging I'm seeing it doesn't have to do with parentheses or anything like that, and more has to do with the length of the name of the update and it being more characters than that table allows.

[jason_bracy]

I also do not have the issue in JSS 9.101.4 running on an internal OS X box and external Windows box.

[cecotter]

@mhasman [https://support.apple.com/kb/DL1951?viewlocale=en_US&locale=en_US](link URL)

[mhasman]

@cecotter Thank you Chris!

[sepiemoini]

+1 for opening a case with Jamf Support.

I've received the following from Jamf Support as a fix to this issue:

Step 1: Create database backup of the current data using the JSS Database Utility. We can navigate to our JSS Database Utility found in: Mac OSX Server: /Library/JSS/bin/JSSDatabaseUtil.jar Windows Server: C:Program FilesJSSinJSSDatabaseUtil.jar Linux: /usr/local/jss/bin/JSSDatabaseUtil.jar Step 2: We will then want to grab a backup of the current JAMFSoftwareServerDatabaseSchema.xml file by pasting it to the desktop or any other location. macOS - /Library/JSS/Tomcat/webapps/ROOT/WEB-INF/xml/ Windows - C:Program FilesJSSTomcatwebappsROOTWEB-INFxml Linux - /usr/local/jss/tomcat/webapps/ROOT/WEB-INF/xml/ Step 3: We will want to modify that file and replacing what is currently there with the new data. Specifically the <type>varchar</type> value in the <table_name>available_software_updates</table_name> section. Old <name>version</name> <type>varchar</type> <size>31</size> New Data <name>version</name> <type>varchar</type> <size>255</size> Step 4: Once we have saved the changes we will want to restart Tomcat, and that can be done by using the JSS Database Utility again.

I haven't implemented this yet but imagine it'll resolve the issue. Have others received the same steps?

[donmontalvo]

Yep that’s the fix.

[StoneMagnet]

@sepiemoini Not trying to be pedantic, but the "New Data" portion of your snippet should really include the <table> and <table_name> tagged lines from the "Old" portion since the fix is to change the value for <size>, without eliminating the <table> and <table_name> lines as your post could be read as to suggest (and I realize you probably just pasted what Jamf support provided).

[sepiemoini]

Agreed @StoneMagnet and also taken directly from Jamf Support. I've modified it now to be more accurate. Thoughts?

Step 3: We will want to modify that file and replacing what is currently there with the new data. Specifically the <type>varchar</type> value in the <table_name>available_software_updates</table_name> section. Old <name>version</name> <type>varchar</type> <size>31</size> New Data <name>version</name> <type>varchar</type> <size>255</size>

[sdagley]

@sepiemoini Thanks for making that change (wouldn't have wanted someone to stumble on this thread and take the original Step 3 at its word and destroy the table if they didn't realize what was actually the fix in that snippet).

Of course the correct fix would be for Jamf to sanity check the inputs for limited size table fields because assuming a field will never exceed 255 characters (Str255 sound familiar to anyone?) is just as dumb as assuming it will never exceed 31 characters, you're just less likely to get bitten by it.

[remyb]

For those that applied this workaround please note:
Updating to a new version of JAMF pro will result in your server not starting back up as in: https://www.jamf.com/jamf-nation/articles/349/troubleshooting-the-jss-startup-suspended-issues

Don't panic, the value in the JAMFSoftwareServerDatabaseSchema.xml file was reverted to it's original state. Stop the JAMF service, re-edit the value back to 255 and start JAMF. Use this at your own risk.

[acaveny]

Does the 10.2.0 update fix the root of the problem? Seems that we should not have to manually update database schema. And, if the value is reverted, it would not seem that JAMF resolved the issue.

[remyb]

The 10.2.0 updated just reverted the database schema for this value to the old state. So for this version it wasn't fixed.

@sepiemoini could you maybe ask for JAMF's view on this by re-opening the supportcase?

[sepiemoini]

@remyb I’ll do you one better! @scafide Would you be able to weigh in on the schema settings not being rolled out with 10.2.0?

[scafide]

Thanks for the tag @sepiemoini!

The fix was done in-code, so this specific change to the JAMFSoftwareServerDatabaseSchema.xml is no longer relevant. If you are having start-up issues after putting your previous JAMFSoftwareServerDatabaseSchema.xml back in place, replace your modified file with the one created by the installer, or contact Jamf Support.

Thanks!

[acaveny]

Good Morning,

So, if I understand completely, the patch inventory issues that were discovered with the 10.13.2 supplemental release have been resolved?

Thanks all for the awesome communication!