Jamf not working on mysql 8.0.30 PI110425

[dhausman]

FYI don't upgrade to mysql 8.0.30. There is a PI110425 for it. Jamf will not run.

[gda]

Oh, to late... 😣

Are there any workarounds available?

[dhausman]

Support was not able to give me a work around or patch. You will need to install a previous version of mysql and restore a database backup. Luckily I was working in my dev environment before updating production. I took a snapshot of the server before I upgraded Jamf and MySql, so it was not a big deal, I just restored the snap.

If you get rolled back, you can install 10.40.0 just don't update sql until you hear form Jamf that it is safe.

[gda]

Failed to downgrade MySQL.

It's working again with a restore of the database server. Before i did the restore I dumped the jss database and imported it after the restore.

Thx for your help. :)

[jamfsucks]

Just to understand: so you're running with MySQL 8.0.30 now and the 'fix' was dumping/restoring the DB contents?

[jamfsucks]

We're also affected since installation docs nowhere mention to disable security updates: https://docs.jamf.com/technical-articles/Installing_Java_and_MySQL_on_Ubuntu_for_Jamf_Pro_10-14-0_or...

We also pay thousands of bucks per year for something called support and now this very support told us to downgrade to 8.0.28 the following way: https://pastebin.com/raw/p3ZeeBQr

Well, there are at least 23 reasons to not downgrade to this version:

* https://nvd.nist.gov/vuln/detail/CVE-2022-21412
* https://nvd.nist.gov/vuln/detail/CVE-2022-21413
* https://nvd.nist.gov/vuln/detail/CVE-2022-21414
* https://nvd.nist.gov/vuln/detail/CVE-2022-21415
* https://nvd.nist.gov/vuln/detail/CVE-2022-21417
* https://nvd.nist.gov/vuln/detail/CVE-2022-21418
* https://nvd.nist.gov/vuln/detail/CVE-2022-21423
* https://nvd.nist.gov/vuln/detail/CVE-2022-21425
* https://nvd.nist.gov/vuln/detail/CVE-2022-21427
* https://nvd.nist.gov/vuln/detail/CVE-2022-21435
* https://nvd.nist.gov/vuln/detail/CVE-2022-21436
* https://nvd.nist.gov/vuln/detail/CVE-2022-21437
* https://nvd.nist.gov/vuln/detail/CVE-2022-21438
* https://nvd.nist.gov/vuln/detail/CVE-2022-21440
* https://nvd.nist.gov/vuln/detail/CVE-2022-21444
* https://nvd.nist.gov/vuln/detail/CVE-2022-21451
* https://nvd.nist.gov/vuln/detail/CVE-2022-21452
* https://nvd.nist.gov/vuln/detail/CVE-2022-21454
* https://nvd.nist.gov/vuln/detail/CVE-2022-21457
* https://nvd.nist.gov/vuln/detail/CVE-2022-21459
* https://nvd.nist.gov/vuln/detail/CVE-2022-21460
* https://nvd.nist.gov/vuln/detail/CVE-2022-21462
* https://nvd.nist.gov/vuln/detail/CVE-2022-21478

Is this a common problem with JAMF (no focus on IT security)? Asking since normally a colleague (currently on vacation) deals with this software.

[dhausman]

To be fair... mysql 8.0.30 was released 7/26 the same day is Jamf 10.40. If jamf were testing against a pre-release version of mysql, it may have worked fine on 7/25.

I'm sure it will be sorted out.  Probably just bad coincidence they were released the same day. This is why snapshots and backups are important.

[jamfsucks]

Well, the 'problem' is JAMF support talking about 8.0.28 being the last supported MySQL version. And want me to downgrade to this version in August 2022. While since April 2022 it is well known that this version suffers from a large number of 'Easily exploitable vulnerabilities' (quoting one of those 23 CVEs that accompanied the security update version 8.0.29).

That's now more than three months between CVE release dates and JAMF ignoring 8.0.29...

[dhausman]

If it helps, I am running 8.0.29 and it seems to work just fine.  I agree that if they are relying on mysql they should be working to validate that Jamf works on current versions of mysql. Realizing day one support for mysql may not happen, but it should be addressed in a reasonable amount of time.  

[dlondon]

Just out of interest - has your DBA configured MySQL so that it only takes connections from the server running Tomcat?  I'd guess that would greatly reduce the exposure.  I know that by default it only accepts connects from the local machine which for many people meens it needs no change.  We had it for another server as we had the database server separate from the Tomcat server so had to add the Tomcat server but that was it - no other access.

[jamfsucks]

What is an DBA?

[miles_c-d]

db admin

[warren_gowen]

MySQL 8.0.30 runs fine with Jamf 10.39.1

[jamfsucks]

We're on 10.39.0 (for reasons unknown to me – colleague gets back from vacation next week). After reinstalling mysql-server_8.0.30-0ubuntu0.20.04.2 and restoring Jamf database the wall of garbage still talks about 'Fatal error logged during server initialization: There was an error updating the database schema. Contact JAMF Software Support' while at the same time 'jamf-pro server status' outputs happily 'running': https://pastebin.com/raw/ZUST1ipf

I guess that's a road to nowhere since 'JAMF Software Support' talks about MySQL 8.0.28 being the last supported version and Jamf just ignoring important security updates now for 4 months.

That's obviously the 10.39.1 'release notes': https://docs.jamf.com/10.39.1/jamf-pro/release-notes/Whats_New_in_This_Release.html – there's not even a release date mentioned. Seriously?

[gsjlusky]

My 10.39.1 server was running fine with mysql 8.0.30 for a few weeks...  until the server got rebooted for other updates.  Then I got that Jamf Pro Startup Suspended banner, and saw this in JAMFSoftwareServer.log:  [ERROR] [erverThread] [TableMySQL ] - Updating encoding: java.sql.SQLException: (conn=8) 'Changing the STORED status' is not supported for generated columns.

mysql doesn't support downgrading, but I was able to do a mysqldump of the 8.0.30 all databases and restore to 8.0.28.  

mysql doesn't have old versions listed in the Packages file in their apt repo, so you have to download the individual debs and install with dpkg.  Hope this helps saves someone a few minutes:

------
#!/bin/bash

BadVer="8.0.30-1ubuntu18.04"
UrlBase="http://repo.mysql.com/apt/ubuntu/pool/mysql-8.0/m/mysql-community"
PkgVer="8.0.28-1ubuntu18.04_amd64.deb"
PkgList=`dpkg -l | grep $BadVer | grep '^ii' | awk '{print $2}'`

for i in $PkgList ; do
  wget "$UrlBase/${i}_$PkgVer"
done
--------

mv /var/lib/mysql /var/lib/mysql_8.0.30.broken

Then dpkg -i *.deb

And apt-mark hold mysql-community-server mysql-community-client mysql-common

This would be a good time to re-run mysql_secure_installation if that's something you normally do on a fresh install.

Next restore the databases, then run a FLUSH PRIVILEGES; in mysql, and finally restart jamf.tomcat8.service

[jamfsucks]

Why do you call it 'mysql_8.0.30.broken'? It's the other way around: the vulnerable version is 8.0.28 for which 8.0.29 is a security update for which 8.0.30 is a bugfix release.

The only software that's broken wrt the issues we experience here wasting our time by pathetically needing to downgrade to vulnerable software is JAMF Pro.

[foobarfoo]

We're also on 8.0.30 and are currently on JAMF pro 10.36.4 now, but we plan to upgrade soon. So happy we saw this in time. JAMF, it's appaling that even the latest versions (including betas) still require 8.0.28!

[jamfsucks]

Since JAMF support still isn't able to name a release date of their software being able to be usable with MySQL security updates (they ridiculously still insist on using 8.0.28) we decided to downgrade to the vulnerable version.

The instructions provided by JAMF support are partially garbage so we ended up with this:

# download vulnerable MySQL version packages in a batch
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-server_8.0.28-1ubuntu20.04_amd64.deb-bundle.tar
tar xf mysql-server_8.0.28-1ubuntu20.04_amd64.deb-bundle.tar 
rm mysql-server_8.0.28-1ubuntu20.04_amd64.deb-bundle.tar

# backups
cd /
tar cf - /etc/alternatives/my.cnf /etc/mysql | gzip -c >/daten/backups/mysql-conf-backup.tgz
tar cf -  /var/lib/mysql* | gzip -c >/daten/backups/mysql-daten-backup.tgz

# deinstallation
apt purge libmysqlclient21 mysql-client-8.0 mysql-client-core-8.0 mysql-common mysql-server mysql-server-core-8.0

# move old dirs out of the way
cd /var/lib/
mv mysql mysql-8.0.30
mv mysql-files mysql-8.0.30-files
mv mysql-keyring mysql-8.0.30-keyring
cd /etc/
mv mysql mysql-8.0.30

# installation
cd $download-dir
dpkg -i mysql-common_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-client-plugins_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i libmysqlclient21_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-client-core_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-client-plugins_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-client-core_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-client_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-client_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-server-core_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-community-server_8.0.28-1ubuntu20.04_amd64.deb
dpkg -i mysql-server_8.0.28-1ubuntu20.04_amd64.deb

# keep the OS vulnerable by declining security updates
apt-mark hold libmysqlclient21 mysql-client mysql-common mysql-community-client mysql-community-client-core mysql-community-client-plugins mysql-community-server mysql-community-server-core mysql-server

Then recreating database and user for the restore of the formerly dumped JAMF database:

mysql> CREATE DATABASE jamfsoftware;
Query OK, 1 row affected (0,00 sec)

mysql> CREATE USER 'jamfsoftware'@'localhost' IDENTIFIED WITH mysql_native_password BY 'secret';
Query OK, 0 rows affected (0,00 sec)

mysql> GRANT ALL ON jamfsoftware.* TO 'jamfsoftware'@'localhost';
Query OK, 0 rows affected (0,00 sec)

Afterwards a jamf-pro database restore succeeded the first time.

And guess what? It's security update time and what's included of course?

apt list --upgradable | grep mysql
libmysqlclient21/focal-updates,focal-security 8.0.30-0ubuntu0.20.04.2 amd64 [upgradable from: 8.0.28-1ubuntu20.04]
mysql-client/focal-updates,focal-security 8.0.30-0ubuntu0.20.04.2 amd64 [upgradable from: 8.0.28-1ubuntu20.04]
mysql-server/focal-updates,focal-security 8.0.30-0ubuntu0.20.04.2 amd64 [upgradable from: 8.0.28-1ubuntu20.04]

But we need to run vulnerable software since JAMF doesn't care about IT security and ignores important security updates of one of the required components for more than 4 months now. Simply unbelievable.

[donmontalvo]

Unfortunately I have to chime in on this thread too. #sigh

Internally hosted Jamf Pro 10.39.1 running MySQL 8.0.29.

We need to update to Jamf Pro 10.40.1, which isn't possible due to this warning:

https://docs.jamf.com/10.40.0/jamf-pro/release-notes/System_Requirements.html

Seems like we're dead in the water, unless Jamf can provide a way forward to 10.40.1 that does not involve rolling back MySQL from 8.0.29.

[dhausman]

@donmontalvo I am running 10.40.1 with mysql 8.0.29 and have not had any issues.

[foobarfoo]

From what I've read 8.0.29 should also work, while 8.0.30 doesn't.

[jamesreynolds]

I'm also self-hosted Jamf Pro 10.39.1 running MySQL 8.0.29. I just updated to Jamf 10.41.0 and it is working fine.

[Axellink]

Anyone knows if PI110425 was resolved with Jamf 10.41.0?  Had to upgrade to MySQL 8.0.30 unfortunately

[foobarfoo]

No it wasn't, since the release notes still state that 8.0.28 is the version you should use. So you most likely have a time bomb, meaning that the next time you restart your JAMF instance, it won't start.

[Axellink]

Yeah had to rollback the server sadly back to .29.  I was told the next version of Jamf should hopefully contain the fix. Due out in October roughly.

[donmontalvo]

8.0.29 works too, based on input from folks on this thread, and confirmed by Jamf Support.

[oli]

Anyone know how to upgrade Ubuntu from 18.04 to 20.04 without the need to install mysql 8.0.30?
do-release-upgrade force me to prior install all app updates prior. I held back mysql* updates using apt-mark hold.

[donmontalvo]

@dhausman, @foobarfoo, @Axellink, and @jamesreynolds sorry for the late response, but thanks for confirming that 8.0.29 works with 10.40.1. Jamf confirmed as well, so based on the feedback you guys gave, we're up to 10.40.1. Love this forum!

[oli]

@donmontalvo

Jamf Pro 10.41 also works with 8.0.29 in my environment!

[gfury]

Any updates on this?  MySQL  8.0.31 is due out soon.
I'm a database administrator who is trying to keep my system patched and secure.

[oli]

Jamf Support told me, that the Jamf Pro 10.42 Beta 2 is supporting MySQL 8.0.30.

[PI110425] Jamf Pro no longer fails to initialize when using MySQL 8.0.30 or later.

[oli]

Jamf Pro is now supporting mySQL 8.0.30 als you can read almost at the bottom of the release notes:
https://docs.jamf.com/10.42.0/jamf-pro/release-notes/Resolved_Issues.html

[PI110425] Jamf Pro no longer fails to initialize when using MySQL 8.0.30 or later.

[oli]

I just upgraded first Jamf Pro to Version 10.42.0 and upgraded after that to mySQL 8.0.31.
Jamf Pro is starting and working normal.
Maybe this information will help you in your decision to upgrade your mySQL

[donmontalvo]

This is great news, we plan to update soon.