MySQL 5.7.9 on 10.11.2 showing weird behavior where process appears to crash, but is still running?

rwatt
New Contributor II

We had our Casper server (Mac mini) crash a few weeks back, so we decided to upgrade to newer hardware.

The server (9.81) is online and running with clients communicating. Nothing seems out of the ordinary from that standpoint. The problem seems to be with the MySQL server possibly. Since this was a new machine I decided to keep everything current. So it's running OS X 10.11.2 and MySQL 5.7.9.
 
I cannot get the JSS database utility to connect reliably. It will report:

Unable to connect to the database: mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

If I restart the MySQL service, I can get the JSS database utility to work and make a backup, but eventually it fails again. Oddly, the MySQL status from the command line says running, but the preference pane in System Preferences keeps jumping between Stopped and Running.
 
I'm wondering if there's a compatibility issue with this version of MySQL? Do I need to downgrade to 5.6.x? I didn't see any warnings against using 5.7.x it in the JAMF forums. Or is it an El Capitan issue? Both? We were originally running Mavericks before. I guess I should have stayed with what was working and not get greedy. :-)

I had read about there being an issue with port 3307 in com.oracle.oss.mysql.mysqld.plist. But that doesn't appear to be it. There wasn't any hardcoded port in the plist file. I added "<string>--port=3306</string>" just in case that was the issue.
 
While MySQL is running, I can run "sudo /usr/local/mysql/support-files/mysql.server status" and it will report:

/usr/local/mysql/support-files/mysql.server: line 365: pidof: command not found
ERROR! MySQL is not running

But the server is running, the site is accessible, and Casper is working. The JSSDatabaseUtil.jar will connect just fine. Then for some unknown amount of time, the JSSDatabaseUtil.jar will stop connecting with the error I mentioned above. So far my only fix is to unload and load the plist file:

sudo launchctl unload -w /Library/LaunchDaemons/com.oracle.oss.mysql.mysqld.plist
sudo launchctl load -w /Library/LaunchDaemons/com.oracle.oss.mysql.mysqld.plist

But that only lasts for so long. I can verify everything...

sudo lsof -i -P | grep -i mysql
mysqld 16110 _mysql 28u IPv6 <removed> 0t0 TCP *:3306 (LISTEN)
mysqld 16110 _mysql 51u IPv6 <removed> 0t0 TCP localhost:3306->localhost:51473 (ESTABLISHED)
mysqld 16110 _mysql 54u IPv6 <removed> 0t0 TCP localhost:3306->localhost:51474 (ESTABLISHED)
mysqld 16110 _mysql 57u IPv6 <removed> 0t0 TCP localhost:3306->localhost:51475 (ESTABLISHED)
mysqld 16110 _mysql 68u IPv6 <removed> 0t0 TCP localhost:3306->localhost:51476 (ESTABLISHED)
mysqld 16110 _mysql 82u IPv6 <removed> 0t0 TCP localhost:3306->localhost:51477 (ESTABLISHED)

Then I attempt to log in to MySQL via the terminal:

mysql -u root -p
Enter password: 
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 102

I go back to check the JSSDatabaseUtil.jar and it can no longer connect again. If it helps, here are some example mysqld.local.err log reports this around the same time:

2015-12-22T19:07:15.716888Z 27 [Note] Access denied for user 'admin'@'localhost' (using password: NO)
2015-12-22T19:15:18.356625Z 0 [Warning] File Descriptor 1104 exceedeed FD_SETSIZE=1024
2015-12-22T19:16:32.938129Z 0 [Warning] File Descriptor 1177 exceedeed FD_SETSIZE=1024

Obviously something is trying to access something with 'admin'@'localhost', but this user doesn't exist. Not sure if this is somehow causing the problems.

+--------------+-----------+
| user         | host      |
+--------------+-----------+
| jamfsoftware | localhost |
| mysql.sys    | localhost |
| root         | localhost |
+--------------+-----------+
3 rows in set (0.00 sec)

I'm at a loss as to what else to try, besides wiping the server and starting over with Mavericks and 5.6.x.

Thank you for the help.

1 ACCEPTED SOLUTION

rwatt
New Contributor II

Just wanted to let anyone who reads this know what I ended up doing to resolve the problem. I completely removed MySQL 5.7.9 and cleared out any MySQL remnants from the server. Then I installed a fresh copy of MySQL 5.6.28, re-ran the JSS installation, and imported the database backup I took before removing 5.7.9. Now everything is running normally. The MySQL process reports correctly and the JSS database utility is actively running its backups at the scheduled time.

For me, there was not any issue with importing a 5.7.9 database in 5.6.28.

Thanks again for the tips and suggestions to downgrade the database server version.

View solution in original post

9 REPLIES 9

rcorbin
Contributor II

There are a couple of other threads about MySQL versions. In this one I was asking if anyone had tried 5.7.

https://jamfnation.jamfsoftware.com/discussion.html?id=10271

In that one @bkvines said that he had tried it and had some issues.

I also talked about some bugs in 5.6 in this thread.

https://jamfnation.jamfsoftware.com/discussion.html?id=12835

keaton
Contributor
Contributor

Currently, only MySQL 5.5.x and 5.6.x are supported by the JSS. https://jamfnation.jamfsoftware.com/article.html?id=28

I would recommend moving to MySQL 5.6.x and have you reach out to your JAMF TAM who will be able to give you a hand. Some of these issues may not be related to the MySQL version but it will be best to move from MySQL 5.7.x until support is added in the future.

Best regards,
Keaton

mattbomarc1
New Contributor

I'm using MySQL 5.7.10 in my dev environment with JSS 9.82 with no issues, but it's better to stick with what JAMF is supporting in production environments.

rwatt
New Contributor II

Just wanted to let anyone who reads this know what I ended up doing to resolve the problem. I completely removed MySQL 5.7.9 and cleared out any MySQL remnants from the server. Then I installed a fresh copy of MySQL 5.6.28, re-ran the JSS installation, and imported the database backup I took before removing 5.7.9. Now everything is running normally. The MySQL process reports correctly and the JSS database utility is actively running its backups at the scheduled time.

For me, there was not any issue with importing a 5.7.9 database in 5.6.28.

Thanks again for the tips and suggestions to downgrade the database server version.

jhalvorson
Valued Contributor

@rwatt Thanks for following up to your post. Ours is currently at 5.5.x. I was considering going to 5.7 but now I'll work and test to get to 5.6.28.

taugust04
Valued Contributor

Please don't take this post out of context...

...but up until October (when 5.7 was released for production use), the stated SQL requirements were not version "5.5 or 5.6", but "5.5+." So there may have been some confusion on whether 5.7 was supported or not until that document changed.

Personally I have it running on 5.7.10 right now without any major issues (that I know of, yet), but I know I'm one issue away from having to revert if there's a major problem that requires a support call. The only difference (from this specific situation) is that my installation is running on OS X 10.10.5 rather than 10.11. As others have said, it would be best to follow JAMF's officially supported versions.

mpermann
Valued Contributor II

@keaton can you comment on whether MySQL 5.7.x is an officially supported version yet or not?

keaton
Contributor
Contributor

Hey @mpermann,

MySQL 5.7.x is not officially supported at this time. We will update our KB and documentation when we feel our product is fully compatible.


Keaton

maxbehr
Contributor II

Was having something very similar on El Capitan. Found an article about having to sym link the mysql socket file from where MySQL uses, to where Mac OS expects it. I ran

ln -s /tmp/mysql.sock /var/mysql/mysql.sock

now have not had any weird MySQL problems.