Update: Apache Tomcat Vulnerability

[Garrett_Denney]

Hi Jamf Nation,

Apache Tomcat recently announced a security fix for a high-severity vulnerability in their product. Because Jamf Pro requires Apache Tomcat and security is of utmost importance, we are passing on the following information so that you can take steps to mitigate the vulnerability if you have an on-premise environment.

Please note: This issue does not impact Jamf Pro instances hosted in Jamf Cloud or other Jamf products. This issue only impacts on-premise Jamf Pro customers.

We recommend immediate mitigation via one of the following actions:
1. Comment out the AJP Connector in server.xml and restart the Jamf Pro Tomcat service
2. Add a rule on your firewall to disable inbound connections to the Jamf Pro server on port 8009

Additional information about this vulnerability is available in Apache’s release notes.

If you have questions, please email success@jamf.com. For assistance mitigating this issue in your environment, please contact Jamf Support.

[tvagle · Accepted solution]

I believe there is a typo in this post and the email that went out recently

This section should reference server.xml if I'm not mistaken not sever.xml

"We recommend immediate mitigation via one of the following actions:
1. Comment out the AJP Connector in sever.xml and restart the Jamf Pro Tomcat service..."

[austin_nill]

Which lines exactly am I commenting out in my server.xml?

<!-- Define an AJP 1.3 Connector on port 8009 -->
   <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443"/>

[nstrauss]

@austin_nill That's basically it.

<!-- Define an AJP 1.3 Connector on port 8009 -->
<!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Don't think it's more complicated than that.

[d428apple]

What’s the location of that file?

[H3144-IT]

Why not update Tomcat to the patched Version?

[seabash]

@jdh1979 Depends on server OS...
https://docs.jamf.com/10.19.0/jamf-pro/install-guide-linux/Installed_Files_and_Folders.html
/usr/local/jss/tomcat/conf/server.xml

https://docs.jamf.com/10.19.0/jamf-pro/install-guide-windows/Installed_Files_and_Folders.html
C:Program FilesJSSTomcatconfserver.xml

[d428apple]

I’m on macOS 10.14.

[levans]

@jdh1979 for macOS, its in /Library/JSS/Tomcat/conf/server.xml

[dreinold]

So, question, what will this impact if we comment out that line? Does JAMF use the AJP connector functionality at all? Can we test to see if AJP is used, if so, how do we go about that?

Is there a way to find out which version of Tomcat we are running with JAMF? Looking through some of the config files now, but on Windows Server 2012 R2, I don't see Apache installed as its own thing in Program and Features, so I imagine it's embedded in the JAMF install?

Found the versions of Tomcat with JAMF Pro installed: https://www.jamf.com/jamf-nation/articles/380/apache-tomcat-versions-installed-by-the-jamf-pro-installer

[damienbarrett]

I've just tried editing /Library/JSS/Tomcat/conf/server.xml by commenting out the AJP section so it looks like this:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Bounced Tomcat and the JSS won't startup. Reverted it back (bouncing Tomcat after each change) and it starts up fine.

Not sure what I'm doing wrong. Seems so simple; getting unexpected results.

[damienbarrett]

Never mind. I'm an idiot. I thought I was commenting out the line but wasn't. Too early; need caffeine.

[scafide]

@H3144-IT - great question. If you are doing manual installs of Jamf Pro in your environment, upgrading to the patched version of Apache Tomcat is a valid remediation as well. For people using the installer, since there are pathing changes, we recommend commenting out the AJP connector for ease of continuous upgrades.

@jdionne - No Jamf products use the AJP connector, we use the HTTP connector in the server.xml, making commenting out this connector in our opinion the easiest path to remediation, if you use our installers, until our next major release of Jamf Pro.

[benducklow]

@jdionne I have the same ask into Jamf about the potential impact of commenting out that line of code in the server.xml. A perfectly good question in my opinion. I see all it does is redirect traffic, but I am not aware of what would use port 8009..

For others, you can find the specific version of Apache you have via the Jamf Summary as well.

[iVoidWarrantiez]

So macOS Firewall. How do you block a single incoming port on the macOS Firewall? I can block all incoming but not really wanting to do that. Thanks in advance.

[dreinold]

@damienbarrett I got the same thing, so I reverted back to having it 'working'. Couldn't reach our JAMF instance locally (on an Azure server) or from an external computer (Macbook in this case).

At this point, with it now up and working, I will leave it as-is. I'll see if I can get a firewall rule on the Azure server to block on port 8009, though.

[Jennifer_Green]

Question: When I comment out the line below, Tomcat will not restart. Is anyone else having this problem?
<!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> --&gt; -->

[benducklow]

@Jennifer.Green I've done 3 servers thus far that were on 10.19. No issues whatsoever.

[Jennifer_Green]

@benducklow, I am running 10.18, could this be the problem? Should an upgrade to 10.19 occur and then make the change to the server.xml file?

Thank you - Jen

[scafide]

Hey folks, if you’re having issues, please don’t hesitate to reach out to Support.

Commenting out the connector is not dependent on the version of Jamf Pro you’re running.

Thanks!

[sgorney]

@scafide Will this be fixed in the 10.20 release installers?

[mschroder]

To everybody that has problems after commenting out that line: You are most likely facing a feature on xml that does not allow nested comments. So make sure you only comment out that one line, or simply remove it, do not try to span your comment including the comment above the line in question. I fell into the same trap ...good that I can still learn a bit from my mistakes...

[scafide]

@sgorney Yes, the next major release of Jamf Pro will have installers which contain the most recent version of Apache Tomcat, which has this mitigation enabled by default.

[atomczynski]

Hello Folks,

Reviewing the thread thus far @austin_nill and @damienbarrett It appears your comments show the same syntax yet @damienbarrett you say this did not work for you. Can you elaborate?
Can you provide syntax that did work?

[levans]

@Jennifer.Green

Looking at yoru screenshot you have a additional characters?

Yours : <!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> --&gt; -->

How it should look: <!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Not sure if that matters, but I'd start there.

Cheers
Lee

[seabash]

@Jennifer.Green aside from this forum possibly mangling your exact syntax (and pardon if obvious), the start <!-- and ending --> should effectively comment-out lines in question, specifically the AJP Connection mention at start of this post. Depending on how you edited the server.xml file, perhaps your perms got changed, which would affect starting of Tomcat. Owner and group should both be jamftomcat . Hope that helps and you are back up and running.

Sounds like @damienbarrett sorted out his issue already?

[Jennifer_Green]

@seabash ,@levans Thank you for the help, I will try this out and let you know.

Jennifer

[mhegge]

Do we have a definitive answer on this?

[benducklow]

@mhegge I've done 4 5 servers thus far (10/18.0 & 10.19.0) by just editing the server.xml file. Done on both Windows server 2012r2 and macOS Mojave. I did the following:

1. Backup the server.xml file for starters (/Library/JSS/Tomcat/conf/server.xml on macOS and /Program Files/JSS/Tomcat/conf/server.xml on Windows)
2. Run Notepad.exe as Administrator (or do a $ sudo nano /Library/JSS/Tomcat/conf/server.xml in Terminal on the macOS)
3. Edit the file (commenting out the line listed above), then Save
4. Restart Tomcat service (unload and load the launchdaemon on macOS)
5. Validate server was back up and running successfully.

Pretty dang easy stuff here. Doing the production server in the AM. No issues whatsoever. Jamf confirmed there are no ill-side-effects by doing this as well.

[bradtchapman]

Just upgraded to Jamf Pro 10.19 (planned) this morning, and then bounced the servers with the modified server.xml file. No issues. Port 8009 wasn't even open inbound to our DMZ nodes, but better to be safe than sorry.

[jules1987]

To everybody that has problems after commenting out that line: You are most likely facing a feature on xml that does not allow nested comments. So make sure you only comment out that one line, or simply remove it, do not try to span your comment including the comment above the line in question. I fell into the same trap ...good that I can still learn a bit from my mistakes...

Thank you for pointing that out.
After commenting out the line, my Tomcat didn't come back up again (Jamf Server itself had no issue though). Only after removing the line completely (after having copied the server.xml file of course for a backup), it worked.

[Frances_R]

@jules1987 I had the same issue and after removing the entire line instead of commenting it out it worked for me as well. Strange. I'm a bit new to Linux administration so I wonder why that would be.

[isaac_yang]

One more thing we noticed today. jamf-pro database config list is all not configured. We have added max connection to 100 (a little excessive for two servers), and max buffer size to 2GB.

We had one big crash today, had to take everything offline to bring back online. So far, stable. Fingers crossed.

Let me know if you want me to pull any log for you ahead of the WebEx.

Thank you.

[dgreening]

The comment out worked fine on my clustered Jamf 10.19 Windows 2012R2 hosted environment!

[atomczynski]

@nstrauss Your instructions were perfect!

Performed this after business hours in my clustered/hosted environment.

[dreinold]

@mschroder, @mhegge; the line in question should be around line 79 in the server.xml file. It should look like this:

<!-- Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Note the comment at the beginning and the end (which I have bolded, but the preview doesn't look much more emphasized). Apparently we cannot span these (that is, open a comment on one line and use it for multiple lines). Each comment must be its own line, with its own open and close syntax

EDIT: Also wanted to share my experience on this. We did the comment, after reading up here that spanning comments is a no-go.

We restarted Tomcat, waited about 10 minutes, trying to login to JAMF every couple minutes. No go, so we thought it might be the MySQL instance needing to be restarted. Got an error for a null table, which baffled us, and led us down a bit of a rabbit hole to fix it (we didn't make any changes to the database).

Restarted Tomcat a second time, and it immediately came up. So for anyone hitting any error, you can try to restart the MySQL and/or Tomcat a few times, it should eventually connect up as it should. At least for JAMF 10.17, in my case.

[donmontalvo]

@damienbarrett wrote:

Never mind. I'm an idiot. I thought I was commenting out the line but wasn't. Too early; need caffeine.

Here ya go:

caffeinate -id “@damienbarrett”

#tongueInCheek

[mhegge]

We are still failing security scans.

[mh53j_fe]

I have been asked to expose our Jamf Pro application to the internet so that our WFH employees can get updates. My plan is to use an Apache server in the DMZ to proxy redirect to the Jamf server inside our firewall. Oh, and we are running Jamf Pro 10.9.0. Since the AJP protocol is necessary for proxy redirects, I assume Jamf won't work if I comment out the line in server.xml.
@Jamf Employees: What do you recommend I do? We installed Tomcat bundled with Jamf. What version of JAMF fixes the AJP vulnerability in Tomcat?

[sdagley]

@mh53j_fe If you're not opening port 8009 to the outside world on your firewall this shouldn't be an issue for you.