Update: Apache Tomcat Vulnerability

Garrett_Denney
New Contributor III

Hi Jamf Nation,

Apache Tomcat recently announced a security fix for a high-severity vulnerability in their product. Because Jamf Pro requires Apache Tomcat and security is of utmost importance, we are passing on the following information so that you can take steps to mitigate the vulnerability if you have an on-premise environment.

Please note: This issue does not impact Jamf Pro instances hosted in Jamf Cloud or other Jamf products. This issue only impacts on-premise Jamf Pro customers.

We recommend immediate mitigation via one of the following actions:
1. Comment out the AJP Connector in server.xml and restart the Jamf Pro Tomcat service
2. Add a rule on your firewall to disable inbound connections to the Jamf Pro server on port 8009

Additional information about this vulnerability is available in Apache’s release notes.

If you have questions, please email success@jamf.com. For assistance mitigating this issue in your environment, please contact Jamf Support.

1 ACCEPTED SOLUTION

tvagle
New Contributor II

I believe there is a typo in this post and the email that went out recently

This section should reference server.xml if I'm not mistaken not sever.xml

"We recommend immediate mitigation via one of the following actions:
1. Comment out the AJP Connector in sever.xml and restart the Jamf Pro Tomcat service..."

View solution in original post

39 REPLIES 39

tvagle
New Contributor II

I believe there is a typo in this post and the email that went out recently

This section should reference server.xml if I'm not mistaken not sever.xml

"We recommend immediate mitigation via one of the following actions:
1. Comment out the AJP Connector in sever.xml and restart the Jamf Pro Tomcat service..."

austin_nill
New Contributor II

Which lines exactly am I commenting out in my server.xml?

<!-- Define an AJP 1.3 Connector on port 8009 -->
   <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443"/>

nstrauss
Contributor II

@austin_nill That's basically it.

<!-- Define an AJP 1.3 Connector on port 8009 -->
<!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Don't think it's more complicated than that.

d428apple
New Contributor II

What’s the location of that file?

H3144-IT
Contributor II

Why not update Tomcat to the patched Version?

seabash
Contributor

@jdh1979 Depends on server OS...
https://docs.jamf.com/10.19.0/jamf-pro/install-guide-linux/Installed_Files_and_Folders.html
/usr/local/jss/tomcat/conf/server.xml

https://docs.jamf.com/10.19.0/jamf-pro/install-guide-windows/Installed_Files_and_Folders.html
C:Program FilesJSSTomcatconfserver.xml

d428apple
New Contributor II

I’m on macOS 10.14.

levans
New Contributor II

@jdh1979 for macOS, its in /Library/JSS/Tomcat/conf/server.xml

dreinold
New Contributor II

So, question, what will this impact if we comment out that line? Does JAMF use the AJP connector functionality at all? Can we test to see if AJP is used, if so, how do we go about that?

Is there a way to find out which version of Tomcat we are running with JAMF? Looking through some of the config files now, but on Windows Server 2012 R2, I don't see Apache installed as its own thing in Program and Features, so I imagine it's embedded in the JAMF install?

Found the versions of Tomcat with JAMF Pro installed: https://www.jamf.com/jamf-nation/articles/380/apache-tomcat-versions-installed-by-the-jamf-pro-installer

damienbarrett
Valued Contributor

I've just tried editing /Library/JSS/Tomcat/conf/server.xml by commenting out the AJP section so it looks like this:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Bounced Tomcat and the JSS won't startup. Reverted it back (bouncing Tomcat after each change) and it starts up fine.

Not sure what I'm doing wrong. Seems so simple; getting unexpected results.

damienbarrett
Valued Contributor

Never mind. I'm an idiot. I thought I was commenting out the line but wasn't. Too early; need caffeine.

scafide
Community Manager
Community Manager

@H3144-IT - great question. If you are doing manual installs of Jamf Pro in your environment, upgrading to the patched version of Apache Tomcat is a valid remediation as well. For people using the installer, since there are pathing changes, we recommend commenting out the AJP connector for ease of continuous upgrades.

@jdionne - No Jamf products use the AJP connector, we use the HTTP connector in the server.xml, making commenting out this connector in our opinion the easiest path to remediation, if you use our installers, until our next major release of Jamf Pro.

benducklow
Contributor III

@jdionne I have the same ask into Jamf about the potential impact of commenting out that line of code in the server.xml. A perfectly good question in my opinion. I see all it does is redirect traffic, but I am not aware of what would use port 8009..

For others, you can find the specific version of Apache you have via the Jamf Summary as well.

iVoidWarrantiez
New Contributor III

So macOS Firewall. How do you block a single incoming port on the macOS Firewall? I can block all incoming but not really wanting to do that. Thanks in advance.

dreinold
New Contributor II

@damienbarrett I got the same thing, so I reverted back to having it 'working'. Couldn't reach our JAMF instance locally (on an Azure server) or from an external computer (Macbook in this case).

At this point, with it now up and working, I will leave it as-is. I'll see if I can get a firewall rule on the Azure server to block on port 8009, though.

Jennifer_Green
New Contributor
New Contributor

Question: When I comment out the line below, Tomcat will not restart. Is anyone else having this problem?
<!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> --&gt; -->

benducklow
Contributor III

@Jennifer.Green I've done 3 servers thus far that were on 10.19. No issues whatsoever.

Jennifer_Green
New Contributor
New Contributor

@benducklow, I am running 10.18, could this be the problem? Should an upgrade to 10.19 occur and then make the change to the server.xml file?

Thank you - Jen

scafide
Community Manager
Community Manager

Hey folks, if you’re having issues, please don’t hesitate to reach out to Support.

Commenting out the connector is not dependent on the version of Jamf Pro you’re running.

Thanks!

sgorney
New Contributor III

@scafide Will this be fixed in the 10.20 release installers?

mschroder
Valued Contributor

To everybody that has problems after commenting out that line: You are most likely facing a feature on xml that does not allow nested comments. So make sure you only comment out that one line, or simply remove it, do not try to span your comment including the comment above the line in question. I fell into the same trap ...good that I can still learn a bit from my mistakes...

scafide
Community Manager
Community Manager

@sgorney Yes, the next major release of Jamf Pro will have installers which contain the most recent version of Apache Tomcat, which has this mitigation enabled by default.

atomczynski
Valued Contributor

Hello Folks,

Reviewing the thread thus far @austin_nill and @damienbarrett It appears your comments show the same syntax yet @damienbarrett you say this did not work for you. Can you elaborate?
Can you provide syntax that did work?

levans
New Contributor II

@Jennifer.Green

Looking at yoru screenshot you have a additional characters?

Yours : <!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> --&gt; -->

How it should look: <!-- <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Not sure if that matters, but I'd start there.

Cheers
Lee

seabash
Contributor

@Jennifer.Green aside from this forum possibly mangling your exact syntax (and pardon if obvious), the start <!-- and ending --> should effectively comment-out lines in question, specifically the AJP Connection mention at start of this post. Depending on how you edited the server.xml file, perhaps your perms got changed, which would affect starting of Tomcat. Owner and group should both be jamftomcat . Hope that helps and you are back up and running.

Sounds like @damienbarrett sorted out his issue already?

Jennifer_Green
New Contributor
New Contributor

@seabash ,@levans Thank you for the help, I will try this out and let you know.

Jennifer

mhegge
Contributor III

Do we have a definitive answer on this?

benducklow
Contributor III

@mhegge I've done 4 5 servers thus far (10/18.0 & 10.19.0) by just editing the server.xml file. Done on both Windows server 2012r2 and macOS Mojave. I did the following:

  1. Backup the server.xml file for starters (/Library/JSS/Tomcat/conf/server.xml on macOS and /Program Files/JSS/Tomcat/conf/server.xml on Windows)
  2. Run Notepad.exe as Administrator (or do a $ sudo nano /Library/JSS/Tomcat/conf/server.xml in Terminal on the macOS)
  3. Edit the file (commenting out the line listed above), then Save
  4. Restart Tomcat service (unload and load the launchdaemon on macOS)
  5. Validate server was back up and running successfully.

Pretty dang easy stuff here. Doing the production server in the AM. No issues whatsoever. Jamf confirmed there are no ill-side-effects by doing this as well.

bradtchapman
Valued Contributor II

Just upgraded to Jamf Pro 10.19 (planned) this morning, and then bounced the servers with the modified server.xml file. No issues. Port 8009 wasn't even open inbound to our DMZ nodes, but better to be safe than sorry.

jules1987
New Contributor II
To everybody that has problems after commenting out that line: You are most likely facing a feature on xml that does not allow nested comments. So make sure you only comment out that one line, or simply remove it, do not try to span your comment including the comment above the line in question. I fell into the same trap ...good that I can still learn a bit from my mistakes...

Thank you for pointing that out.
After commenting out the line, my Tomcat didn't come back up again (Jamf Server itself had no issue though). Only after removing the line completely (after having copied the server.xml file of course for a backup), it worked.

Frances_R
New Contributor II

@jules1987 I had the same issue and after removing the entire line instead of commenting it out it worked for me as well. Strange. I'm a bit new to Linux administration so I wonder why that would be.

isaac_yang
New Contributor

One more thing we noticed today. jamf-pro database config list is all not configured. We have added max connection to 100 (a little excessive for two servers), and max buffer size to 2GB.

We had one big crash today, had to take everything offline to bring back online. So far, stable. Fingers crossed.

Let me know if you want me to pull any log for you ahead of the WebEx.

Thank you.

dgreening
Valued Contributor II

The comment out worked fine on my clustered Jamf 10.19 Windows 2012R2 hosted environment!

atomczynski
Valued Contributor

@nstrauss Your instructions were perfect!

Performed this after business hours in my clustered/hosted environment.

dreinold
New Contributor II

@mschroder, @mhegge; the line in question should be around line 79 in the server.xml file. It should look like this:

<!-- Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

Note the comment at the beginning and the end (which I have bolded, but the preview doesn't look much more emphasized). Apparently we cannot span these (that is, open a comment on one line and use it for multiple lines). Each comment must be its own line, with its own open and close syntax

EDIT: Also wanted to share my experience on this. We did the comment, after reading up here that spanning comments is a no-go.

We restarted Tomcat, waited about 10 minutes, trying to login to JAMF every couple minutes. No go, so we thought it might be the MySQL instance needing to be restarted. Got an error for a null table, which baffled us, and led us down a bit of a rabbit hole to fix it (we didn't make any changes to the database).

Restarted Tomcat a second time, and it immediately came up. So for anyone hitting any error, you can try to restart the MySQL and/or Tomcat a few times, it should eventually connect up as it should. At least for JAMF 10.17, in my case.

donmontalvo
Esteemed Contributor III

@damienbarrett wrote:

Never mind. I'm an idiot. I thought I was commenting out the line but wasn't. Too early; need caffeine.

Here ya go:

caffeinate -id “@damienbarrett”

#tongueInCheek

--
https://donmontalvo.com

mhegge
Contributor III

We are still failing security scans.

mh53j_fe
New Contributor

I have been asked to expose our Jamf Pro application to the internet so that our WFH employees can get updates. My plan is to use an Apache server in the DMZ to proxy redirect to the Jamf server inside our firewall. Oh, and we are running Jamf Pro 10.9.0. Since the AJP protocol is necessary for proxy redirects, I assume Jamf won't work if I comment out the line in server.xml.
@Jamf Employees: What do you recommend I do? We installed Tomcat bundled with Jamf. What version of JAMF fixes the AJP vulnerability in Tomcat?

sdagley
Esteemed Contributor II

@mh53j_fe If you're not opening port 8009 to the outside world on your firewall this shouldn't be an issue for you.