JSS Tomcat Clustering Documentation?

[dgreening]

I am working on clustering our two JSS test servers behind one of our load balancers, and have found the official JAMF documentation to be extremely sparse at best. Does anyone have a good "soup to nuts" set of documentation on the procedure for clustering Tomcat? What I am having trouble with in particular is exporting the Tomcat certificate and key pair so that I can send it to the staff member who is responsible for our load balancers.

I will say that it is pretty frustrating that the more advanced Casper functionality is not documented as well as more routine tasks. I understand that JAMF makes money off of engineering engagements and the CJA cert process, however, it would be nice to have more technical documentation on items such as clustering publicly available on JAMF Nation.

Thoughts?

[psliequ]

You could instead just create your own SSL cert from scratch in whatever format the load balancer needs, one that matches the hostname of the JSS URL you're using. You then pass the unencrypted traffic to the clusters.

I've also seen cases where you simply leave the SSL certs on the clusters and pass the traffic through to port 8443 directly. There is some encrypt/decrypt performance penalty with that config but it's very simple to manage.

[were_wulff]

@dgreening

We have our overview KB on clustering (and DMZ setup) available at https://jamfnation.jamfsoftware.com/article.html?id=174.

The main reason we haven't got much for a highly detailed version of setting up clustering and load balancing has to do with the fact that it can be wildly different from environment to environment depending on what they're setting up, which OS their JSS is using, the brand of load balancer they're using (as they all have different setup and configuration instructions), whether or not they're using the built in CA for their SSL certificate, and if a third party SSL certificate is being used, which vendor it's from (as they each have just slightly different instructions). It would be next to impossible to maintain a document with enough detail to cover the majority of situations, and we don't endorse one load balancer or SSL certificate vendor over another, so we offer a couple of overview articles and, if additional help is needed we have the options of contacting your Technical Account Manager, asking on JAMF Nation, or setting up a Professional Services engagement to have them get the information needed to tailor the session to your specific environment.

Without knowing whether we're using a wildcard, third party, or built in JSS SSL certificate, or knowing which specific load balancer we're using here, it's difficult to link to specific threads that may be helpful, but here are a few:

Load balancer with wildcard cert: https://jamfnation.jamfsoftware.com/discussion.html?id=10699
Using OpenSSL to create a certificate keystore: https://jamfnation.jamfsoftware.com/article.html?id=138
Enabling SSL on Tomcat with a Public Certificate (goes into generating keystores and exporting certificates a bit): https://jamfnation.jamfsoftware.com/article.html?id=115
JSS Behind a Load Balancer: https://jamfnation.jamfsoftware.com/discussion.html?id=5254

Page 65 of the Casper Suite Administrator’s Guide has some instructions for creating a certificate from a CSR as well, though it doesn't go into great detail for the reasons I mentioned above.

If you haven't yet contacted your Technical Account Manager to get a case open, I'd recommend doing that as well.

Thanks!

Amanda Wulff
JAMF Software Support

[dgreening]

Fantastic! Thanks Amanda! I'll dig into the info which you posted.

[Kumarasinghe]

As @amanda.wulff][/url][/url said it would be harder to give specific information without knowing about your Load Balancer.

Most recommended method is terminating the SSL at the Load Balancer and pass the unencrypted traffic into clustered WebApps.

Clients-----(HTTPS)-------Load Balancer--------(HTTP)----WebApps (JSS)

In this method you need a trusted communication between Clients and the Load Balancer.

So if you can get a properly signed certificate for the Load Balancer from a third-party vendor like Comodo, VeriSign, ..etc, clients will automatically trust it. You don't need to get anything from your Tomcat to send to your other staff member in this instance. Just give them your JSS URL you are going to have on the Load Balancer and ask them to create a SSL cert for it.

But If you are creating a self-signed certificate, you need to get the CSR signed from your JSS' CA.

If you are going to use a self-signed certificate;
1) You create a Key and CSR on the Load Balancer
2) Get the CSR signed from JSS's CA
3) Upload it to Load Balancer. Depending on the Load Balancer you use, you might need to combine the Key and JSS signed Cert to single file (e.g.- if you use Pound combine them to single .pem file)

As @psliequ][/url][/url said If you are going to get a properly signed certificate from a third-party vendor like Comodo, Just use the third-party signed certificate created on the Load Balancer. No need to get JSS's CA as it has been signed properly.

[tlarkin]

Hi @dgreening

I understand the lack of documentation part, and there are so many factors involved it would be near impossible to document every scenario. However, I have set up several clustered environments with load balancers personally. Each load balancer will have a different interface, and maybe even different terminology for certain things. The most typical setup I do is the following:

- setup 3 tomcat web apps and a Casper Database all on separate virtualized boxes (includes configurations)
- configure the LB and get the CA from an external service (verisign, digicert, etc)
- Configure all certs in all locations, make sure they are signed, generate a CSR from the JSS, and grab any intermediate certs that may need to be in the full chain of trust from the CA if external.
- terminate SSL at the load balancer
- point all client traffic to only 2 of the 3 web apps. The third web app is for the admin web console only, the master web app, and make the two client facing web apps limited JSSes (headless). Also you would use the hostname of the master web app for all Casper Admin connections. That way none of the clients can hit the web interface unless they know the actual hostname of the master web app. Which if it is behind NAT and a firewall they cannot connect to it anyway (except for maybe when on premise or over VPN, depending on how your network is setup). This is also very helpful if you open up your JSS to the public and route traffic in via HTTP Proxies in the DMZ. - test and validate that all web apps are configured properly to work behind a web app and are scaled for the proper amount of traffic
- in the LB control panel I try to use real load balancing and not round robin, I also try to use a GET response of the API URL to the JSS instead of ping (ping can lead to false positives) and if you try to GET the REST API page with out authentication it will always return 401 (not authorized) if the web app is fully up, for the health check. So I configure the GET response to always look for 401
- Use persistent (sticky) connections for clients coming in to connect to the web apps
- if you are going to use the DMZ for off premise communication to devices, also setup Split-Horizon DNS

Once you grant each webapp permissions in MySQL and connect them, you can configure clustering with in the web app. You can also view current connections of each web app as well in the JSS. You should see that the web app is clustering aware from here as well. Then if you need to scale out more, you simply revamp your DB box and add another web app to the mix. So, I really use this as a starting point. From here on out I simply monitor the data center with whatever tools are available to check memory usage, disk I/O, processor usage, network traffic and so forth.

Most load balancer control panels have easy buttons to turn a feature on or off, or a button to upload a cert, create another virtual IP, and so forth. The problem is, they all have different features, and sometimes the terminology is not 100% the same on all of them. I, however, almost always have to refer to the manual when working with one since I don't work with them all day every day, nor do I always work with the same ones.

All the links Amanda posted are probably the best starting points, so she already covered that. I think I am covering more of the design of the infrastructure in this post. Again, there are so many ways to implement this, and maybe what I just outlined won't fly at some orgs, and you might have to look at a different way to designing it.

I hope this all helps, and maybe things are a bit more clear here.

Thanks,
Tom

[dgreening]

I love jamfnation! Thanks guys!!

[dgreening]

Ok, a bit more background on our particular setup.

We are using SSL-Offloading through a Juniper load balancer. We use the built it JSS certificate authority for SSL in the JSS.

This has become a bit more urgent of a need in terms of timeline, as the Thawte SSL certificate for the JSS on our load balancer is set to expire in the coming weeks, and I need to nail down the process both for renewing it, and also for setting up a similar configuration for our test environment.

I wish that documentation was created when this setup was implemented, but alas, it was not.

[jhbush]

@tlarkin][/url as a person who has taken the CJA twice (8 & 9) I highly suggest you have education services add some of the points you touched on into the training.

[michaelhusar]

If I may, I would like to come back to: ... if you open up your JSS to the public and route traffic in via HTTP Proxies in the DMZ. How would I design that? Would I have two JSS-WebApps - both in my internal LAN one on a machine with hostname casper-internal.acme.com (fully accessible for administration) and the other with hostname casper-hitbyproxy.acme.com (headless). Both configured to use the same database. Both using the same JSS URL (like casper.acme.com). Internal DNS points to the IP of casper-internal.acme.com and public DNS points via Proxy through firewall to IP of casper-hitbyproxy.acme.com
Thank you for reading that far.
When using the self signed cert of the JSS` CA: The keystone is duplicated to both machines. The cert holds the hostname of only one( - the fully accessible, internal casper-internal.acme.com). So you get error messages when you hit the public machine (casper-hitbyproxy.acme.com) saying: "This certificate is not valid - hostname mismatch"

Is there a way to get the second hostname into the certificate? Thanks, Michael

[tlarkin]

Hi @michaelhusar,

The design could be one of many ways, but the most typical way would be to put a proxy or a load balancer in the DMZ, that has the FQDN of the JSS, and then routes traffic to a load balancer inside the firewall, that would have your JSS URL as well, and it could terminate SSL there. I don't have any diagrams to show you on how it would look, but maybe if I can figure out how to make an Omnigraffle document that actually looks decent I could share one. Terminating SSL at your internal load balancer is the easiest setup, and has the last overhead. However, if your org requires SSL secure transmission internally you would have to terminate at the web app and add the keystore to each web app. If you also terminate SSL at the load balancer you won't have to have the hostnames of the web apps in your full chain of trust in your certs, since traffic from the termination point will be unencrypted.

The proxies when transparent will not require any configuration on the client side, and DNS would need to be in place. You would set up the split horizon DNS, so that externally clients point to your proxy and/or load balancer in the DMZ, and internally they look at your internal load balancer.

Now behind the internal load balancer I like to setup the web apps in a particular way. The clients will only ever hit limited access web apps, so that means all client traffic hits web apps that have no GUI. Then the master web app is off to the side and used for all IT/admin work, and the load balancer will never send traffic there. The web apps would still be clustered, but clients checking in from the proxies to the load balancer will never be pointed to the master web app, which would be a full access web app.

So, you would not need to put a tomcat in the DMZ if you are using proxies. The idea of using proxies is so that you don't put a tomcat in the DMZ which also has database credentials stored in it. So in your case I would keep both your web apps in your data center, cluster them. Use the proxy in the DMZ to point all traffic to one web app, and make it a limited access one. Then use the other web app as your master web app and use it for all your admin work. Depending on your client base, workflows, and technical needs would depend on how I would build it out.

I guess I haven't mentioned this yet, but there would also be a database box in there as well that each web app connects to. I hope this sort of better explains the concept.

Thanks,
Tom

[michaelhusar]

Hi @tlarkin
Great - thank you so much for explaining. I did not pay enough attention to "where to terminate the SSL".
To terminate at the proxy is definitely nicer.
If I have to go the whole distance: Would I make a new cert with both names using the "-ext" operator - like

keytool -genkey -alias tomcat -keyalg RSA -keypass “secretpw“ -storepass “storepassw“ -dname ’CN=servername1.acme.com, OU=Mysoftw, O=Software, L=SanDiego, ST=CA, C=US’ -keystore “/var/lib/Tomcat7/keystore.jks“ -validity 3650  -ext SAN=dns:servername2.acme.com,ip:1.1.1.1

and remove the old one

or just make a second one with the second server name as CN ?

keytool -genkey -alias tomcat2 -keyalg RSA -keypass “secretpw“ -storepass “storepassw“ -dname ’CN=servername2.acme.com, OU=Mysoftw, O=Software, L=SanDiego, ST=CA, C=US’ -keystore “/var/lib/Tomcat7/keystore2.jks“ -validity 3650

have that signed by my Built-In JSS-CA and import that as a second web server certificate?

Thank you very much,
Michael

[michaelhusar]

..there is one more thing... sorry
our network guys set up a reverse proxy, but "funny" enough when we hit the JSS through that proxy we the html is not interpreted by the web browsers :-( it shows the coding:```
<html> <body> <table class="alignLeft" width="100%" border=0> <tr> <td align="left"> <h1>Web Application Disabled</h1> <font class="assistantNote"> Your web application has been disabled. <BR/><BR/> </font> </td> </tr> </table> </body>
</html>
```
Thank you very much, Michael

[michaelhusar]

ok, ok - the laugh`s on me

DefaultType None

https://issues.apache.org/bugzilla/show_bug.cgi?id=13986

[donmontalvo]

@jhbush1973 wrote:

@tlarkin as a person who has taken the CJA twice (8 & 9) I highly suggest you have education services add some of the points you touched on into the training.

As someone who flunked out of an early 2010 CJA course pilot, it would be nice to have some training material to study up for the course. :):):)

LOL, in fairness to JAMF, they do explicitly state the course is geared towards datacenter/server folks. I plan to take it again, determined to pass this time.

"You can't keep a good frog down!" - Kermit

:D
Don

[sedden]

Is there any info out there about setting up a headless JSS? Is it as simple as adding into the setenv.sh the headless line.

I was looking to follow Tom Larkin's recommendation of having two headless JSS' and one admin one but even after configuring the setenv.sh file i can still hit the webapp from a browser on the headless JSS', unless i've misunderstood this.

Many thanks

[sedden]

Ah is this done through the JSS interface by restricting to Computer and Mobile access, or Computer access only etc?

[tlarkin]

@sedden

Yes you are correct. The Limited Access Mode is set in the GUI. This will then disable all JSS web interface options. You can then flip it back into limited access mode by altering a table in the database for those specific clustering settings. I flipped my VM really quick into Limited Access Mode, here is what it looks like in the database:

mysql> select * from limited_access_mode_settings;
+-----------+-------------+
| address   | access_mode |
+-----------+-------------+
| localhost |           3 |
+-----------+-------------+

To change each web app back to full access mode you can run this MySQL command:

UPDATE limited_access_mode_settings SET access_mode=0 WHERE address='hostname_of_webapp';

You would need to replace hostname_of_webapp to the actual hostname of the Tomcat web app you wish to change back into full access mode. The access modes are set by an integer value of 0, 1, 2, or 3.

• 0 = full acces
• 1 = Computer Access Only
• 2 = Mobile Device Access Only
• 3 = Computer & Mobile device access

So you could set the access_mode that way in the database, using access modes 0 through 3. If you want to default a Tomcat back to full access you would need to set it to access mode 0.

Hopefully this helps answer your questions?

Thanks,
Tom

[tlarkin]

I would also like to add, as my co-worker @mike.paul just pointed out to me, as long as you have one full access web app you should be able to use the web app to actually flip the Tomcat from different access modes. It appears I was over engineering my answer a bit. So, the only time my workflow would be valid is if all your Tomcats were in limited access mode.

Cheers,
Tom

[sedden]

Thanks Tom, that is indeed super helpful!

Thanks,

Scott

[sedden]

One other question, if the JSS' are setup in headless mode, does this mean that you cannot use URL enrollment? Or would this be limited to using the Admin JSS instance?

[tlarkin]

@sedden

OTA enrollment should work, you just have to set your limited access Tomcat to allow for OS X connections, iOS connections or both. All connections should still work on the /enroll URL to OTA devices.

Hope this helps.

Thanks,
Tom

[sedden]

Thanks Tom,

Has anyone got any experience configuring sticky sessions on haproxy? I think that is the issue I'm having with enrollement mentioned above, when i go to login at the enrollment URL it just returns me back to the login screen, If i down/stop the second headless JSS, so that there is now only one JSS behind the loadbalancer it then works. So it appears that, if they are both up, then when i login it is switching to the other jss.

[ftiff]

hi @amanda.wulff and thanks for your insights.

I understand it's difficult to make a comprehensive guide on this matter. However, I'd like to understand the broad concept of using a DMZ (no LB).

1/ Clients connect to the JSS DMZ

2/ JSS DMZ:
- Limited Access enabled (= no web console)
- https://jss.company.com (will be first point of contact)
- need access to DB (TCP/3306)
- need access to Apple (TCP/2195-2196 for APNS + TCP/443 for VPP)
- need access to internal AD (TCP/3268 for Global Catalog)

3/ Internal JSS:
- ?? Is only for admin purposes (=web console)
- need access to DB (TCP/3306)
- ?? need access to Apple (TCP/2195-2196 for APNS + TCP/443 for VPP)
- need access to internal AD (TCP/3268 for Global Catalog)

4/ Internal DB:
- hosts all the configuration
- need to create one user (jamfsoftware@HOST) per host

Does it look ok to you?

[kerouak]

Hey all..
So, I've got 2 Webapps running as client access behind a load balancer (KEMP)
I have a master Webapp (not LB'd) which acts as admin
SSL is terminated at the LB.

For some reason, when I apply "limited access" to the 2 LB'd webapps, they stop communicating and are shown as 'down' in the Kemp monitor.
Anyone any ideas??
Does Limited access disable a port?

TIA

[rigualj]

The port does not get disabled in limited access mode as it's still needed to serve device traffic.

I would investigate what KEMP is using for a healthcheck on your webapps. If it's just the hostname then try to append /healthCheck.html to the end.

Hope this helps steer you in the right redirection.

[kerouak]

@rigualj
Yep, Indeed it was.. Healthchecking on the KEMP LB was configured wrongly..

Thanks for the pointer mate!

[kerouak]

Can I just qury the following from a previous post
"point all client traffic to only 2 of the 3 web apps"

So, we have 2 webapps natted and resolving to our fqdn.
the 3rd webapp is admin.

The clients and certs are fqdn but the admin webapp does not resolve to this .
for some reason, clients seem to be still going through the Admin webapp (which is set as the master in the cluster

????