F5 Load Balancer and Casper

[bmarks]

When I recently expanded our JSS, I worked with our networking team and they configured two of my RHEL Tomcat VM's to sit behind a load balancer. Since then, I've been troubleshooting various odd issues and the one thing that hasn't been looked at is the load balancer. I have no load balancer experience and our networking team has no JSS/Tomcat experience, so I'm wondering if anyone out there has a baseline of settings to use on the load balancer, in this case an F5. I know there are a ton of variables, but this seems like one of the more common setups, so basic guidelines or direction would be great. I can't seem to find much info searching. The only setting I know we need to change since initial deployment is to turn on the "remote IP valve" setting, or whatever the F5 calls it, so that client IP's are reported correctly. Other than that, I don't know anything about the methods like round robin or the other default pool settings.

[jhbush]

@bmarks the folks at JAMF have experience with F5 setups. I had a lot of issues with a VM clustered setup getting access to the DB correctly. I ended up keeping everything on the same VLAN which got al to of bugs worked out. I still have a few odd issues like duplicate licensed software added to inventory records. The upside is terminating the SSL at the LB which saves all of keystore awesomeness.

[kitzy]

Here are some common things I run into a lot that cause problems when load balancing a JSS.

Persistence isn't enabled on the load balancer
It's super important in the load balancer setting to enable persistence (or keep alive, or whatever your load balancer calls it). This ensures that a machine talks to the same webapp when doing something like running a policy or submitting an inventory update. I find 300 seconds is a good starting point.

SSL isn't being terminated on the load balancer
The SSL should be terminated on the load balancer, and the traffic should be passed to the JSS webapps via HTTP.

Proxy port isn't enabled
If your load balancer is listening on a different port than the JSS (and it most cases it will be), you want to make sure the proxy port is enabled in each webapp. You can change this in the Apache Tomcat settings in the web interface. A typical setup would be a load balancer listening on 443 and Tomcat listening on 8080. You'll want to set the proxy port in the JSS to 443.

[bmarks]

Thanks for the info. What would be the ramifications of passing through SSL so that it terminates at the VM's? I believe that may be how we set it up because of security requirements.

[bmarks]

Regarding SSL, is there a significant performance hit if it passes through the load balancer and terminates at Tomcat or are there other issues too? I have it working with SSL terminating at Tomcat, and I'm reluctant to change it since I think it was a security requirement forced upon me. But, if there are other issues and/or it's a matter of working vs. not working, than I may be able to revisit this specific configuration.

[kitzy]

There will be a performance hit, which you might not notice if you're managing a small number of devices. The other issue I've run into with that is that the JSS reports every computer as having the IP address of the load balancer instead of whatever IP they're connecting from. This may not be an issue if you're not utilizing network segments.

[cwaldrip]

@kitzy I was about to ask about the IPs. We just moved to F5s ourselves and I have started the work to get our Casper setup behind them yet. I was hoping the F5 would resolve the issue we have getting IP's of machines outside our firewall. Just something to look harder at when we get to it.

[bmarks]

The reporting of IP addresses is one of the few load balancer topics that is documented:

https://jamfnation.jamfsoftware.com/article.html?id=305

You may need to find corresponding settings on your load balancer though. That's where I am in this process as I wait for the team member who owns our load balancers to get back to me.

[dwenger]

@cwaldrip We have F5s in our environment and had the same scenario at first. I asked our network admin that manages them how he addressed the issue, and this was his reply.

It’s something called X-Forwarded-For. When the F5 talks to the server, it still uses it’s own IP to communicate with it, but it lets the server know that it is forwarding it on behalf of a client at the forwarded IP. Both the F5 and the server need to be configured to use X-Forwarded-For. On the F5, it’s a setting that I had to enable for the jss http profile.

Once that was done, I checked the Enable Remote IP Valve option in System SettingsApache Tomcat Settings and that solved the issue for us.

[bmarks]

@kitzy I'm still dealing with some issues that I think are related to the F5, but after re-reading this thread, I'm slightly confused. A header above says "persistence isn't enabled on the load balancer" but then the comment mentions a specific setting. Is that just a typo? Or, Wouldn't you want the load balancer not to do anything in regards to keeping a session alive? The session is going to stay alive as long as there is traffic, right? So, using a simple check-in as an example, wouldn't a session persistence of 300 seconds mean that a check-in would stay open for 5 minutes even if it only needed to remain open for a few seconds?

We have our F5 set to 300 seconds. Pretty much since its deployment, I've been trying to troubleshoot performance issues of unknown cause. Among other things, every night at 6pm both of my web apps (behind an F5) spike and take up all 90 connections each to our database (max_connections is set to 90.) 6pm corresponds with our log flushing time. It's escalated to the point that if I don't manually restart Tomcat each evening, the spike never subsides. The challenge is, I don't own the F5 or our database, so I have to go through other teams at a glacial pace in order to get data or try new settings. With that said, the load balancer to me seems like it might be the culprit if it's allowing connections to stay open way longer than they need to be. This is not my area of expertise.

[bmarks]

I ended up simplifying my JSS by removing the load balancer and the issue instantly went away at the next log flushing interval.

The problem now is I don't own the load balancer, another team does, and the guy on that team that was helping us is no longer in that position so I have no resources available to figure out why the load balancer might have been causing this issue.