Every time I issue a policy with a inventory my JSS goes down...

[Sonic84]

HELP! every time I push out a policy with an inventory to all my systems my JSS goes unresponsive and I need to reboot it. The server itself has more than enough resources, but the JSS just seems to not use them....

[Fveja]

Are you running a recon after this policy?

[Krem]

Upvoted.

[Fveja]

@Krem You do realize that this is not a feature request, right?

[Krem]

@Fveja JSS unresponsive = FEATURE, JSS should use resources correctly = REQUEST

[DBrowning]

I've had that happen before. Usually means the memory allotted for the Tomcat service is maxing out. Check what you have set in the JSS Database Utility. Goto the Utilities menu then Change Tomcat Settings.

[kilodelta]

As noted above, this really sounds like a resource management issue, either with SQL or with Tomcat. In my experience, the issue is often with SQL - especially when you have a ton of clients submitting data at once. One way to quickly help this is to increase the check-in time (15 is standard, but 30 or 1 hour may be helpful on a temporary basis), and to reduce the amount of data being reported - do you really need application logs, for example?

Some questions that would help clarify things:

• How many clients do you have, and what is the check-in frequency?
• Are MySQL and Tomcat on the same server?
• What are the Max database connections allowed to SQL? Look at the DataBase.xml file to find this.
• Tomcat memory allocation settings, and the number of threads allowed. Look at Tomcat under System Settings to find this.

Assuming there isn't a major issue with your JSS server, like having no disk space, or a corrupted ROOT.war, it should be straightforward to adjust things to improve performance.

[Sonic84]

We have ~10,000 clients with a mandatory inventory once a week.
Both Tomcat and mySQL are on the same MacPro
Max DB connections is 901, max tomcat threads is 1500. Mysql averages ~90 connections. Tomcat has 8GB allocated to it and is using ~6GB.

We're not collecting application logs, running services, or usage logs.

JSS: MacPro mid-2013, 2.7GHz 12-core xeon e5, 32GB RAM, 512GB SSD with 400GB free.
Mac OS X 10.11 JSS 9.82
mysql 5.7
Java JDK 1.8.0_66

[kilodelta]

H'okay, yeah, that is way too many MySQL connections - assuming the MySQL server's max_connections variable allows that many in the first place.

As a rubric, you really only need 10-25 MySQL connections, except for the largest of sites. Even if having less connections causing some requests to queue, the reduction in parallel operations (especially on a system that doesn't have RAID) ultimately allows MySQL to work much faster. It's a tradeoff: you can work really slowly on a ton of requests at once, or really fast on a few requests at a time. Generally, the latter is better - you just want to keep an eye on the responsiveness of the JSS to ensure it doesn't get bogged down by queued requests.

In addition, at 10k clients, you really should be looking at separating out the SQL server from the JSS, and ideally be running two JSS' behind a load balancer. A single JSS can handle 10k clients if it's on sufficiently powerful hardware - but having a load balanced solution helps deal with sudden spikes of activity (like a policy with a recon), and ensures some redundancy. Ideally, you'd throw in a third "Admin" JSS on a VM somewhere that acts as the Cluster Master, and ensures you have access to the admin interface if something goes wrong.

[Sonic84]

I'll give reducing the number of DB connections a try. We're actually in the middle of getting more app server stood up. We had that kind of multi-server arrangement a few years ago, but had to abandon it because of bugs in the JSS...

[Sonic84]

unfortunately reducing the number of database connections did not help. the performance overall is about the same wether the connection limit is 24 or 450.

[kilodelta]

OK. Out of curiosity, what's the size of the Database overall, and how often are you cleaning up the tables? It may also be a good idea to chat with you TAM about best practices for Tomcat configuration and database sizing.

[Sonic84]

Our DB is only about ~6GB.