I'd stick with VM's running on enterprise grade server hardware. I think moving to a Mac Mini would be a downgrade in just about every aspect. You also might evaluate how many JSS's you really need. With an appropriate amount of resources 1 JSS and 1 MySQL server should handle your 10,000 devices. Then possibly add a second JSS in your DMZ setup for limited access.

I second Chad's comments. I would absolutely NOT use Mac Minis. Get yourself some enterprise VMs or enterprise hardware running linux.

As of Jamf Server 9.99.0—notwithstanding any announcements about 10.0—there is no support for database clusters or high-availability database configurations.

Therefore, my advice would be to put the database on a dedicated server (real or VM) with a lot of cores and as much RAM as you can get. For 10,000 devices, you'll want 8 cores and at least 16GB of RAM, plus a very fast disk for the database. 250GB should be adequate for a long time. The inventory and smart group calculations for iOS devices will be smaller and a lot less complex than a computer.

One other thing... if you set up a Tomcat cluster, you also need to set up Memcached as per instructions from Jamf.

Memcached can run on the web app servers alongside Tomcat; however, it should never be installed on the database server.

as much RAM as you can get

Don't take that too literally. You really just want more RAM than the size of your database (plus some overhead for the OS). If you have a 10 to 15GB database, 24GB RAM is probably plenty. If you have a 20GB database, you might want 32GB RAM.

Do not use macOS as a large scale Casper deployment server.
There are some serious scaling issues and bugs to overcome.