Migration to a fresh instance would require re-enrolling all of your devices into that new instance. Depending on how many devices, whether your users are admins on their devices, and other factors, that could be a very painful process.
I would start with identifying things that can be deleted. There are some tools for checking out if objects are in use, etc:
https://github.com/BIG-RAT/Object-Info
https://github.com/jamf/Jamf-Pro-Object-LookUp
https://github.com/ninxsoft/Kmart
And then you can use Prune to delete:
https://github.com/BIG-RAT/Prune
I always recommend backing up objects before going through and deleting, and I'm not just talking about a database backup. You can use Jamf Migrator to save out XML versions of your objects. Or, go look at Rich Trouton's scripts for downloading XML copies:
https://github.com/rtrouton/rtrouton_scripts/tree/main/rtrouton_scripts/Casper_Scripts
Best thing about either of those methods is that if you accidentally delete something you can use the API to put it back.
As far as items scoped to sites, you can re-scope those items from the top level so that when you do remove the site it does not break anything. Configuration Profiles would be the big thing since removing those can cause pop-ups in your user's faces. Create Static Groups for each site, or tag your systems in some way and use a Smart Group. Tagging can be a plist hidden somewhere on the device that you read into an Extension Attribute (or a text based EA or pop-up based EA).
trying to picture this mess that has you considering a new instance.. guess to me I would not think it would be hard to build the desired groups structure and profiles side by side with the old, and when ready change the device assignments. If its just hard to look at just put an asterisk at the beginning of what you are building (so alphabetically it sits on the top of the list).
@stevewood @jpeters21 Thank you! I probably worded my question in a way that made it seem more complicated than it actually is, but both of your answers are extremely helpful. Much appreciated!
Just wanted to say that I've been where you are just this year! It's possible to get through it! Documentation and planning are you friends! We're gonna be heroes by the time we're done!