The method that I was told to use by JAMF during the beta was to create a group, LDAP or Standard, that has Full Access and create a custom Privilege Set that allows access to Casper Admin>Save with Casper Admin and/or Casper Admin>Use Casper Admin. This will also automatically check Create/Read/Update/Delete permissions for objects needed by Casper Admin under the JSS Objects sections.
You can then add your Site Admins to that group to give them access to the Casper Admin application. Doing this does show them the Full JSS in the Sites drop-down menu when logged into the JSS website, but in testing it does appear that they can't see anything since they will get Access Denied for everything they click on while in the Full JSS.
I can confirm this is the method. However in 9.65 there is a defect about creating Standard Groups. You can not add any members to them. The work around is to use LDAP groups.
You can concatenate permissions with groups. Having an LDAP user and LDAP group does work.
Create a group: LDAP_Casper_Admin_Access
Site Access: Full Access
Casper Admin > All
If you need enable Disk Encryption Settings in the JSS Objects tab.
Create another group: LDAP_Casper_SITENAME_Acess
Site Access: Site in questions
If the user is in both groups, when they log into the JSS they will see the site drop down next to users in the top navigation bar. When in "Full JSS" They will be able to go to Computer Management and use the Packages, Scripts and other functions. They can also use Casper Admin. They will only have access to the Computers, policies and other objects assigned to that site.