Regarding the Criteria of Only Individual or Only Institutional keys, my guess is its picking up extra systems because its possible to have the Institutional key installed without the Mac being encrypted. For example, if it was previously encrypted using a policy, but later decrypted, the Institutional key remains installed on the system. Later if you enable encryption, it sees and uses the previously installed Institutional key.
That would also explain why its picking up all 39 Macs and the other SG criteria only looking for encrypted boot partitions is only picking up the ones that are actually encrypted, i.e, the 30 of 39. So, my guess without being able to examine anything is, you have 9 Macs that were either previously encrypted but now aren't, or got the Institutional key installed on them at some point, but never actually encrypted.

What results do you get if you use criteria of just?:
FileVault 2 Recovery Key Type is Only Institutional
My guess is you may pull up over the 39 Macs.

Filevault status reporting on Yosemite Macs is a known defect (D-007885) - see this thread: https://jamfnation.jamfsoftware.com/discussion.html?id=12390#responseChild71923

Hopefully a fix roles in soon. Our JAMF rep recommended using @rtrouton's filevault extension attribute (found here http://derflounder.wordpress.com/2011/10/13/filevault-2-encryption-status-check-script/) until the native reporting is fixed. I made a separate smart group based on that EA criteria to use for auditing the accuracy of our primary encryption report in the mean time. The downside to this approach is you're going to need to wait for the next inventory update from a machine to get an accurate status from it.

Thanks @jasonaswell that's what I was looking for.

I was also reading about the @rtrouton's FV2 EA. (To be fair he told me about it himself on IRC)

Happy to help! I'm guessing we'll see 9.62 drop with a bunch of defect resolutions any day now...

Sometime in the next week or two is what I'm hearing. Keep waiting for it. Hopefully it has a ton of defects resolved with minimal new ones introduced...

A bit sooner than that it appears. Just got the email that 9.62 is out.

Looking through the 9.62 release notes I see that the issue of Yosemite FileVault 2 status being incorrectly reported was fixed. D-007885 for reference.
There are actually a ton of fixes under JAMF Software Server listed in the notes for this version, so looks like they squashed a good number of bugs. Nice work JAMF!

9.62 did fix where computers upgraded to Yosemite computers weren't showing up as encrypted BUT the other issue where the recovery key is still in the database but it disappeared from the management tab in the GUI is still there even though I was told by JAMF 9.62 would fix it. So we have a group of machines out there that I can't get to their recovery keys.

@chriscollings - Ouch, that sucks! So, no indication on how to get them to show up again?

I hate to harp on things, but this problem illustrates exactly why I would love for JAMF to give us some other way of accessing those Recovery keys outside of the web app GUI. See my FR here: https://jamfnation.jamfsoftware.com/featureRequest.html?id=1861
Right now, that is the sole way they can be accessed. You can't see them via the API, you can't export them into a file. They are locked into the web app view.

The FR is Under Review, so it gives me some hope, but losing access to those keys can be a real problem. This is especially true if you happen to not be using an Institutional key as well, and/or if you're seeing this happen with some 10.8 Macs, since the process to re-issue a key can't be done on Mountain Lion systems.

That really sucks! I have been re-issuing keys to all of our 10.10 upgrade tester machines, and it looks like I will have to keep doing so. Boooo!!

Well one piece of kind of positive news is that the last few machines that have upgraded to 10.10 after we upgraded to 9.62 have not lost their recovery keys in the GUI. My pure speculation is that there was some logic tied to the association of the computer to the recovery key that gets severed when the inventory is updated to say the machine is no longer FV2 encrypted and because of the 9.6/9.61 bug with Yosemite inventories reporting back erroneously that the machine was not encrypted that that association got severed by mistake.

As far as I could tell from talking to our rep that the recovery key disappearing from the GUI was not an actual targeted fix of that defect number for the smart group reporting issue.

But yeah, still got to fix the machines that did upgrade and lost their recovery key. If any of you haven't voted up that one feature request for API access to the recovery keys then please do so we can get these things out of the JSS easily :)