Yeah, that's why I was originally thinking the script checking FV2 status and firing off, or not, accordingly. Without some way to tell it not to execute your users will be nagged about something they already did. Not so good.
I would use a combo of the existing Smart Group, but put some logic in the script up front to check FileVault 2 status. Just off the top of my head...
#!/bin/sh
FV2Stat=$( fdesetup status | grep "On" )
if [[ "$FV2Stat" == "" ]]; then
*do your messaging stuff here*
else
exit 0
fi
Take a look at the man page for fdesetup. There's a isactive verb that will report exit status of 0 if on and 1 if off, or 2 if FileVault is "busy" which could mean its in the middle of encrypting or decrypting. In 10.9 Apple changed that command a little and you get somewhat different results with it. In 10.8 you had to do an echo $? to see the exit status. In 10.9 it now reports "true" or "false", but will still report the exit status.
Unfortunately, I've found that in 10.8 I could reliably tell if FileVault was deactivated with the isactive command, even before a reboot. Thats no longer the case in 10.9. It will still report true, even if a user completely disabled FileVault in the Preference Pane or from the command line.. Only after a reboot will it report correctly as "false"
So in short, the fdesetup status seems to be the most reliable way I've seen on both 10.8 and 10.9 to get the encryption state.