Detect Casper Restore Partition with Smart Group

My guess is you'll need to create an Extension Attribute. I don't think there's any built in criteria to detect that.
I haven't used the Casper Restore partition in a long while now, but if I recall, its always there, mounted, but just hidden in the Finder, so a simple diskutil list and grepping out the Restore Partition should work.
Someone who actually has some Macs using it would need to give you more details though.

In the JSS you'd go to Settings > Computer Management > Extension Attributes > Name the EA and set Input Type to script. This is an example of something to start with, the JAMF community here is great and there are lots of bash experts on here that can help improve upon this.

#!/bin/sh

# Set the Name of the Restore Partition here
partitionName="Restore HD"

# Lists disks and looks for name of partition
diskCheck=`diskutil list | grep "$partitionName"`

# Returns true if any value is returned, false if null
if [[ "$diskCheck" ]]; then
    echo "<result>True</result>"
else
    echo "<result>False</result>"
fi

exit 0