Create Smart Group based on Activity Monitor process?

Two options. Can you see it with launchctl?

sudo launchctl list

(make sure you do sudo so you see root's processes as that's how jamf will run it) and see if it's listed there. If so, you can create a smart group with "Running Service 'Has' com.name.handbrake".

If it's only as a process, then you need to create an Extension Attribute

#!/bin/sh                                                                                                                                                         
hazIt=$(ps auxw |grep "Activity_agent"| grep -v grep)
if [ -z $hazIt ]; then
    echo "<result>Not on this one</result>"
else
    echo "<result>Has Activity Agent</result>"
fi

Another thing you may want to do is set up a software restriction to kill the process if it tries to spawn on the machine.