The criteria name changed at some point during the later 8 series releases.. Its now called "Enrolled"instead of "Managed"
But unfortunately that information still won't do you any good. By definition, only Enrolled or Managed devices can even be in a Smart Group. so its impossible to make a Smart Group of non managed Macs. You can locate unmanaged Macs in an Advanced Search, but since these aren't groups you can't get notified of changes.
The best you may be able to do is create a Smart Group for Macs that haven't checked in within x amount of days. You will undoubtedly get legit systems appearing there, for example, someone on a long vacation or maternity leave or whatnot, but it should still alert you to systems that have potentially been unmanaged.
Also, you could look at what Rich Trouton put together based on the preso from the Facebook team. He has info on his blog on how to use his process that re-manages any un-managed systems using local LaunchDaemons and scripts and a QuickAdd package, etc. Its not perfect, because anyone that can run a sudo jamf command can also disable any LaunchDaemons, so there's still a possibility of something being permanently unmanaged. But its a first line of defense anyway.