I would think this would be much easier done with rules inside of your mail server than with using JAMF or OS X controls.

Solving disciplinary tasks with IT tools usually won't get you far, especially when it comes to students. They will find a way to circumvent your restrictions and you will soon find yourself in an arms race unnecessary complicating your environment. That being said: if they are using your internal mailserver then i agree with @ammonsc - that should be the point of control, but this might become a problem if they are using same file formats for studying purposes: e.g. scans from books are images too.

Technological Babysitting will never beat the Shadow IT Dept. But it can be fun to try!