how to tell if end users are using VPN over Wi-Fi on Mac

You could test a policy with the "Network State Change" trigger containing a script to check if the expected utun interface is in use if the WiFi is actively connected.  The things I would consider checking here are the utun results from ifconfig, and networksetup -getairportnetwork <interface>.  scutil --nc may also work, depending on the VPN.  Maybe also check to see if some IP that should only be available over VPN is responding.  

As for enforcement, depending on the VPN and your needs you could add to the script concept above to start your VPN app if WiFi is connected with no VPN connection -- or just notify your users to connect.  

From what I can tell the VPN config profile doesn't offer this, but there may be other ways I'm not aware of.

You'd want to really test that the "Network State Change" trigger sensitivity is not too high/low for this use case -- things like the computer sleeping, etc -- the script should account for reconnect times in those cases.  It could also potentially be done with a launchagent that checks at some interval.  There's some potentially helpful info in this thread as well:  https://community.jamf.com/t5/jamf-pro/network-state-change/m-p/178738/highlight/true#M167587