Whether this will be possible might actually depend on what version of the JSS you're using. Under the 9 series you can pull information about a Licensed Software title, including the number of used licenses, using the API. This data doesn't exist under the 8 series though. The trick of course will be to set up your Licensed Software title so the JSS can actually track it properly. Is it even possible to set up a Licensed Software title for a Windows VM? I'm not certain, but maybe?
Assuming you can, if you're on version 9 of Casper Suite, you can use a script that runs as a Before script prior to the actual policy to have the Mac query the API and pull the number of licenses used that the JSS knows of. Then have it compare the humber against an integer you put in the script. If the used licenses is equal to or greater (hopefully not) than the # of available licenses, exit without moving on the actual policy to install the software. or display a message as well that all licenses are used, etc.
Edit: BTW, it would make a neat feature request to be able to set a policy to only run X number of times (successfully) before it disables itself. Would avoid needing to use scripted workarounds like I outline above. I'd vote that up!
