Use the actual process name and not the application name. So for example, if the application in question is named "My Great App.app", you can find the process name either by running the app and then using the Terminal to list running processes - ps axwww for example, and locating the name there. Or you can usually use Activity Monitor and find the running process.
Finally, in many cases the process name is the same as the executable located inside the app bundle's MacOS directory. I would probably use one of the first 2 methods above as they are usually safer than the last one.

Once you know the process name. Say for our made up app above it is just mygreatapp, then you enter that into the Restricted Software title. No .app or any other extension should be included. Jamf will see the app running, even if the user changes "My Great App.app" to "Foo.app" The process name will not be changed and it actually can't be changed short of getting the code and refactoring it and compiling it under a different name, which I'm guessing would be out of reach for almost everyone.

The above is the way I've been using Restricted Software in Jamf for years now and it hasn't let me down. That said, there is one item you should be aware of. If you use the Delete function in the Restricted Software, it's not going to delete the entire app bundle this way. It will only delete the process, usually again the executable in the MacOS directory of the app bundle. But this effectively destroys the application from running again, since it's dead without its main executable.