Have you tried file://Applications/Self Service.app/ ?
Yep. I tried that with the same result.
I can add any of the other Dock ions with no problem on El Capitan.

(even when I change Self Service to the exact same Path structure as the others, it fails)
I've experienced this in post-enrollment and via Casper Remote. I've tried having the Self Service icon be inserted at the beginning of the dock and at the end - same problem. It is very specific to Self Service and El Capitan. it has no problems at all with other apps and other OSes.
I have seen this exact same thing in 10.11.3 testing, though not all the time and not just from the Self Service dock icon itself.
Was receiving that exact 'no application set to open...' when trying to run anything from Self Service on 10.11.3 after a test full image. Self Service would open and everything scoped to my device was there but nothing would install.
Re-enrollment did nothing to fix the dock shortcut on one device or the actual content itself on another. The message persisted.
In the end I re-imaged the test devices with the exact same setup and the message has not re-appeared in any way.
Understand that that is not an answer or real resolution but figured that I would share that I have experienced the same message.
@AVmcclint are you editing the existing dock item object in the JSS? I've noticed that with dock items, this doesn't always work. Try deleting the dock item completely, and then setting it up from scratch. Use this path.
file:///Applications/Self%20Service.app/
Hope that helps!
Deleting the entry and creating it again in JSS did the trick. Weird, weird stuff.