I have both Chrome browser and Eclipse available via SelfService and per policy. Built up some new Macs running Sierra and joined them to the JSS. Updated via 'sudo jamf policy' and both Chrome and Eclipse are installed fine.
When you launch Chrome or Eclipse the first time all is normal, set the icons in the dock to 'keep'. Restart and the icons become ? also now, launching the same apps your presented with 'this was downloaded via Safari...' and you have to go through the first run dialog again.
The apps were packaged and uploaded to JSS on a machine running El Cap, could there be some issue because of that?