I know I experienced this way back. I think the issue (for us) was that we needed to update our ciphers on the JSS side - but I'm still trying to remember (and check my email). Or that we had to remove some older (weaker) ciphers from the list.... I feel like it was something like that.
@bentoms - I'm still struggling to recall but I'm fairly sure the issue was that we had those ciphers plus some old ones mixed in, I had to remove the old ones before the binary and Self-Service started working correctly.
Our issue was that we did the JSS upgrade - all of a sudden Self-Service policies weren't working. The logs on the local machine kept showing a handshake error (slightly different than OPs) every time a policy was called via SS.
I think the tl;dr of it was that we needed to upgrade our Java version because older versions of Java needed to use an old cipher which was upsetting Self-Service. Once we updated Java (and removed the old ciphers) I think everything was fine. Man it feels like forever ago.