What is the reason to run your own Software Update server?
If it is that you want to hold back updates from reaching the clients, then yes, you do need to publish your Software Update server with a public IP, and at least firewall it to be reachable from the JSS instance's IP block.
If the reason is that you want updates to be quick and save bandwidth, you should use the Content Caching functionality instead. This is built into macOS High Sierra, so you may setup any computer as a caching server (you don't need to install Server.app). The caching functionality is actually very dynamic and mature, compared to the outdated Software Update Server (which is removed hidden in Server.app 5.4, and removed for virtual machines). Also, Content Caching doesn't need any setup on the client computers (it is automatically discovered and load balanced on the local network).
But you can't make the Caching server hold new updates until you get to test them for production. That remains the only reason to use your own Apple Software Update server, but then it needs to be reachable from the Internet in order to work with Jamf Cloud.
