The quick start guides there are basic but give step by step. If you are looking for super detailed, check out the Administrators Guide, its roughly 500 pages of knowledge with step by step setup of all of the different functionalities of the Suite. Most things are listed there, otherwise search the KBs on JAMFNation
But in regards to your error, we would need to know the actual error in the policy log (from the policies page, "View Status", form the logs page, Policy Logs) to help you troubleshoot. What you are seeing is what self service presents which isnt telling you the cause for the error, just that it erred out. End users shouldnt see the cause, just know that it failed.
Shot in the dark, you probably have a new JSS setup using the installer and you have HTTP enabled (hence the bom messages) but you dont have a symbolic link created from your casper share to the tomcat instance.
If the log says that its trying to connect to https://servername:8443/caspershare/packagename and then says something like package cant be found, thats probably it.
To fix HTTP downloads, follow this KB: https://jamfnation.jamfsoftware.com/article.html?id=116
To force it to use AFP/SMB either do it in the policies "Override default policy settings" or go to settings>servers>(distributionpointname) edit server>HTTP and un-check "HTTP Downloads are enabled for this Distribution Point"
hope that helps
