Announce: v1 autopkg integration with JSS, pre-alpha but working

[Banks]

Hey folks,
As mentioned elsewhere, I've found out enough of what's possible to get started integrating autopkg with the JSS. (Want to learn more about autopkg? It's in the upcoming MacTech magazine, and a video from Greg Neagle's talk at MacSysAdmin.se 2013 is here: http://docs.macsysadmin.se/2013/2013doc.html As further background and explanation of what I'm aiming at, my pretty-complete design document is here: http://arubdesu.roughdraft.io/6aadb5f6ac8b5d57b854
And... drumroll... v.1 of the code is here: https://github.com/arubdesu/autopkg/blob/jss-integration/Code/autopkglib/JSSImporter.py

I'm not certain how idempotent it is, and I didn't build in checks for if the pkgname already exists on the file share point yet - folks should not integrate this until they have a firm understanding of what it provides! NO WARRANTY GRANTED OR IMPLIED! I'm not looking for testers just yet, but please feel free to critique anything of what I've done so far ;)
Allister

[donmontalvo]

Awesome work, kudos to you and the open source community!

[Banks]

Update: made pkg copy idempotent and lightly tested. I'd still definitely appreciate feedback about use cases, but at this point will probably consider it alpha once I've implemented creating a policy(w/self-service trigger) and update a testing group to use it.
Allister

[Banks]

Update the second: starting on smart group implementation, but just a general reminder to the brave among us - although it should go without saying, this may blow stuff up, so don't let it anywhere near prod. It is nowhere near extensively tested. I'm comfortable with calling it alpha at this point, but haven't even tried loading in the autopkg-added pkg into a policy and watching it go(having too much fun with unholy regex building.)
Allister

[sam]

This is really starting to take shape. One quick sanity check while we are at the point of groups. The ideal workflow from my standpoint would be that we already have a static group setup as our 'canary' collection of devices or otherwise that we want to deploy the package. The automation would just add the new package to a policy and scope it to the group that was passed from the recipe. From the standpoint of the JSS, it doesn't really matter if the passed group name is static or smart. The input variable of 'group_name' would be sufficient. This means we wouldn't be creating groups at all from the API side. All we do is confirm that the group name exists prior to allowing it to be the scope to the policy.

Anyone out there find this workflow to be useful, or have suggestions on other ways of achieving the scoping mechanism? Maybe this is how we are intending it to work anyway. If so, please ignore my ramblings.

Once again, @Banks][/url, thank you!

[sam]

One thing I came across that I found useful. As we are creating objects such as packages using the API, we can send the post request to two urls that will generate the object.

/packages/id/{id}
/packages/name/{name}

It isn't as apparent that the latter can be used with a POST from the documentation. Also, when used with the {name} URL, it is not necessary to include the ID tag. This is an auto-incremented value, and will be assigned regardless if we pass it or not.

Maybe useful, likely not :)

[Banks]

Re: groups, I was pretty sure I hadn't communicate my targeted workflow quite clearly enough. mrowell said in the IRC that once he manually set up the workflow I'm proposing it made more sense why I'm approaching the design this way. This new pipeline is meant to implement modularity, so there's one policy per stage in deployment - when testing it's only live in selfService. Additionally, having one policy per version of the pkg(while obviously overkill) means when it's passed the testing stage you can just scope/enable it - that's the only manual step when a human decides to make it live (presumably in a logout-in/shutdown context.) Groups do stay somewhat constant, but I'm targeting two, one of each kind, because of the common 'less than current version' smartGroup and the recommended testing static group. It's not because that's the cleverest thing and everyone, wether or not they already use a similar workflow, want to adopt it - just that if I can perform all of the steps to transfer people onto the autopkg system in a way that likely doesn't stomp on what they already have, it may influence what might be considered a best practice going forward. You can then override it to target whatever group/policy interaction you want, but something starts working immediately that could interact with however you currently deal with pkgs. Closer to making sense?

(Oh, and the db/api is smart enough on the receiving end that posting to the next currently unused id, if it had ever been assigned to an object, increments to the next never-used id anyway. I'd rather just operate all the time on ids, since I can still search and update on names every time anyways, my regex does the mapping/heavy lifting I need.)
Allister

[Banks]

Policies don't seem to do the same helpful 'iterate until you hit an available id' behavior, nor does POSTing the name seem to be working - I'm getting a '409 conflict' response with either. Traceback: https://gist.github.com/arubdesu/1ed5afabd258f616a791

First pass of smart group creation is done(will get to updating the version/pkgname values in that and the policy, once sorted out, soon.)
I've posted my template for the policy, formatted(instead of the one-liner the API accepts,) here:
https://gist.github.com/arubdesu/ec661743b27a2e5734da

I'd really appreciate scrutiny of this policy to ensure it does as little as we want.

If folks could point me in the right direction regarding this policy API interaction issue it would be great, as it seems I'm at a hard stop re: policies without it.

[brockma9]

@Banks I'm not sure if this would help you at all. But I started to work on making it so that AutoPKG will make DMGs that Casper Admin would like. https://github.com/darklordbrock/autopkg-jamf-recipes/tree/master/firefox The issue I have on it right now is the permissions on the firefox app inside the DMG are not right. I have posted something about this on the autopkg google group.

I was just thinking if anything having it so AutoDMG will build DMGs for apps that can be easily added to Casper Admin and get all the functionality, would be a good place to start.

[Banks]

Okay, I've finished the rudimentary functionality and it's live in my jss_integration branch. As recommended, all POSTs are to id/0, and I've verified pkg's show up in the interface on the newest JSS. New features: if it finds the pkg_id is out of sync in the smart group it will re-post the template xml with the correct values, whereas there being a mismatch with either the pkg_id or smartGroup_id in the policy will likewise prompt a fresh PUT.
Caveats: the process, if you weren't inspecting it thoroughly, is a somewhat cumulative addition - ideally you want the entire smorgasbord: if you're adding a pkg, while not required, it guides you to have its category set to product name. Smart groups target the version based on the parent pkg recipe, and the only policy at present relies on the the smart group and pkg_id being nested. But making progress!
Allister

[Banks]

Well the 8.x version of the API is unfortunately giving me agita, in particular when POSTing or GETing smartGroups. I can successfully create all the objects freshly in one pass, but the computergroup doesn't show the criteria I'm targeting(app name+version,) which would therefore leave the policy useless without a specific group(although I will be implementing that static testing group soon.) The other symptom of this degraded functionality on 8.x(I've tested the necessary tweaks I've made to the template for 8 compatibility on 9.x and seen no issues) is that you are not able to browse to the created smartGroup, and when you manually change the URL to view it the criteria is not there. So it seems we've got two issues:
1. API not allowing proper creation of the computer group (still using id/0, for the record)
2. The GUI not reflecting a POSTed addition when browsing, no matter how dysfunctional it is(haven't been able to test PUT yet)
We checked the API docs(mrowell and rtrouton were invaluable in independently verifying symptoms) on 8.x, and I see no discrepancy in how it's supposed to behave. Any input/suggestions greatly appreciated.

[sam]

I am also seeing a problem of posting a smart group to v8.x with the API. I recall that in previous builds you could post smart groups, but they would not update the computers automatically until the next inventory report. Not sure which is worse in this case. In version 8, the criteria is not present when we GET a smart group using the API (it is in version 9). I would assume this is further indication that creating smart groups does not work in version 8.

Does

The GUI not reflecting a POSTed addition when browsing

mean that you are not able to find the group in the GUI? If so, try looking in the static groups. In my experience, even if we are specifying <is_smart>true</is_smart> in the XML, it will create it as a Static Group.

I suppose this does pose a broader question about how the intended workflow would be best automated. For those of you out there that will use this, what would you like AutoPKG to do in the JSS:

1.) Create a computer group based on the version of the software to deploy the packaged software.

2.) Install the software to a specified group name.

In my thinking, I thought it would be more useful to have a group name that you would be able to use as an input into the recipe/parent/plist. Therefore you could specify 'All Managed Clients' (a default group for all computers), or a 'Canary' static group you create of test machines in your environment.

Either way, it looks as though if we go the route of creation of the smart group, that will only be a functionality of 9 and beyond.

[Banks]

We can have it both ways... it still looks like it wouldn't be worth it to dynamically find out what version of the JSS/API I'm talking to and branch the workflow accordingly(there is that distinct 'standalone' identifier I'm seeing in the XML header, but that's probably not dependable,) folks can just modify the recipe as appropriate, since they'll only be targeting one version of the JSS. We can ship the recipes with smart group creation commented out, at least for the time being. I'll need to also ensure static group creation/update works on 8.x, since scoping to something is the minimum dependency I'd like to satisfy in the policy.

[johnnasset]

Brand new to AutoPkg and this awesome plugin. I can execute the .pkg recipes fine but get something similar to this when I attempt to run a .jss recipe.

xxx:~ john$ autopkg run -v Cyberduck.jss
Processing Cyberduck.jss...
SparkleUpdateInfoProvider
SparkleUpdateInfoProvider: Version retrieved from appcast: 14140
SparkleUpdateInfoProvider: Found URL https://update.cyberduck.io/Cyberduck-14140.tar.gz
URLDownloader
URLDownloader: Storing new Last-Modified header: Thu, 28 Nov 2013 13:56:59 GMT
URLDownloader: Storing new ETag header: 51a024f9b47d483d1d4683af7ea13e39
URLDownloader: Downloaded /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/downloads/Cyberduck-14140.tar.gz
EndOfCheckPhase
PkgRootCreator
PkgRootCreator: Created /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/Cyberduck
PkgRootCreator: Created /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/Cyberduck/Applications
Unarchiver
Unarchiver: Guessed archive format 'tar_gzip' from filename Cyberduck-14140.tar.gz
Unarchiver: Unarchived /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/downloads/Cyberduck-14140.tar.gz to /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/Cyberduck/Applications
Versioner
Versioner: Found version 4.4.3 in file /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/Cyberduck/Applications/Cyberduck.app/Contents/Info.plist
PkgCreator
PkgCreator: Connecting
PkgCreator: Disconnecting
Couldn't connect to autopkgserver: No such file or directory
Failed.
Receipt written to /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/receipts/Cyberduck-receipt-20140115-104042.plist

The following recipes failed: Cyberduck.jss Error in com.github.autopkg.jss.Cyberduck: Processor: PkgCreator: Error: Couldn't connect to autopkgserver: No such file or directory

The following new items were downloaded: /Users/john/Library/AutoPkg/Cache/com.github.autopkg.jss.Cyberduck/downloads/Cyberduck-14140.tar.gz

Thoughts?

[gregneagle]

Sounds like the autopkg server process isn't running. There were some issues with the 0.2.6 and 0.2.7 package releases that installed a launchd plist with the wrong permissions which would cause the issue you are seeing. Try installing the latest autopkg release here: https://github.com/autopkg/autopkg/releases/tag/v0.2.8

[johnnasset]

Perfect, the update seems to have cleared it up. This and my recent discovery of AutoDMG have made me very happy. Thanks!!