Slow pkg "Install on boot drive after imaging"

TomDay
Release Candidate Programs Tester

I have a pkg file that is set to "Install on boot drive after imaging". The pkg just sets the time zone for me. Imaging process with all other pkgs (not set to install on boot drive) after imaging work very fast, scripts as well, then when the pkg install in question kicks off (I get the "The Imaging process is finishing installing the software") and takes abut 10-15 minutes to complete. Its should be super quick, its on a 38 Kb file. Any thoughts?

34 REPLIES 34

cstout
Contributor III
Contributor III

I've experienced the same thing. Please keep me posted on any updates you find. In testing I've deployed a base image and one <1 MB file and it took about 5-10 minutes to finish the first boot install.

were_wulff
Valued Contributor II

@tommyday & @cstout

That is curious behavior; does it seem to be specific to that particular .pkg?

A couple things that might be useful for narrowing it down:

- See if it does it with any .pkg, regardless of size.

- See if it seems to only happen with .pkg files under a certain size.

- Does it seem specific to .pkg files, or is it happening with .dmg files as well?

- What are we using for the distribution point? File share (AFP or SMB?), JDS, HTTP? I seem to recall @cstout having an SMB sharepoint set up due to problems that were going on with AFP chewing up resources, is that still the case?

- Pretty sure I remember @cstout being on 9.32, what version are you on, @tommyday?

- Which OS is the JSS running on?

In some quick testing here, on 9.32, with an AFP file share on a 10.9.3 server I'm not seeing the behavior described, so narrowing environments down could at least let me tweak my test environment to be closer to what you two have set up.

Thanks!

Amanda Wulff
JAMF Software Support

TomDay
Release Candidate Programs Tester

@amanda.wulff I will test another pkg but in the mean time, answers to your other questions:

- DS points are afp
- We are running JSS v9.32
- JSS is on Mac OS 10.9.3

cstout
Contributor III
Contributor III

@amanda.wulff
JSS 9.32 on Windows Server 2012
Distribution share using SMB (no more JDS's being used)
-No AFP

I haven't dived into any logs on this particular issue since everything ends up working in the end, but I have noticed this when testing the installation of a first-boot install of an application.

TomDay
Release Candidate Programs Tester

@amanda.wulff Some more testing complete.

I tried numerous pkgs with the same long install time result, file sizes range between 38kb and 51mb. I also tried a dmg which is 571mb, same result of long install time. I timed the dmg and it took 22 minutes.

-Tom

were_wulff
Valued Contributor II

@tommyday ,

Thanks for testing that out!

I do have a couple more things to ask to narrow it down just a bit further (as I've still been unlucky in replicating it so far!):

If we go into Casper Admin and UNcheck the box that says "Install on boot drive after imaging", then go ahead and image as usual (and if that package not being included as part of a config isn't 'usual', do what we need to do to have it be included) does it seem to affect the behavior at all?

Ah, and I do know it's kind of a ridiculous thing to ask, but as it's something I've overlooked myself before: We are running Imaging 9.32 as well, correct?

I'll continue testing on this end to see if I can get the behavior to happen, but do let me know what you find out with the above; if we haven't come up with any good reason after that, I'll go ahead and grab your Technical Account Manager as we'll probably need to look into it a bit more than we can on the forums.

Thanks!
Amanda Wulff
JAMF Software Support

were_wulff
Valued Contributor II

Hey @tommyday

Just a quick update: I finally got it to misbehave for me.

Installed a base image + Firefox with "Install on boot drive after imaging" checked.

I've been staring at the "The imaging process is finishing installing software" screen for a good 15 minutes now; when it finally finishes, I'll give it a shot again with that box unticked.

Amanda Wulff
JAMF Software Support

TomDay
Release Candidate Programs Tester

Thx @amanda.wulff for the update, good to see it can be replicated. Confirming that I am running 9.32 Imaging. Any of the packages run as normal without the box unticked "Install on boot drive after imaging"

were_wulff
Valued Contributor II

@tommyday

Yeah, that's what I'm seeing as well; tried it with 9.31 and 9.32.

In some specific cases it may be "expected" due to D-006627 if we're using Thunderbolt Ethernet adapters to image as that defect revolves around the OS not immediately detecting that it's connected via Ethernet after reimaging when Thunderbolt adapters are in use and if any packages needed to have Internet or network connectivity to properly run/install it could cause a delay due to that.

However, I'm not imaging via Thunderbolt Ethernet, just by straight up Ethernet and I know the packages I'm installing don't require Internet connectivity to install (Text Wrangler, Gimp, and Firefox) so D-006627 is obviously not the root cause.

I'm going to take a look this morning and see if I can find any reference to this behavior being a known issue that I might have missed; if I don't find anything I'll open up a new defect on it and post back with the ID for reference.

For now, I'd say the workaround is just going to be not checking that box.

Thanks!

Amanda Wulff
JAMF Software Support

TomDay
Release Candidate Programs Tester

Interested to hear what you come up with. Problem is with this pkg is that if I dont have this option set, it doesn't install properly to the OS

were_wulff
Valued Contributor II

@tommyday

Had a quick chat with one of our PSEs, and I'm going to re-image mine again and kill the jamfhelper process this time so I can launch console to look at the jamf.log and system.log (and install.log for pkgs, as we call the installer.app for those) and see what on earth the system thinks it's doing during that long pause.
If that doesn't show anything, I'll see if ssh-ing into the machine shows anything more useful.

I'm also going to see if changing the priority from its default of 10 makes any difference at all; I can't think of a good reason that it would, but then, I can't think of a good reason why checking that box would cause such a massive 'pause' in the imaging process either.

We did dig up a very, very, very old (8.6 beta) defect (D-003011) that showed similar behavior, though that behavior was only with scripts set to run before and there were policies set to run after the script had run, and the workaround was to kill the jamfHelper process; I doubt that they're the same thing, but if the behavior is similar I'll be sure to reference that old defect for development's reference in a new one if a new one needs to be created.

Amanda Wulff
JAMF Software Support

were_wulff
Valued Contributor II

Hey @tommyday & @cstout

I went ahead and got a new defect opened up ( D-007139 ) with a link to this thread as well as details of testing.

It seems, when we have that box checked, to be just hanging until the jamfHelper process gets moving (after that 20 minutes or so) or we kill the jamfHelper process, at which point the package install kicks off almost immediately.

There are three ways I've found that get around it:

1) Uncheck the "Install on boot drive after imaging" box. Not viable for your particular situation, however.

2) Wait it out. Not viable at all, in my view; that's a LONG wait, especially if you're imaging dozens (or hundreds) of computers.

3) ssh into the affected machine(s), do a ps -A | grep jamfHelper, get the PID, and kill it with a sudo kill -9. Click back on the screen on the affected machine if the jamfHelper doesn't disappear on its own and it'll go away. Everything appears to wrap up at that point if it hasn't already.

All three of the above are listed in the workarounds for the new defect; if it would be possible for you to open up a case with your Technical Account Manager and reference this thread and that defect (D-007139) that'd be great for tracking and would let us attach the case to the defect (your TAM can get that done).

Wish I had a better answer instead of 'new defect', but at least we now know that the behavior is obviously not something that's intended!

Thanks!
Amanda Wulff
JAMF Software Support

TomDay
Release Candidate Programs Tester

@amanda.wulff Your support on this issue has been fantastic, thanks very much. I will do as you have requested with my TAM.

Best, Tom

cstout
Contributor III
Contributor III

I agree completely with @tommyday. @amanda.wulff, you've helped me out of sticky situations way too many times. As always, I greatly appreciate your help and effort to resolve problems as fast as possible.

were_wulff
Valued Contributor II

@tommyday & @cstout

Thanks guys! Glad to be of assistance!

Amanda Wulff
JAMF Software Support

TomDay
Release Candidate Programs Tester

Hope to meet you at #JNUC2014 if you'll be there @amanda.wulff

were_wulff
Valued Contributor II

@tommyday

Pretty sure I’ll be there! I got to go for two days last year, and I’m guessing I’ll get at least two this year as well.

smb_samba
New Contributor III

Hey guys,

Just wanted to chime in here. I've seen similar behavior where sometimes it looks like that dialog "The imaging process is finishing installing software" is just stuck. I have on occasion seen that doing a CMD + Q quits out of that dialog box and at least gets me to the desktop (assuming it's not actually doing anything). If I do interrupt something, a reboot will usually result in the dialog appearing again.

cstout
Contributor III
Contributor III

@smb_samba, thank you for mentioning that. I forgot about that particular issue. I have seen this as well in my imaging. Probably 1 out of every 10 will not quit the imaging splash screen and I'm unable to quit it which forces me to do a hard shut down.

were_wulff
Valued Contributor II

@smb_samba & @cstout

Are we seeing that when the "install to boot drive after imaging" box is checked, or does it not seem to matter?

If it's just when that box is checked, I'll update the defect to note that.

Amanda Wulff
JAMF Software Support

cstout
Contributor III
Contributor III

@amanda.wulff All of my various images require packages/applications to be installed to the boot drive after imaging. I don't believe I've come across that issue with no first-boot install apps selected.

smb_samba
New Contributor III

@amanda.wulff I'm in pretty much the same situation as @cstout . we have many packages that require a reboot to work so yes, the box is checked. I don't know that I've encountered super slow install times for something as simple as a few items requiring to be installed after imaging (because we have many that would understandably take a bit to install), but the "The imaging process is finishing installing software" has definitely been stuck for me before.

were_wulff
Valued Contributor II

@smb_samba & @cstout

Updated the defect to note that, and to note that if we kill the jamfHelper 'too soon' (or whatever the system thinks is too soon) we can end up in a situation where it reboots and gets stuck in the same loop again.

scottb
Honored Contributor

We've never used this option, but have been looking at it just recently. Has this issue just appeared in the 9.xx or 9.3x versions? We're still on 8.73, but the plan is still to go to 9.3x soon.

were_wulff
Valued Contributor II

@boettchs

In the testing I did, it was on 9.31 and 9.32; haven't tested the earlier versions of 9.x just yet.

cstout
Contributor III
Contributor III

@boettchs I didn't see this issue in previous revisions of 9, but I think I started noticing splash screen issues after 9.2. It could also be around the time Mavericks came out. I'm not convinced this whole thing isn't a conflict with new loginwindow handling in Mavericks. I also have another defect I'm working with JAMF on regarding the imaging splash screen not loading at all.

scottb
Honored Contributor

@amanda.wulff][/url: Thank you much. I will not upgrade from 8.73 to a lesser version of 9.xx so I guess we'll wait on trying out this feature. Who knows, by the time we get the JSS upgraded, it may be 10.33 ;)

@cstout: Thanks as well. Good info to have. I try to balance between paranoia over upgrading and just dealing with it...

were_wulff
Valued Contributor II

@cstout

That's also a possibility; I know we have a few open issues revolving around Mavericks not playing nice with parts of the JSS, though nothing was specific to the issue that started this particular thread.

We did have a few loginwindow defects that were specific to Mavericks (those are listed as fixed as of 9.31), and we do still have an open defect in which login/logout hooks don't always work (D-006636) that is specific to Mavericks, even more specific to the 10.9.3 version.

Looks like the majority were taken care of by 9.31, but the ones that are open appear to be issues between the OS and the JSS with development looking into how to get our stuff working nicely with Mavericks while not breaking it for earlier versions of the OS that we still support.

That does make me curious though: For this particular issue, are the computers being imaged all Mavericks computers, or do we have a mix of Mavericks and older versions of the OS that have the same problem?

If it's just Mavericks, that'll narrow things down a bit more. If it's a mix, that should be noted.
I can certainly test it here, but it's also helpful to know what's being seen in live environments as well.

Thanks!
Amanda Wulff
JAMF Software Support

Sandy
Valued Contributor II

HI!
Just thought I'd chime in here:
I am booting to a 10.7.5 netboot image, using Casper Imaging 9.3.2 and JSS 9.3.2
Laying down a 10.9.2 image, and almost ALL my packages are installing after reboot using the checkbox: "install to boot drive after imaging", including the 10.9.3 update. I do this routinely with anything in .pkg format.
I am imaging Airs using TB Ethernet adapters.
I have it set this way in part to get past the net booted portion, since that is my imaging bottle neck.
I am not seeing this issue. My JAMF Helper screen comes up, and once everything completes goes away, temp profile is deleted, machine reboots, and I can tell everything is complete when the WIFI policy activates the local SSID
The only Configuration Profile I'm running is a login window profile, which is in effect at reboot, and all policy history is flushing as expected and policies are running on their triggers.
Thought this may be helpful in determining where the issue lies.
Sandy

cstout
Contributor III
Contributor III

@amanda.wulff Nearly all of the issues that I am working with JAMF on are defects that have been previously closed. We're all pulling our hair out over it because I keep reporting issues that were resolved in previous versions. I'm running the latest JSS and suite apps (9.32). My various configurations are all running Mavericks.

bentoms
Release Candidate Programs Tester

I used to see this back in the day when having multiple installers in a config that had architecture limits.

Not sure if it still applies, but does the pkg taking time have itself set to only install if a certain OS or Processor?

were_wulff
Valued Contributor II

@Sandy

Thanks for the additional input!

I just wrapped up testing with my 10.8.4 image, and didn’t see the behavior there either, so it’s possible that this may be specific to 10.9.3.

One of my co-workers tested with 10.8.5 and it seemed to work as expected in that version of the OS as well.

If nothing else, it certainly helps narrow things down for development and QA when it gets into their hands.

The defect has been updated to reflect new info from this thread, and the thread is linked in the defect notes itself so it’ll be easy for anyone to find and look through if need be.

Amanda Wulff
JAMF Software Support

were_wulff
Valued Contributor II

@tommyday @cstout @smb_samba

So, I’ve got some updates from dev on this one!

The long, detailed version is that it’s expected behavior as of 9.31.

No, wait, don’t stop reading! Explanation of what’s going on on the backend follows:

In 9.31 we changed the way the post install items/scripts run to take care of all the old Device Signature Error related defects from 9.2x and 9.30; in the cases of those defects, it was a timing issue that was causing them, in many cases the enrollment not being finished before a machine rebooted, the enrollment process trying to kick off at the same time as something else and butting heads with it, or the enrollment process not getting a chance to kick off before things rebooted.

We do know, if Thunderbolt Ethernet is involved in imaging, we can see a 5-10 minute delay before the OS realizes it has network connectivity (D-006627), which is why I’d thought in a previous comment there might be some relation there.

The way the enrollment portion of the post install works in 9.31 and 9.32: It polls 6 times every 5 minutes to look for a connection to the JSS connect. If it finds it, great, it continues on. If it doesn’t find it, it goes idle for 4 1/2 minutes, then tries again, and repeats until it gets a valid response from running the jamf checkJSSconnection command.

So, in 9.31 they switched it so nothing else can start in the post install until enrollment is complete; in my test machine’s case, we found that it was taking 2 1/2 minutes to set the home page (specified in the configuration I was using) and another 3 minutes for the enrollment to actually finish, so the end result is that it ended up being between 6-8 minutes total (depending on when it polled again for a JSS connection) before the entire enrollment process finished and it continued on on its own.

During that time, I just got to sit and look at the splash screen that the jamfHelper puts up; we sat here, ssh’d into the test machine, and just ran tail -f on /var/log/jamf.log every couple of seconds to see what it was doing and to see if we could find out what seemed to be ‘hanging’ it.

We also checked to see if the enrollment portion was still going with ps -auxx | grep jamf. In my case here, that was a running process for just over three minutes (which seems a bit long, but I’m more inclined to think that’s just my test environment needing a good cleaning up).

The above two things are things we can easily check in other environments to see if yours is seeming to stall on the enrollment or if there’s something else there that’s causing it.
Once a computer gets to the splash screen, ssh in and monitor the jamf processes with:

ps -auxx | grep jamf

If it’s sticking on enrollment taking a long time we’ll see the enroll command with an invitation attached; hard process to miss since it’s a long one.

And, every few seconds, run a tail -f /var/log/jamf.log to see if anything there has changed or looks odd/off.

I did have a bit of a chat about how, while I understand why the change was made, we do need to take a look at it again to see if there is any way we can make it more efficient to cut down on the wait time (or to speed up the enroll time) or, at the very least, modify the splash screen that shows up to indicate what part of the process is going on in the background instead of just the, “Please wait…” style splash screen we have right now.

If you already have a case open with your TAM, I’d recommend going through the above and letting your TAM know what you found as we are still trying to determine the exact root cause of the behavior and/or to verify if it is the result of the fixes for the Device Signature Error defect or if it’s something else.
We’re still looking into that, primarily because we’re only seeing the problem show up in 10.9.3.

If you don’t have a case open with your TAM already and are experiencing this issue, please do consider opening one up; feel free to reference this thread, the defect, etc…and attach either the jamf.log or screenshots of the above to the case as well.

Having information from live customer environments can help dev & QA (not to mention support!) out with testing.

Thanks so much for your patience and assistance while we get the cause of this behavior pinpointed!

Amanda Wulff
JAMF Software Support

michael_lewis
New Contributor

Amanda,
I can confirm exactly what you said in your last post. I imaged 265+ MacBook Airs in December last year running JSS 9.22 (I believe) and did not experience any issues with the Post Install packages. Now on 9.32 I am experiencing wait times which are not acceptable...miniumum of 8 minutes. I say this because when you are handing out a new laptop to a student and you are walking them through it 8 minutes feels like an eternity. When I dug into the jamf.log in /var I noticed what you were seeing regarding it taking a couple minutes to set homepage or management framework. The actual install packages didn't take that long it was the jss processes slowing it down at the beginning. We use Thunderbolt to Ethernet adapters and they worked great in 9.22....not sure if it was an improvement to make the process change described above in 9.32 as you could get around the enrollment issues by making an enrollment package including the QuickAdd and changing its priority to be last? That is at least how we did it. It is definitely making life harder here!