Build up of pending management commands

[CasperSally]

Does anyone else notice when poking around their machines management history a build up of pending management commands (CertificateList & ProfileList over and over).

We're seeing this for our shared machines mostly, the ones in carts or labs. Like 10 to dozens of pending commands per machine, but on same machines there are successful management commands same day as well, it's not like they're all stuck pending.

JAMF's answer has been to run a sql command to clear them out, but I'm curious as to why it's happening, and really not sure running a sql command every X days is something that's normal or to be expected.

Appreciate any thoughts.

[rcorbin]

Yes I have seen this. It's almost like the JSS loosees it's ability to push out profiles etc to a machine. Sometimes I've found that if I recon the machine again it then starts to work ok after that. I haven't really looked at it much beyond that. I think the re-reconing operation renews all the certificates on the client machine and then the communication starts back up.

[CasperSally]

In my case, machine is still communicating fine. I'll have 10-40 pending cert/profile commands in management history , and then some successful ones mixed in with the pending commands going by time stamps.

Jamf says the pending ones queue up and should be manually cleared...

[rcorbin]

And the solution is to run sql commands ? Does one command clear the pending ones on all machines ? Or do you use separate commands for each machine ? If that's the case what if you have hundreds or thousands of machines with this happening ? Is this Casper 9.32 or higher ?

[CasperSally]

9.32, the command clears out all pending commands in the system. Just don't really love the idea of running it every X days.

[npynenberg]

Anyone care to share the command... I'm seeing some of the same build up that I would like to clear. (Yeah.. I know.. backup first!)

[denmoff]

I'm seeing this as well. In most cases, configuration profiles are not being set. I also notice on login, these machines are hanging for several minutes, with the 'Updating managed settings..." popup window.

[tadholyfamily]

I see similar behavior with iPads. Sometimes the list of pending gets to 200 or so repeats of Install App List, CertificateList, or ProfileList. When it happens to an iPad, other commands are unable to process.

[denmoff]

I've had some luck running the following two commands to get the 'pending commands' to go thru. But it involves using ARD which isn't ideal. It just clears the profiles that are installed and reloads them. This seems to open the gate back up for those stuck config profiles. It also fixes the issue with long login times.

sudo jamf removeMdmProfile
sudo jamf mdm

[CasperSally]

For me the commands are pending/building it up it seems because I wipe local student home directories on logout. All of the pending commands are tied to student user names (that no longer exist on the machine).

Still working with JAMF on it.

[dlondon]

Hi Everyone,

We have been seeing this too. James here would love to blame it on our lack of Reverse DNS entries for machines. Anyone getting this who has Reverse DNS?

Regards,

David

[Bobst]

I'm seeing the same issue as everyone else. What SQL commands are being run to elevate this issue? Has anyone else seen the issue with RAM being fully utilized by these pending commands? 10.9.5 OSX, Mac Mini 2012, i5, 4Gb of RAM. Run jamf removeFramework, computer was removed from Casper and RAM utilization dropped. Any help would be appreciated.

[Kaltsas]

@denmoff thanks for the suggestion to use those mdm commands. I've been seeing these crop up on quite a few machines after I updated to 9.51, that fixed it up right as rain on the afflicted machines.

[Bobst]

We are running 9.52 still seeing same issues, before we were on 9.32 and noticed it there as well.

[Kaltsas]

It's entirely possible it was prevalent prior to 9.51 and I hadn't been paying attention. I made a configuration profile change along with our update to 9.51 and that was when I noticed machines sitting in a pending state with these commands.

[jamesdurler]

We are seeing the same thing where numerous pending commands are building up. We also delete accounts on logout but we do this by a scripted profile.

[cmsweet]

I've noticed it as well. We're running 9.32, also running Deep Freeze 5.8 on the machines. Profiles are removed when machines are rebooted.

[libertyuniversi]

I am seeing this as well along with complaints that the computers are freezing on the login screen.

[AlanSmith]

Have been experiencing similar issues! Have updated JSS to 9.61 and running Mac OS X v10.9.5

I haven't been using Configuration Profiles as yet, but in preparing an image for 2015, was wanting to utilise them more. So have a small test group (2 machines) that I've been using for the Config Profiles. I had both machines with the profiles ok, then I re-imaged them and they no longer would get their config profile, with the status listed as 'Pending'.
Re-enrolling them via a QuickAdd package didn't fix the issue, however running the commands listed by @denmoff did fix the issue!

Doesn't give me a lot of confidence to use these though!!

[rcorbin]

As I said I have seen this before, but I'm now wondering if the cause was bad or invalid MDM profiles on the client end. Recently I determined that we had a number of machines had invalid or non working MDM profiles. I think the cause was an OS image that was somehow enrolled in the JSS at the time it was converted to an OS Image file via Composer.

What that caused was MDM Profiles on the machines that had SCEP Enrollment Requests that looked rather odd. In the Certificate Field it just said ?Invalid_Keychain_Item? And it didn't have an expiry date. So they really didn't have proper certificates etc.

We did try to just Recon the machines again, but that didn't seem to fix it. So I removed the profile all together and then Reconed the machine again. That fixed this bad MDM profile.

Apparently this is caused the binary that exists on the OS Image ends up conflicting with the one the imaging process is trying to lay down, and we end up with a bad MDM Profile.

If any of you have this issue let me know as I came up with one way to fix it and the folks at Jamf came up with another as well. It's not hard to fix.

Anyway I think this is why we were seeing some of these management commands build up.

[npynenberg]

Interesting rcorbin. I'd love to hear what you did... I wonder if we have the same issue.

[Kallendal]

Recon commands aside, anyone know if this has been resolved with the latest release? I have been seeing several machines on our network experiencing this issues (even freshly imaged ones).

Should we be setting up a Monthly (or possibly weekend) check in item, something like this:

sudo jamf manage -verbose
sudo jamf recon

fi

To help MDM along?

Due to the nature of our environment, using sudo jamf removeMdmProfile can't be done. Not with out causing other issues.

[rcorbin]

We found in some cases were not not able to push profiles to machines because of an invalid MDM profile on the client end. Recently I determined that we had a number of machines had invalid or non working MDM profiles. I think the cause was an OS image that was somehow enrolled in the JSS at the time it was converted to an OS Image file via Composer. That caused an issue where MDM Profiles on the machines that would show SCEP Enrollment Requests that looked rather odd. In the Certificate Field it just said "?Invalid_Keychain_Item?" And it didn't have an expiry date. So they really didn't have proper certificates etc.

To solve that we had a rather simple script that looked like this.

sudo jamf removeMdmProfile
sudo jamf manage

This simply removes all profiles from the other end and re-enrolls the machine back in and it grabs a fresh profile. As we were warned by Jamf support I would do a bit of testing with this first as there is a danger that if something doesn't go correctly you could end up un-enrolling the machine and loose contact with it. But in our testing we didn't have any issues. What I can say is that if you do have a bad or invalid MDM profile simply running recon will not always work. You really need to run the above commands.

Take a real close look at the MDM profiles on the client side. Do they have the "?Invalid_Keychain_Item?" in the Certificate Field. Do they have what looks like a valid expiry date. If that look fine your MDM profile is probably fine.

Other than that I would think that its maybe an APN issue. Most of all of those MDM management commands are all pushed out via MDM.

I have seen a few apps out there that will test to make sure APN's are functioning within your network.

https://itunes.apple.com/ca/app/apn-tester-free/id626590577?mt=12
https://github.com/Zambiorix/Cocoa-APNS-Test
https://developer.apple.com/library/ios/technotes/tn2265/_index.html

[pereljon]

The problem I'm seeing with running

sudo jamf removeMdmProfile
sudo jamf manage

is that when it is run on a computer on a Wi-Fi network which has been pushed out using a Configuration Profile, the Wi-Fi profile is removed and the computer looses its network connectivity. At this point the jamf manage step can't run to apply the management framework and Configuration Profiles.

Has anyone run into this issue? One solution is to use networksetup -setairportnetwork to re-establish the network (but this works only on networks with a single fixed password). Otherwise you can bring up a dialog instructing the user to select the network from the Airport menu.

JP

[bpavlov]

Do your users have access to Ethernet at all? If so, the only thing I can think of is having a check in place so that if Ethernet is active then proceed, if it's not then do not proceed. I'm just laying out the logic. This may be a bit easier too if you have network segments setup in Casper and your network is setup in such a way that wired and wireless are on different IP ranges. Just some things to think about.

[pereljon]

Most users are wireless only. Hence the conundrum.

[pereljon]

Here's my solution for cleaning up MDM profiles for a client who is on a wireless connection which is installed by an MDM profile. As the wireless connection will be lost when the MDM profile is removed, the script checks for which wireless network it is on and displays a jamfHelper dialog asking the user to reconnect to the wireless network before proceeding to run jamf manage and jamf recon.

#!/bin/bash
JAMFHELPER="/Library/Application Support/JAMF/bin/jamfHelper.app/Contents/MacOS/jamfHelper"
# Constants
mdmAirport="AirportNetworkName"
jhWindowType="utility"
jhTitle="System Management"
jhDescription="Please select the $mdmAirport network from the AirPort menu in the menu bar."
# The icon to show in the jamfHelper dialog box
jhIcon="/Library/Application Support/JAMF/icon.png"
# Name of the AirPort network which will be removed and added by removing and adding MDM profiles
wifiDevice=$(networksetup -listallhardwareports | grep -A 1 Wi-Fi 2> /dev/null | grep Device 2> /dev/null | awk '{print $2}' 2> /dev/null)
if [[ -z "$wifiDevice" ]]; then 
    echo "No Wi-Fi device found"
else
    echo "Wi-Fi found on device: $wifiDevice"
    wifiNetwork=$(networksetup -getairportnetwork "$wifiDevice" | sed 's/Current Wi-Fi Network: //')
    if [[ -n "$wifiNetwork" ]]; then
        echo "Wi-Fi network: $wifiNetwork"
    else
        echo "No Wi-Fi network on device $wifiDevice"
    fi
fi
echo "Removing MDM Profile"
/usr/sbin/jamf removeMdmProfile -verbose
echo "Sleeping"
sleep 5
if [[ "$wifiNetwork" = "$mdmAirport" ]]; then 
    echo "Displaying dialog"
    killall jamfHelper
    "$JAMFHELPER" -windowType "$jhWindowType" -icon "$jhIcon" -title "$jhTitle" -description "$jhDescription" 2> /dev/null &
    while true; do
        echo "Checking connection"
        jssConnection=$(/usr/sbin/jamf checkJSSConnection > /dev/null; echo $?)
        if [[ "$jssConnection" -eq 0 ]]; then
            echo "Connected to JSS"
            killall jamfHelper 
            break;
        fi
        echo "Connection timed out"
    done
fi
echo "JAMF manage"
/usr/sbin/jamf manage -verbose
/usr/sbin/jamf recon
exit 0

[skeb1ns]

bump

Experiencing this behavior with a lot of machines since a few months. Didn't noticed it at first but a few machines didn't received the config profiles that I've build so after some digging I stumbled upon this:

5 out of the 15 machines that I've checked had a few commands pending so I think that roughly 1 third of my clients has this issue.

I would like to try the sql commands to clear out the "stuck" entries. Which commands are used to do this? Running Casper 9.81 btw.

Thanks!

[skeb1ns]

Anyone?

[AdamH]

I'v seen this behavior as well.
I'd love some sort of solution to this as I feel a bit stuck with my hands tied when this stuff happens.

[jescala]

I'm having issues with this as well, seems to have started a few days ago. It is happening with our iOS and OS X clients. The management commands are stuck in a "pending" state but occasionally move to the "failed" with the following error:

There was a problem communicating with a push server

After a few seconds, it cycles back to "pending."

[bwiessner]

@jescala

Are you using JAMF Cloud or on Prem?

[skeb1ns]

After examining an affected system I came to the conclusion that this occurred in my environment due to the fact that the user deleted his System.keychain which holds the certs that Casper uses to do mdm related stuff.

Re-enrolling fixed this! Ugh, users with local admin rights sometimes give me a headache...

[jescala]

@bwiessner We have an on prem cluster.

[holbertonschool]

We have the same issue here, we are using Jamf Cloud, Jamf client 9.82.

[holbertonschool]

The issue is still going on, commands are getting pushed but the client is not executing them. Is there anything that could help me to debug it on the client? I checked /var/log/jamf.log but nothing in there.

[rcantrell]

Casper 9.82

I'm too am having similar issues pushing out specific profiles to random machines in our fleet at this time. Using the command @denmoff proposed has worked for us. Would someone mind sharing the SQL commands used to purge the database of pending commands?

Thanks,
Roger

[georgecm12]

I'm beating my head against this problem as well. As with CasperSally above, the problem may be exacerbated by the fact that we also delete home directories.

I'm trying to use device assignments to push an AppStore app out to all of my lab machines, and so far, of 103 targeted machines, only 11 have actually installed the app. Checking a random sampling, all of the computers have a fairly lengthy list of stacked up MDM commands, and on most of them, the install app command is mixed in.

More frustratingly, the "Cancel all pending and failed management commands" action doesn't seem to do anything. I've run that, then gone in and the list is untouched.

I've just updated us to 9.96, and we're running 10.11.6 on our clients.

[allanp81]

I'm totally fed up with this as well. We have about 600 macs and as others have said, we delete any local student accounts on a restart.

We have constant build up of pending and failed commands I can't keep going through and manually clearing them all of the time!

I'm thinking about creating a policy that will run once a day, outside of the hours between 9am and 6pm which just removes the mdm profiles and reapplies them. I've asked Jamf if there's any danger in doing this.

[donmontalvo]

Hopefully no 802.1x profiles are required else computers won't connect again to get profiles reapplied.