JSS 9.31: "Flush history on re-enroll"...possible bug

[donmontalvo]

We recently upgraded from 8.73 > 9.31 and one of the things we noticed is our once-per-computer policies were running again on the computers as they cut over to the new JSS. We suspected our policy logs were being dumped on re-enroll.

Thanks to our JAMF sleuths (Tim H and Dan K), we ran some tests and it looks like we may have found a bug.

If we pull a JSS Summary we see:

Flush history on re-enroll fase

However, MySQL shows the opposite:

$ /usr/local/mysql/bin/mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 1264
Server version: 5.5.36-log MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights
reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input
statement.

mysql> use jamfsoftware;
Database changed
mysql> select flush_management_history_on_remanage from
mac_os_x_enrollment;
+--------------------------------------+
| flush_management_history_on_remanage |
+--------------------------------------+
|                                    1 |
+--------------------------------------+
1 row in set (0.01 sec)

mysql>

So if you're going to migrate, make sure you check MySQL to make sure you won't lose your logs.

HTH,
Don

[were_wulff · Accepted solution]

@donmontalvo

I worked with Tim and Dan a bit on this this morning, and we were able to replicate it here; Tim opened up a defect for the behavior (D-007057).

Figured you'd want that for tracking purposes.

Amanda Wulff
JAMF Software Support

[were_wulff · Accepted solution]

@donmontalvo

I've been testing this when I have little bits of spare time today and have found the following odd bits of info about it:

So far, I'm back to 9.24 in my testing and am seeing that behavior in situations where the database flag is set to 1, but is reporting false in the summary; the database I'm using was an upgrade from 8.73 in which I verified that flush_management_history_on_remanage was set to 0 because I went in and purposely set it to 0.

We did see it flip to 1 on the upgrade from 8 and the history gets flushed if we run a QuickAdd.pkg or a re-enroll with Recon, which is the behavior you were seeing.

The odd bit is that when that option gets set to 1 in the database (but displaying false in the summary) during the upgrade from 8 to 9, it looks as though QuickAdd packages (OTA or Recon created) or re-enrollment via Recon (network scanner or remote) are trying to bless the system, which probably shouldn't be happening since we're trying to re-manage, not re-image.
With the option set to 0 in the database, of course, that doesn't happen.
If you have a chance sometime, could you grab a jamf.log from one of the clients this happened to in your environment and see if there's a line in there about it blessing the system? If we're seeing that on your end too, I'd like to add a note on that to the defect.

Running a sudo jamf enroll -prompt, however, re-enrolls as expected, based on the setting that was present in the 8.x database, and doesn't actually kick off a re-run of the policies. I'm not quite sure why that one is different but, then, that's probably why I'm in support and not development. :)

Whatever is happening is certainly not expected behavior.

Thanks!
Amanda Wulff
JAMF Software Support

[were_wulff · Accepted solution]

@boettchs

You should be okay to upgrade to 9.32; all we'll need to do to make sure you don't run into the issue is either run through the steps @donmontalvo detailed in his posts in this thread, or get in touch with your Technical Account Manager (and reference either this thread or the defect ID) and have them walk you through checking the setting and making sure it's set the way we want it to be set.

We have the steps @donmontalvo posted in the defect in the workaround section, so it shouldn't be a problem for your Technical Account Manager to find and walk you through if you'd be more comfortable doing it that way.

As long as you don't re-enroll anyone before checking and, if necessary, fixing the setting, it shouldn't be a problem.

However, we always recommend upgrading in a test/dev environment first in any situation, but doubly so when doing big jumps like 8.73 to 9.32, just to be on the safe side, since doing that will let you head off any potential problems in a test environment rather than having to wrestle with them immediately in production.

As for release dates, they're pretty tight lipped about that; as of now, we don't have a time frame on 9.33.

Amanda Wulff
JAMF Software Support

[were_wulff · Accepted solution]

@jhbush1973

That is actually a Feature Request that is currently under review.

If you haven't yet added your vote/comment to it, it's certainly helpful to get more input.

Amanda Wulff
JAMF Software Support

[were_wulff · Accepted solution]

@Kumarasinghe & @CasperSally

We’re tentatively thinking it may be related since the same table is involved. Since the other defect (D-007057) is so new, however, we haven’t had much of a chance to really dig into it on a code level yet, so I can’t say for certain whether or not they’re related.

On reimage, it’s expected behavior that management history flushes as, in most cases, that’s the desired behavior.

Where it gets confusing is that, up until 8.31, it did automatically flush policy history on reimage.

In 8.31, there was a defect that stopped it from automatically flushing, so we had the workaround of adding a jamf flushpolicyHistory to either a policy or to a script run during imaging to cause the flushing to happen.

That defect was fixed in 8.4, and most people just left the workaround in place even though it wasn’t strictly necessary anymore and should have just started flushing automatically again as it was intended to do.

For customers that did not want to flush on reimage, we usually just went in to MySQL and set the option of flush_management_history_on_remanage to 0 and they could choose to flush manually for specific devices if necessary.

In 9, it was also set to flush policy history after a reimage because, as I mentioned, in most cases that is the desired behavior; a computer has been reimaged (as opposed to just re-enrolled), so we want it to re-run the policies it needs to to get the correct profiles, packages, etc…in place again.

Then, in 9.24, we had another defect, which is still open (D-006509) that caused the automatic policy history flushing on reimage to stop again.

The workaround is the same as it was for the old 8.3 defect: Add a policy to run jamf flushPolicyHistory, usually at the ‘enrollment complete’ trigger, but any trigger will eventually get the job done.

We can also manually change mac_os_x_enrollment.flush_management_history_on_remanage to 1 to cause it to automatically flush logs on reimage again, however, in light of this newest defect (D-007057) that may cause other problems as, currently, it will flush policy history on re-enrollment via OTA, QuickAdd.pkg, or Recon Remote, when it is not supposed to.

At the moment, given the two defects and how they appear to interact, the safest option may very well be to set flush_management_history_on_remanage to zero and, if we need policy history to flush on reimage, set up a policy to run on the Enrollment Complete trigger to run a jamf flushpolicyHistory.

Hopefully, that wasn’t too difficult to follow, I know it’s a bit of a winding road to this point with that particular set of behaviors. If you do have additional questions on it, I’d recommend getting in touch with your Technical Account Manager. They’ll likely have a faster response for you.

Thanks!

Amanda Wulff
JAMF Software Support

[were_wulff · Accepted solution]

@CasperSally

Hm, that is interesting behavior, especially since it's set to 1.

I haven't had a chance to look up the case yet (on a pause on a webex, waiting to be taken off of hold :) ), but I'd be curious as to what the jamf.log and system.log files from one of the affected computers have in them; sometimes we find weird little clues buried in there.

If it's not already in place as a workaround while troubleshooting, it may be worth testing the workaround of tossing in a policy to run at enrollment complete (sometimes I put it at that trigger, startup, login, or logout at once per computer just to make sure it runs, but that's probably overkill on my part) that just runs a jamf flushpolicyHistory.
Doesn't fix the underlying problem, of course, but it can at least get the history flushed in the meantime.

I'm also curious, in light of the newer defect, if you run a QuickAdd on the package after imaging if it flushes the logs; I did find that just a sudo jamf enroll -prompt doesn't cause the flush to happen, so that wouldn't do it, but maybe a QuickAdd.pkg from Recon might?

That's not a totally good workaround, at least in my view, though as it would cause the policy history to flush on computers we may just be intending to re-enroll independent of a reimage.

Amanda Wulff
JAMF Software Support

[donmontalvo]

JAMF opened D-006509 for this, apparently another client was reporting the opposite (wanting to purge logs on re-enroll, but not happening).

[donmontalvo]

Test result...

On our test JSS we changed the MySQL value to "0" (do not flush logs on re-enroll), then we bounced the box:

$ /usr/local/mysql/bin/mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 1264
Server version: 5.5.36-log MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights
reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input
statement.

mysql> use jamfsoftware;
Database changed
mysql> update mac_os_x_enrollment set flush_management_history_on_remanage=0;
mysql> exit

When it came back up we checked JSS Summary to see if it changed, still shows "false" which was wrong but is now right since we changed the value (IOW it may not be properly reading/displaying the value).

We took a Mac that has policy history, installed the QuickAdd to cut it over to the test JSS...all logs are still there.

So changing the MySQL value to "0" is a must for now, unless you want to lose your logs when upgrading to 9.31. Just know that JSS Summary won't accurately reflect this value.

HTH,
Don

[were_wulff · Accepted solution]

@donmontalvo

I worked with Tim and Dan a bit on this this morning, and we were able to replicate it here; Tim opened up a defect for the behavior (D-007057).

Figured you'd want that for tracking purposes.

Amanda Wulff
JAMF Software Support

[donmontalvo]

@amanda.wulff You rock! :D

[were_wulff · Accepted solution]

@donmontalvo

I've been testing this when I have little bits of spare time today and have found the following odd bits of info about it:

So far, I'm back to 9.24 in my testing and am seeing that behavior in situations where the database flag is set to 1, but is reporting false in the summary; the database I'm using was an upgrade from 8.73 in which I verified that flush_management_history_on_remanage was set to 0 because I went in and purposely set it to 0.

We did see it flip to 1 on the upgrade from 8 and the history gets flushed if we run a QuickAdd.pkg or a re-enroll with Recon, which is the behavior you were seeing.

The odd bit is that when that option gets set to 1 in the database (but displaying false in the summary) during the upgrade from 8 to 9, it looks as though QuickAdd packages (OTA or Recon created) or re-enrollment via Recon (network scanner or remote) are trying to bless the system, which probably shouldn't be happening since we're trying to re-manage, not re-image.
With the option set to 0 in the database, of course, that doesn't happen.
If you have a chance sometime, could you grab a jamf.log from one of the clients this happened to in your environment and see if there's a line in there about it blessing the system? If we're seeing that on your end too, I'd like to add a note on that to the defect.

Running a sudo jamf enroll -prompt, however, re-enrolls as expected, based on the setting that was present in the 8.x database, and doesn't actually kick off a re-run of the policies. I'm not quite sure why that one is different but, then, that's probably why I'm in support and not development. :)

Whatever is happening is certainly not expected behavior.

Thanks!
Amanda Wulff
JAMF Software Support

[scottb]

This begs the question - with a planned upgrade form 8.73 to 9.32, would it be better to hold off on this thinking that 1) It will be fixed on 9.33 and 2) 9.33 would come relatively soon?

[were_wulff · Accepted solution]

@boettchs

You should be okay to upgrade to 9.32; all we'll need to do to make sure you don't run into the issue is either run through the steps @donmontalvo detailed in his posts in this thread, or get in touch with your Technical Account Manager (and reference either this thread or the defect ID) and have them walk you through checking the setting and making sure it's set the way we want it to be set.

We have the steps @donmontalvo posted in the defect in the workaround section, so it shouldn't be a problem for your Technical Account Manager to find and walk you through if you'd be more comfortable doing it that way.

As long as you don't re-enroll anyone before checking and, if necessary, fixing the setting, it shouldn't be a problem.

However, we always recommend upgrading in a test/dev environment first in any situation, but doubly so when doing big jumps like 8.73 to 9.32, just to be on the safe side, since doing that will let you head off any potential problems in a test environment rather than having to wrestle with them immediately in production.

As for release dates, they're pretty tight lipped about that; as of now, we don't have a time frame on 9.33.

Amanda Wulff
JAMF Software Support

[jhbush]

@amanda.wulff this issue was discovered in another recent build. Is there any chance JAMF will create some kind of customer only bug system where we can check if what we are seeing is a bug before beating our head against the wall and trolling forums for information.

[were_wulff · Accepted solution]

@jhbush1973

That is actually a Feature Request that is currently under review.

If you haven't yet added your vote/comment to it, it's certainly helpful to get more input.

Amanda Wulff
JAMF Software Support

[Kumarasinghe]

@amanda.wulff Is this the same database flag for existing machines to flush policy history at re-image?

[CasperSally]

thanks for posting this @donmontalvo. I have been testing our upgrade and this explains why my policy logs have been flushing on reimage even though our TAM said they shouldn't be. I want ours to flush, so I guess I just have to be careful to test each upgrade in test environment to see when the db flag matches what JSS summary is reporting.

[were_wulff · Accepted solution]

@Kumarasinghe & @CasperSally

We’re tentatively thinking it may be related since the same table is involved. Since the other defect (D-007057) is so new, however, we haven’t had much of a chance to really dig into it on a code level yet, so I can’t say for certain whether or not they’re related.

On reimage, it’s expected behavior that management history flushes as, in most cases, that’s the desired behavior.

Where it gets confusing is that, up until 8.31, it did automatically flush policy history on reimage.

In 8.31, there was a defect that stopped it from automatically flushing, so we had the workaround of adding a jamf flushpolicyHistory to either a policy or to a script run during imaging to cause the flushing to happen.

That defect was fixed in 8.4, and most people just left the workaround in place even though it wasn’t strictly necessary anymore and should have just started flushing automatically again as it was intended to do.

For customers that did not want to flush on reimage, we usually just went in to MySQL and set the option of flush_management_history_on_remanage to 0 and they could choose to flush manually for specific devices if necessary.

In 9, it was also set to flush policy history after a reimage because, as I mentioned, in most cases that is the desired behavior; a computer has been reimaged (as opposed to just re-enrolled), so we want it to re-run the policies it needs to to get the correct profiles, packages, etc…in place again.

Then, in 9.24, we had another defect, which is still open (D-006509) that caused the automatic policy history flushing on reimage to stop again.

The workaround is the same as it was for the old 8.3 defect: Add a policy to run jamf flushPolicyHistory, usually at the ‘enrollment complete’ trigger, but any trigger will eventually get the job done.

We can also manually change mac_os_x_enrollment.flush_management_history_on_remanage to 1 to cause it to automatically flush logs on reimage again, however, in light of this newest defect (D-007057) that may cause other problems as, currently, it will flush policy history on re-enrollment via OTA, QuickAdd.pkg, or Recon Remote, when it is not supposed to.

At the moment, given the two defects and how they appear to interact, the safest option may very well be to set flush_management_history_on_remanage to zero and, if we need policy history to flush on reimage, set up a policy to run on the Enrollment Complete trigger to run a jamf flushpolicyHistory.

Hopefully, that wasn’t too difficult to follow, I know it’s a bit of a winding road to this point with that particular set of behaviors. If you do have additional questions on it, I’d recommend getting in touch with your Technical Account Manager. They’ll likely have a faster response for you.

Thanks!

Amanda Wulff
JAMF Software Support

[CasperSally]

@amanda.wulff - great post, thanks for detailing all of that as it's been hard for me to wrap my brain around.

We're working with the support team because even with our db set to 1, our management history isn't flushing and maybe the root of my reimage enrollment issues.

Just also wanted to also get it out there my TAM says if you flush policy logs, VNC logs go with them. Probably not a huge deal for all environments, but it is here. We'll probably be forced to go to rethink how we push items that are set to once per computer & start utilizing the enrollment complete trigger.

[were_wulff · Accepted solution]

@CasperSally

Hm, that is interesting behavior, especially since it's set to 1.

I haven't had a chance to look up the case yet (on a pause on a webex, waiting to be taken off of hold :) ), but I'd be curious as to what the jamf.log and system.log files from one of the affected computers have in them; sometimes we find weird little clues buried in there.

If it's not already in place as a workaround while troubleshooting, it may be worth testing the workaround of tossing in a policy to run at enrollment complete (sometimes I put it at that trigger, startup, login, or logout at once per computer just to make sure it runs, but that's probably overkill on my part) that just runs a jamf flushpolicyHistory.
Doesn't fix the underlying problem, of course, but it can at least get the history flushed in the meantime.

I'm also curious, in light of the newer defect, if you run a QuickAdd on the package after imaging if it flushes the logs; I did find that just a sudo jamf enroll -prompt doesn't cause the flush to happen, so that wouldn't do it, but maybe a QuickAdd.pkg from Recon might?

That's not a totally good workaround, at least in my view, though as it would cause the policy history to flush on computers we may just be intending to re-enroll independent of a reimage.

Amanda Wulff
JAMF Software Support

[CasperSally]

@amanda.wulff - I wasn't clear, sorry. I'm 9.32 and my flag is 1. My policy logs are flushing. My vnc and imaging logs aren't (which is actually what I want to happen, but not expected behavior from what I'm told).

What isn't flushing for me is under computer record - history - management history. The JAMF rep I'm working with said this is supposed to flush with that flag at 1. When I try to push computer level config profiles post image, I get a 'unable to decrypt profile' on reimage. If I delete the computer from JSS and reimage, the computer level profiles come down ok (there's no management history obviously on new computer).

Sorry if I'm clouding the issue, but figured I'd throw it out there in case others see this.

[were_wulff · Accepted solution]

@CasperSally

Hmm, yeah, that does sound like it's slightly different behavior from these two defects, though I wouldn't rule out the possibility of it being somehow related just given the table that's involved.

Just for some level of clarity, in 9, the table in question is mac_os_x_enrollment.
The field that we’re seeing the defects around is flush_management_history_on_remanage.

We'd probably have to dig into it a bit more to see if what you're seeing is related to that table or if it hits a different set, though.
If you and your TAM haven't shifted the works into full debug and grabbed a clean, debug only JAMFSoftwareServer.log, then gone through a re-image of a computer that we know will be affected by this behavior, it may be worth doing that so we can hopefully get some more insight by looking at the whole massive set of full SQL queries going back and forth.

It's also possible that putting Imaging into debug mode could give us some additional information, but that could be hit or miss depending on where in the imaging process it's failing to do what it's supposed to do.

To get that, after getting things set up for full debug, just rename the current JAMFSoftwareServer.log file before restarting Tomcat to kick full debug off; that will give a log file that is just debug only. Makes it a little easier to dig through, and I may very well be the one that ends up digging through it at some point as I managed to get myself the reputation of being really good at finding things in log files...and at my first response for any question being, "What do the log files say?" :)

Amanda Wulff
JAMF Software Support

[CasperSally]

For anyone who cares about VNC logs staying in tact through the image process - be careful with jamf flushpolicyhistory. The flush wipes vnc logs as well, at least in my testing.

[Kumarasinghe]

Imaging log for the current imaging gets deleted too.
https://jamfnation.jamfsoftware.com/discussion.html?id=9410

[CasperSally]

Posting in case it helps anyone else as it had us perplexed for days.

We have flush_management_history_on_remanage from mac_os_x_enrollment set to 1 - but on reimage, only sometimes were our policy logs being wiped.

Using jamf flushpolicyhistory isn't an option for us as it wipes VNC logs and imaging logs.

Jamf said this is expected behavior when using Autorun data, they said enrollment is different and policy logs don't get wiped. To continue to use Autorun data, flush policy history, but still keep VNC logs in place through reimage, jamf told us to use a quickadd package set to install on boot drive after imaging.

[GabePPS]

@CasperSally][/url

This is what I'm seeing as well on 9.32 with auto run and reimaging our cart machines. However I never get to see the machines re-enroll. I have had to add a script to trigger the enrollment complete policy and now I'm realizing that enrollment never completes (or at least in the eyes of Enrollment Last Completed field in the JSS). I have tried to add a quick add pkg to install on boot drive after imagining but this seemed to not work amazingly well either causing some logs to reset and others not to.

Maybe I should start a new discussion with these problems, but ever since moving from 8 to 9, Ive have nothing but issues with imaging.

Gabe Shackney
Princeton Public Schools

[CasperSally]

@gshackney are you working with support? I think you may need some fixes that I think are going to be part of 9.4... at least I did.

[GabePPS]

@CasperSally
I've been with various support reps going since Feb about imaging issues. I thought we solved a few of the problems and until recently was using a beta of casper imaging that redid timing of the enrollment script since we were also getting device signature issues. This was supposed to have been fixed in 9.3 for the imaging issues I saw, but seems that other pieces are still not working as expected.

I just opened another ticket as I realized the enrollment was not completing on any of my reimaged machines.

Gabe Shackney
Princeton Public Schools

[GabePPS]

@CasperSally
Are you creating the quick add with Recon or via the enrollment web page? I'm trying to get it to run properly after imaging to also initiate the enrollment complete trigger. I think my problem is that I have other packages installing at reboot as well.

Gabe Shackney
Princeton Public Schools

[CasperSally]

I forget, I had tried both, sorry.

I don't have other packages installing at reboot.

I do have other post image scripts that run at reboot, that then call manual triggers to install software. Maybe that would get you by to install those packages?

[activitymonitor]

It looks like this issue was 'fixed' in 9.4 and now the JSS Summary correctly displays whether "Flush history on re-enroll" is enabled or disabled.

What is the best way to re-enable this setting? It has always been 'off' since upgrading to v9, but it worked anyway. Now I would like to enable it for real. Thanks.

[Chris]

On my 9.4 test-JSS the summary doesn't seem to reflect my changes to the DB

external image link

external image link

[were_wulff]

@Chris][/url][/url,

I know it's a bit of a silly question but, after you made those changes in MySQL, did you restart Tomcat before running the Summary?

Amanda Wulff
JAMF Software Support

[Chris]

@amanda.wulff Yup, restarted Tomcat, and then the whole server, just for good measure

[were_wulff]

@Chris

Noted! I just wanted to make certain that was done before I dropped one of my test machines back to 9.32 to test whether or not the display discrepancy requires toggling the option off and on again (or on and off again) to get them to match up. In your case, if you flip flush_location_information_on_remanage back to 1, restart Tomcat, then flip flush_location_information_on_remanage back to 0 and restart Tomcat, does it reflect accurately in the Summary?

If it does seem to require manual toggling in MySQL to get the Summary to accurately reflect the setting, I'll create a new defect for that.

We did have a defect in 9.3x and, as part of it, we saw the Summary not accurately reflect the setting in the database (typically, it reported false when management history flushing was set to true), but that is listed as fixed in 9.4.

Thanks!
Amanda Wulff
JAMF Software Support

[jeffk]

I noticed there is also a 'flush_management_history_on_remanage' setting in the 'client_check_in' table now. Is this where we should be changing this setting in 9.4+?

It seems like this should be an enrollment setting to me, but it is listed under Check-In in the JSS Summary.

[Chris]

@amanda.wulff][/url i just tried switching both flush_management_history_on_remanage and flush_location_information_on_remanage to 1 and then back to 0, restarting Tomcat after every change.
My summary did not pick up any of the changes, however the changes definitely take effect as expected.

[Chris]

Ok, now i think i know where i'm wrong:

The table "enrollment_settings" i was modifying applies to User-Initiated Enrollment, which apparently includes QuickAdd packages created by Recon.
Including "User-Initiated Enrollment" in the summary displays the correct values from the "enrollment_settings" table

The table "client_check_in" applies to the Check-In section of the JSS summary.
These settings seem to be used when enrolling a client during Imaging.

@amanda.wulff can you confirm that? Sorry for the confusion.

[were_wulff]

@Chris

No worries there, it stumped us for awhile as well, but that does look like that's the difference between the two tables and what they do/where they show up on the summary.

One set shows up under the User Initiated Enrollment section of the summary, the other, as you saw, under Check-In.
A couple of us had a, "Wait, why would it do that at check in?" moment when looking into it because of that.

I think it was the wording in terms of where they show up and the table names that gave a little head-scratching pause.

Amanda Wulff
JAMF Software Support