JSS not using correct DP's since v9 Update

[bentoms]

Hi All,

Since migrating to v9 in January, we've had issues where the Network Segments are not always using the correct DP.. BUT this seems to be with the JAMF Binary & not the apps as Casper Imaging, Remote & (we think) Self Service all seem to be respecting their defined DP's.

We have quite a complicated setup, but here are the Network Segments:

• External 1.1.1.1 - 255.255.255.255
• BCP DMZ 10.1.11.1 - 10.1.11.255
• Sunderland 10.1.13.1 - 10.1.13.255
• Lakeside Server VLAN 10.1.2.1 - 10.1.2.255
• Nottingham 10.1.32.1 - 10.1.33.255
• Canterbury 10.1.44.1 - 10.1.44.255
• Stockport 10.1.44.1 - 10.1.44.255
• Manhattan 10.1.52.1 - 10.1.52.255
• Tokyo 10.102.4.1 - 10.102.4.255
• New York 10.103.1.1 - 10.103.1.255
• Lakeside Wireless 10.110.1.1 - 10.110.2.255
• Sunderland Wireless 10.110.13.1 - 10.110.13.255
• Nottingham Wireless 10.110.32.1 - 10.110.32.255
• Stockport Wireless 10.110.44.1 - 10.110.44.255
• Manhattan Wireless 10.110.52.1 - 10.110.52.255
• Lakeside Mobile Device Wireless 10.112.1.1 - 10.112.1.255
• Hong Kong Wireless 10.120.2.1 - 10.120.2.255
• Hong Kong 10.2.2.1 - 10.2.3.255
• Lakeside 10.252.2.1 - 10.254.129.255
• Lakeside DMZ 10.4.1.1 - 10.4.1.255

The below is from a Mac today, that even though it falls within the Nottingham network segment.. & so should be using the DP defined in that segment (which is a local server).. instead it was trying to use a server on another site.

• Last Inventory Update:4/17/14 at 3:53 PM
• Last Check-in:Today at 5:57 PM
• IP Address:10.1.33.166
• Reported IP Address:10.1.33.166

This Mac was 1st built (3/15/13) & used at the location it's using the DP's.. even though it has an IP that falls within Nottingham & it's building has been updated as per the Network Segment.

This issue has effectively stopped us deploying PKGs though the JSS as we had 200 clients trying to download a 170MB from the DMZ Segment assigned DP.. despite being on our WAN.. this brought our MPLS links to a crawl.. :(

Anyone else seen anything similar?

[lisacherie]

I see this too - its been a problem for a while with both 8 and 9.

D-004624 in the 9 series
D-004623 in the 8 series

Apparently this is because of using last IP vs last reported IP.

[bentoms]

Thanks @lisacherie.

That's what I was advised too, but in eh example given both Client IP's are the same.

This had been working great from V5-8. :(

[were_wulff]

Hey @bentoms ,

In addition to the defects that @lisacherie mentioned, one thing I see here are some overlaps in segments, specifically that big one at the beginning.

External 1.1.1.1 - 255.255.255.255

That can cause really odd behavior, including not respecting the proper distribution point.

So, first External segment would be 1.1.1.1 - 10.1.10.255

And a few others would look like:

10.1.12.1 - 10.1.12.255

10.1.14.1 - 10.1.19.255

10.1.21.1 - 10.1.31.255

10.1.34.1 - 10.1.43.255

10.1.45.1 - 10.1.51.255

10.1.53.1 - 10.102.3.255

And so on, filling in the ‘gaps’ between segments with more “external” segments. I’d also recommend double checking my partial list there as the numbers started to melt together in my brain after a bit.

After the highest numbered segment, the final external segment would be:

Whatever.the.next.IP.is - 255.255.255.255.

We could try setting multiple “external” segments, so that they pop in and out between the various 10.1.x.x segments.

All of the new “external” segments could then be defaulted to the DP/DPs that you’re using for external clients.

Oddly, I remember that trick because I once spent almost two weeks trying to figure out why it was randomly happening (that’s the best part of it, sometimes it does manage to hit the right one) to another customer and once we cleaned up the segments to that there were multiple “external” ones to get rid of the overlap, everything started hitting the right DP.

Doesn't fix the defect, but it may work as a workaround.

Might be worth a shot here.

Thanks!

Amanda Wulff
JAMF Software Support

[bentoms]

@amanda.wulff, thanks.

The reason for the 1.1.1.1 - 255.255.255.255 is that pre v9 the most restrictive network segment will apply.

So I guess from what you've said we need to better define the segments with no overlap?

Ok, but what IP is used 1st? "IP Address" or "Reported IP Address"?

[were_wulff]

Hey @bentoms ,

I did a little digging into that one, and it does look like the logic was changed a bit between 8 and 9, which is why the overlaps don’t work the way they used to. In the cases I’ve had in the past, we’ve straightened it out by having better defined segments that did not overlap with other segments.

In regards to which IP it pulls, it kind of does a “round robin” sort of polling to determine which one it’s supposed to grab.

We’ve seen, on occasion, the logic doesn’t seem to work like we would expect it to, which is part of that defect (D-004624) that was previously mentioned. Given that, even if we do segment the external IP ranges into non-overlapping segments, we may still see occasional odd behavior if the JSS doesn’t grab the ‘correct’ IP address (if the “Reported IP Address” and “IP Address” differ) when it checks on any particular occasion for determining the distribution point to be used.

Development is aware of the odd behavior and the defect is one that is actively being worked on, so hopefully they’ll have a fix for the inconsistent behaviors that we see around network segments.

Thanks!

Amanda Wulff
JAMF Software Support

[bentoms]

Hi @amanda.wulff, thanks for the follow up.

I've also been working with @drew.duggan on this, & so am linking him in & will follow up on the ticket.

We now have some 67 Network Segments (!) with no overlaps, & after deploying my ADPassMon fork to our clients.. I checked each of our servers HTTP access logs (these & a screenshot of the network segments will be sent on the call soon)... it seems much better, we have maybe 120 out of 200 with the ADPassMon & maybe only 10 of those used the incorrect DP.

As luck would have it, I managed to get hold of one of the Macs that used the wrong DP.. (Casper Remote commands we're also going to the wrong DP FWIW).. No local JAMF binary commands seemed to correct the DP used, in the end I installed a QuickAdd & after which the mac started to use the correct DP.

I'm going to keep checking that Macs DP's as well as try & run the QuickAdd on those that seem to have the issue...

Just a little confused as why a QuickAdd would resolve (possibly), does this mean that the issue maybe client side? I thought the JSS itself would tell the JAMFBinary what DP to use, but if installing the QuickAdd resolves.. then that would possibly point to the issue being a client side one.. right?

[were_wulff]

Hey @bentoms

I can’t say it’s the first time I’ve seen re-running a QuickAdd fix off the wall situations before (once, it corrected a problem where the client machines couldn’t install anything that was in .dmg format through anything Casper; DMGs alone worked fine.), but it would be interesting to dig into it and see if we can’t find out what broke in the first place.

The JSS would tell the binary what to do, but if something happened to the binary somehow, it could manifest in odd behavior.

It’s possible that the system.log from one of the clients that was (or is) exhibiting the behavior would give a few clues.
The jamf.log might as well, but I'd look more into a system.log as it tends to have a bit more info; jamf.log would be most useful for looking for a time/date stamp of the last time it hit the wrong DP, then cross-referencing that with the same date/time in the system.log to see if anything looks odd.

It may also be worth a cross-reference with the JAMFSoftwareServer.log as well, just to see if any odd exceptions, errors, or messages were thrown at the same time.

Amanda Wulff
JAMF Software Support

[bentoms]

Thanks @amanda.wluff.

I've had a nose at a troublesome clients system.log & am seeing:

The domain/default pair of (/Library/Preferences/com.jamfsoftware.jamf, do_not_upgrade_jamf) does not exist

This is after login as we run the below at every login:

sudo jamf recon -endUsername <username>

On my mac I have the error a few times, but other recons seem fine.

[Kumarasinghe]

@amanda.wulff][/url][/url
Thanks for your great technical support on the forum.

On Page 89 of the Casper Suite Administrator's Guide 9.31.pdf it says; "If a computer belongs to multiple network segments, the JSS uses the network segment with the fewest IP addresses. If the number of IP addresses in each network segment is equal, the JSS uses the network segment with the lowest starting IP address."

So is this identified as a defect? What you described on the previous step was not inline with what it says on the Admin Guide.

We also have external network segments as bentoms described (External Networks 1.0.0.1-255.255.255.254) and we want to avoid creating lots of segments in between our internal segments.
Can we please get a official response regarding this issue?
We want to get things right before we move to v9.

Thanks

[were_wulff]

@Kumarasinghe

Given the disparity between what our Admin's Guide says and what we're obviously seeing happen, I've gone ahead and opened up a defect for this behavior ( D-006908 ).

I do know there were a few logic changes between 8.x and 9.x but, either way, we do need to determine whether this is actually defect behavior, or whether it's expected behavior and we need to get our Admin's Guide updated to reflect the change.

Development is pretty good about cornering me to discuss defects and if I don't hear anything for awhile, I know where they all sit. :)

Thanks!

Amanda Wulff
JAMF Software Support

[Kumarasinghe]

@amanda.wulff
Excellent.
Thanks for your support.

[bentoms · Accepted solution]

This issue was resolved with JAMF Support.. thanks @drew.duggan & mat from dev!

Here's how: http://macmule.com/2014/05/14/jss-using-wrong-distribution-points-after-v8-v9-upgrade/

[were_wulff]

@bentoms

Good to hear! I'll be adding that link to my giant folder of bookmarks.

Thanks for all your patience while getting it straightened out!

Amanda Wulff
JAMF Software Support

[Kumarasinghe]

@bentoms

Thanks for the info. Do you still have to use 67 Network Segments with no overlaps or can you have the Network Segments (lesser) we had in v8 with overlaps?

Thanks

[bentoms]

@Kumarasinghe, I left them inplace.

[yan1212]

@bentoms

Thanks for posting the fix in MacMule!

This has been troubling me for a while as well and I resorted to breaking up Network Segments and re-enrolling clients.

[dpertschi]

@bentoms in your experience then, this is maybe the result of failure during the database upgrade, but some of the other comments here suggest it's more generic than that? Anyone seeing this on fresh v9 installs?

I'm planing a upgrade from 8-9 soon. Global opps. - US, EU, APAC, 60 network segments (should be more), 15 DP's and more to come. The though of having to add in-between segments makes me contemplate jumping off a ledge...

[GabeShack]

I had this issue as well and the engineer I spoke with at Jamf had me run the same commands that were posted in MacMule which seemed to resolve it. I also use the External Networks 1.0.0.1-255.255.255.254 catch all.

Gabe Shackney
Princeton Public Schools

[were_wulff]

Hey all,

@bentoms article should be sufficient, in most cases, to keep the behavior from happening.

Just as an FYI, however, we do still have three open defects relating to network segments to be aware of as they can cause occasionally undesirable behavior:

D-006908 is listed as open by development, and has to do with the overlap issue; what appears to be happening is that the JSS isn’t always respecting the smallest/most restrictive network segment, but the issue is intermittent. It’s related to what support & @bentoms found, but doesn’t appear to be 100% the same thing.
I updated that defect to add a link to @bentoms article in the workarounds section, so any of our support people will easily be able to find it if necessary.

D-004624 has to do with how the JSS evaluates which IP address to use when deciding which network segment a device belongs in; the defect goes into Last IP vs. Last Reported IP.
If they’re the same, we don’t have any problems.
If they’re different, we see that sometimes the JSS picks the one we may not want it to pick. While it is working as written (Network segments currently evaluate both Last IP and Last Reported IP at the same time), we’ve found (and customers have found) that it can be a problem when trying to dynamically assign distribution points based on IP address. This one is more a ‘change how it works’ than it is a defect since, as it’s written, it’s currently working as intended.
We’re just finding that “as intended” possibly isn’t the best way the JSS could be doing things.

D-006267: Network segment distribution point override can be ineffective if a computer default is set. We see this in the 8 to 9 upgrade environments. This is one that is mentioned in the steps in @bentoms article, however, if you’re not super comfortable in the MySQL command line, or have any questions about the process, please get in touch with your Technical Account Manager as the workaround for this one goes into the territory of manually changing table data in MySQL.
Better safe than sorry when doing that. I’ve also added @bentoms link to the workaround section for this one as well, so it’s easy for anyone in support to find.

All three of those are related in terms of behavior seen, but not quite the same things.

As always, if you have additional questions, aren't sure if what you're seeing in your environment is a defect or not, or have gone through the steps listed throughout this discussion and are still having problems with computers using the wrong distribution points, please get in touch with your Technical Account Manager either by giving them a call directly, calling our support line, sending an e-mail in to support@jamfsoftware.com, or using the My Support section of JAMF Nation.

Thanks!

Amanda Wulff
JAMF Software Support

[bentoms]

Thanks @amanda.wulff..

Darrin, the network segments overlaps was something I tested & it wasn't that.

The "fix" I posted can be verified by updating to v9 on a dev box & then run the command to view the assigned DP's. If the values are not -1 then you're seeing the same defect.