Unknown AD traffic

[mattskaj]

I just started with a new company and they have been moving from 10.6 to 10.7. Machines are bound to AD but not utilizing "the golden triangle" with the JSS. For some reason the machines that are running 10.7.x are hammering the ACtive Directory DC's with some type of network traffic while at the login screen and pegging the cpu usage on those domain controllers. The network guys can't give us any other info than "your macs are hammering our DC's when they're at the login window and your users need to shut them down instead of logging them off to stop the traffic". If they're logged in everything is fine. If they log out we get a firestorm of network traffic. Anyone experienced this or have any ideas?

Thanks in advance!

[jarednichols]

Logs. Need them.

[mattskaj]

Understood. Will-co asap.

[jarednichols]

Server-side logs would be helpful, but you can crank up your client-side logs with this: http://support.apple.com/kb/HT4696?viewlocale=en_US&locale=en_US

(Yes it works on Lion client)

Be warned though, the 'debug' level makes ridiculously large logs to slog through.

[mattskaj]

Thanks Jared. I doubt the AD boys are going to provide any logs for us from their end. I'll see what I can find on the client side.

[jarednichols]

That's BS. Tell them if they want their DCs to not be hammered they best cough up some logs.

That's like you dropping your car off at the mechanic with a note that says, "fix it" and telling them nothing about what's wrong with it. Hate that crap.

[rockpapergoat]

if they don't tell you what's happening on their end, it makes it more difficult for you to identify the problem (if any) and provide a resolution.

make sure they have some evidence for you to look at. otherwise, it could be a classic "macs are the problem" response from windows admins.

[jarednichols]

make sure they have some evidence for you to look at. otherwise, it could be a classic "macs are the problem" response from windows admins.

ding ding ding ding

"Something's wrong. what's different in the environment? I know! Macs!"

Ug.

(yes I'm in a ripe mood today.)

[mattskaj]

It's very comforting to know you all can truly feel my pain. ;-) LOL

[mattskaj]

@jarednichols which log(s) in particular would you like to see? I'm not sure which logs are being written while the machine is just sitting at the login screen.

[jarednichols]

/var/log/opendirectoryd.log

It'll be huge if you've gone to full debug mode.

A packet capture with Wireshark could be helpful too.

[mattskaj]

Sorry for the delay. I got pulled off of this to to fight another fire that broke out. I looked at those logs. There is no communication happening. It is not network traffic that is pegging the CPU's on there DC's. I did manage to get the AD boys to send me a screenshot.

It looks more like the problem is with the mac workstation objects in AD. I am no AD guru but common sense seems to say the objects are either corrupt, broken or not in the right OU or the schema is not configured to recognize the newer 10.7.4 workstations. We did not have this problem with 10.6 and earlier. All machines running lion while sitting at the login screen appear to be pegging the DC CPU's with lsass.exe. It seems as though when the workstations are not authenticated and logged in the windows server is seeing the mac workstation objects and has to think about what to do with them. Once they log in their link to AD is established and everything is good. The AD people's solution is to completely shut down the macs rather than logging them out and completely break the link to the controller which stops the CPU's from "thinking". Would extending the schema in AD to include the macs fix this? Could it be a matter of duplicate mac workstation objects within AD or the objects not being located in a proper OU. I seem to remember when they are first imported they are dumped in a default bucket if not setup to be placed in an assigned OU.

Thoughts?

[jarednichols]

What version of AD is it?

Extending the schema isn't required.

If your AD allows duplicate computer objects, you've got far larger issues than Macs hitting your AD.

Macs shouldn't care what OU they're in. By default they'll join CN=Computers but you can specify what OU they should join either with the built-in plugin if you're binding manually or with the Casper plugin. Depending on your AD's security framework, you may need to pre-create the computer object in the appropriate OU before joining it (manually or with Casper).

[mattskaj]

Having a meeting with the AD guys. Will update soon.

[mattskaj]

Finally got some useful info from the AD guys. What they are seeing on their end is a continued query from the mac workstations that is searching through the AD tree at over 20,000 objects, then it times out at 20,000 and repeats the search again up to four times. This is what is causing the CPU utilization to peg at 100% on their DC's. Here is the log file they shared from the AD side. They are running AD 2008. One possible solution we came up with was to limit the search to just the OU the 600 or so Macs are located and not the entire forest.

Log Name: Directory Service
Source: Microsoft-Windows-ActiveDirectory_DomainService
Date: 7/26/2012 10:46:24 AM
Event ID: 1644
Task Category: Field Engineering
Level: Information
Keywords: Classic
User: OPRmpwdcr4c$
Computer: WPSDGP9D.XXX.XXXXXXX.ORG
Description:
Internal event: A client issued a search operation with the following options.

Client:
10.33.10.88:65133
Starting node:
DC=OPR,DC=XXXXXX,DC=ORG
Filter:
( & ( & (objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=XXXXXXX,DC=ORG) (objectClass=user) ) ) Search scope:
subtree
Attribute selection:
displayName,sAMAccountName,objectGUID,homeDirectory,userPrincipalName,loginShell,primaryGroupID,unixHomeDirectory,objectSid
Server controls:

Visited entries:
82803
Returned entries:
20000
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-ActiveDirectory_DomainService" Guid="{0e8478c5-3605-4e8c-8497-1e730c959516}" EventSourceName="NTDS General" /> <EventID Qualifiers="16384">1644</EventID> <Version>0</Version> <Level>4</Level> <Task>15</Task> <Opcode>0</Opcode> <Keywords>0x8080000000000000</Keywords> <TimeCreated SystemTime="2012-07-26T15:46:24.242Z" /> <EventRecordID>1659258</EventRecordID> <Correlation /> <Execution ProcessID="640" ThreadID="1648" /> <Channel>Directory Service</Channel> <Computer>WPSDGP9D.XXX.XXXXXXXXX.ORG</Computer> <Security UserID="S-1-5-21-2064994994-1422208674-226520608-1081897" /> </System> <EventData> <Data>DC=OPR,DC=XXXXXXX,DC=ORG</Data> <Data> ( &amp; ( &amp; (objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=XXXXXXX,DC=ORG) (objectClass=user) ) ) </Data> <Data>82803</Data> <Data>20000</Data> <Data>10.33.10.88:65133</Data> <Data>subtree</Data> <Data>displayName,sAMAccountName,objectGUID,homeDirectory,userPrincipalName,loginShell,primaryGroupID,unixHomeDirectory,objectSid</Data> <Data> </Data> </EventData>
</Event>