JAMF SCCM Plugin - Inconsistent Hardware Inventory - some fail, some succeed

warrenpilkingto
New Contributor III

I'm having some issues with the newly installed JAMF SCCM Plugin.

The plugin is correctly installed with all the relevant certificates, and is able to communicate with our JAMF Pro Server and authenticate. The ISV proxy certificate is also correctly set up and referenced correctly by the settings.xml file in the SCCM Plugin.

When an inventory scan happens, some machines report back all their inventory, including hardware inventory, correctly, and this is reported in the jamf_proxy_service.log file showing all the DDR discovery, software, hardware etc.

However, within the same scan, and sometimes just a few seconds later than a successful inventory, another machine is able to bring in its DDR disvcovery and software inventory, but the hardware inventory shows an error:

There was an unexpected error sending the hardware inventory for 'ABC1234' SMSID: F326C9CD-4E33-5BF8-AF18-C856D2253606

Where ABC1234 is the machne name, and the SMSID is a correctly assigned unique machine ID for SCCM.

What would lead to the inconsistencies, and how can they be resolved?

Also, the plugin appears to use a SCCM messaging DLL file based on the SCCM 1706 SDK. This SDK along with the version of SCCM in use is at 1902. Are there any plans to update the plugin or the DLL files?

1 ACCEPTED SOLUTION

warrenpilkingto
New Contributor III

I noted that version 3.70.0 was released yesterday, as it was available in the My Assets part of JAMF Nation (so for other users, it should be too).

Having performed the upgrade to 3.70.0 (backup settings.xml, uninstall old, install new), the plugin is a huge improvement.

Hardware inventory worked correctly with UK based dates, and in the SCCM Admin console side I can see the hardware inventory date reporting back correctly (a fair number stated 30/07/2019 which is what I'd expect.) It worked correctly in SCCM 1902 with update KB4500571 applied, and will re-check functionality once 1906 is released and post back.

A couple of minor points which may need to be looked at for future releases:

  • Even though my account has admin equivalent priveleges, I still had to call the MSI installer (using msiexec /i "Jamf SCCM Plug-in-3.70.0.0.msi") from an admin command prompt, or else I get the "you do not have enough rights" message. The installer should have intelligence to know you are an admin equivalent

  • The default run time for the proxy is every 30 minutes (it shows in settings.xml as a default when installed), which can cause some resource hog if you've installed the service on the SCCM server. Could this be set by default to 1440 minutes (once per day) in future, or be an option during the install for the user to set what they would like? This way it matches a reasonable compromise in hardware inventory pulls. I did set it to 1440 and restart the service afterwards and that sorted it, but be nicer to have it selectable.

View solution in original post

21 REPLIES 21

drhoten
Contributor II

Hello @warrenpilkington

Have you tried running about the proxy service and the SCCM plug-in in debug mode to see if it provides more details?

Stop the service, and edit the settings.xml file by changing the line

<level value="INFO" />

to

<level value="DEBUG" />

Leaving the service off, make update the following registry setting on your SCCM server to enable ISV Proxy events to be logged. (By default, SCCM does not log these events.)

HKLMSoftwareMicrosoftCCMLogging@GlobalLogLevel to 0

Then run the Helper application and send over a couple of test devices to SCCM, including the machines that previously failed to see if that provides more detailed errors in the jamf_proxy_service.log on the computer with the proxy service, and the following log files on the SCCM server typically in C:Program FilesSMS_CCMLogs

  • CcmMessaging.log
  • ClientAuth.log
  • MP_CliReg.log
  • MP_RegistrationManager.log <= Look here first for registration related errors
  • MP_Ddr.log
  • MP_Hinv.log
  • MP_Sinv.log
  • and C:Program FilesMicrosoft Configuration ManagerLogsdataldr.log

Send these logs to support along with the contents of the debug folder from the application folder of the service, and have them forward it to the development team. (Also, don't forget to turn off DEBUG mode and revert the registry settings.)

Yes, we have a new version of the plug-in currently in private beta that is using version 5.1902.1006.1000 of the SDK.

warrenpilkingto
New Contributor III

@drhoten Many thanks for your response, much appreciated.

I followed your instructions, and attempted to send a couple of devices over to SCCM where I knew that the import of hardware inventory had failed previously (but everything else had succeeded)

Interestingly, the debug log file for JAMF's proxy did not show any errors, but MP_Sinv.log showed this at the time the machine was attempted to be sent over:

Sinv Sax: Skipping unexpected token at Machine MP_SinvEndpoint 17/06/2019 09:42:39 35664 (0x8B50)
Sinv Sax: Skipping unexpected token at /Machine MP_SinvEndpoint 17/06/2019 09:42:39 35664 (0x8B50)

The unexpected token error normally relates to some form of JSON handling, although the debug logs did mention a HTTP status 200 (OK) when connecting to the JAMF Pro server to retrieve the XML information.

There was no errors in MP_RegistrationManager.log (clients are registering as devices in SCCM when being sent over) - however the lines in MP_ClientAuth.log might prove more useful:

Error verifying local message from client (0x80070057). ClientAuth 17/06/2019 09:42:39 14676 (0x3954)
Local verification of CCM message failed. (0x80070057) This may be a result of two machines with different FQDNs but the same NetBIOS names, falling back to normal verification. ClientAuth 17/06/2019 09:42:39 14676 (0x3954)

The same errors above happened when the data pull from the proxy service happened during its cycle there, so this is probably the best place to start looking. I'll send the logs to support in the meantime, but if there's anything you can point us to, please do let me know.

drhoten
Contributor II

You're welcome @warrenpilkington, and it sounds like sending it a second time may have worked but would definitely be interested in reviewing your logs.

I've seen the Sinv Sax related error before, even with working Software Inventory Reports. For data parsing errors, you can follow the steps outlined in this post to dig a bit deeper.

When send over the logs, be sure to note the NetBios name of the computers you were sending along with the time. The goal would be to review both your proxy and SCCM logs to see the data flow from the Proxy to SCCM.

warrenpilkingto
New Contributor III

@drhoten Thanks for that. I did actually see the same hardware inventory sending error even when sending it from the helper tool. In essence, the two machines with different FQDNs error as mentioned earlier.

Only the hardware inventory particularly according to the XSL stylesheets is looking for the domain name, or the proxy domain, depending on the settings config and what it finds. The XSL for hardware inventory states that the domain can be hard coded - I was wondering if it's worth setting that as a hardcode (I'd need to know how to set that) and then re-send from the helper tool to see if that helps, just in case there's some data flow errors.

All the other parts (discovery data record (DDR), software inventory) work fine, the GUID matches exactly, hence I was wondering if the messaging, due to it using an older SCCM SDK release, somehow isn't as consistent and causing an error.

I made sure the log files were newly created when sending the error over, so the debug folder should only show the machine name I attempted when support receive that. I should also point out that from a Windows perspective, all the hardware inventories etc are successfully populating in SCCM without any errors.

warrenpilkingto
New Contributor III

Also, the hardware inventory scan appears to be sending as US instead of UK dates, but all other data being sent from JAMF is in the correct date format:

48266520d138468a8acc59b12e13343f

Maybe this is also why the error occurs because the dates are inconsistent - 06/12/2019 is a valid UK date, even though in US date terms it's 12th June. So anything from 13th onwards would attempt to put through a date not valid - 06/13/2019 would be rejected I take it?

drhoten
Contributor II

Yes, @warrenpilkington you're absolutely correct an improperly formatted date can cause the HINV record to be rejected and is something we fixed in development and will be released in an upcoming version of the plug-in.

warrenpilkingto
New Contributor III

@drhoten Good to know. To give you a little background on this, and which may help, I used the Helper utlity to fetch one machine's XML from Jamf Pro. I was then able to see this in the XML:

</user_approved_enrollment><user_approved_mdm since="10.4.0">true</user_approved_mdm></management_status><report_date>2019-06-17 06:21:25</report_date><report_date_epoch>1560748885237</report_date_epoch><report_date_utc>2019-06-17T06:21:25.237+0100</report_date_utc><last_contact_time>2019-06-17 13:21:48</last_contact_time><last_contact_time_epoch>1560774108043</last_contact_time_epoch><last_contact_time_utc>2019-06-17T13:21:48.043+0100</last_contact_time_utc>

So, the date is coming in as US date (YYYY-MM-DD) but it seems that in translation it attempts to translate into SCCM as YYYY-DD-MM, hence any date past the 12th will come in as invalid. In essence, ClientAuth.log from the SCCM side can't recognise the date of the data and assumes it's invalid.

As it is, any pulls taken from the service from the 1st to 12th of the month currently would work, which is probably why I saw those inconsistencies from 13th onwards, existing machines with hardware inventory had a Delta pull in, but those without would attempt, and then fail.

Am I able to obtain the upcoming version of the plugin so I can see if it rectifies the issue here?

warrenpilkingto
New Contributor III

JAMF Support have confirmed to me that there is a product issue PI-006838 - where a non-US locale is used, the SCCM plugin does not report the correct date and time.

This is resolved in version 3.70.0 of the JAMF SCCM plugin, when released. Worth knowing for any other UK based users especially.

drhoten
Contributor II

@warrenpilkington Yes, that is correct. We are starting the process of getting 3.70.0 released but I do not have a specific date we can communicate yet, and will post back here when it's released.

warrenpilkingto
New Contributor III

I noted that version 3.70.0 was released yesterday, as it was available in the My Assets part of JAMF Nation (so for other users, it should be too).

Having performed the upgrade to 3.70.0 (backup settings.xml, uninstall old, install new), the plugin is a huge improvement.

Hardware inventory worked correctly with UK based dates, and in the SCCM Admin console side I can see the hardware inventory date reporting back correctly (a fair number stated 30/07/2019 which is what I'd expect.) It worked correctly in SCCM 1902 with update KB4500571 applied, and will re-check functionality once 1906 is released and post back.

A couple of minor points which may need to be looked at for future releases:

  • Even though my account has admin equivalent priveleges, I still had to call the MSI installer (using msiexec /i "Jamf SCCM Plug-in-3.70.0.0.msi") from an admin command prompt, or else I get the "you do not have enough rights" message. The installer should have intelligence to know you are an admin equivalent

  • The default run time for the proxy is every 30 minutes (it shows in settings.xml as a default when installed), which can cause some resource hog if you've installed the service on the SCCM server. Could this be set by default to 1440 minutes (once per day) in future, or be an option during the install for the user to set what they would like? This way it matches a reasonable compromise in hardware inventory pulls. I did set it to 1440 and restart the service afterwards and that sorted it, but be nicer to have it selectable.

drhoten
Contributor II

Thanks for the update @warrenpilkington, great to hear it's working correctly for you now.

Is the issue with "you do not have enough rights" prompt occurring on a machine, where you are logged in as the non-default Administrator account but the account is still part of the Local Administrators group? We've run into other permission related issues in the past without this check.

Are you seeing a negative impact on your server having the proxy service run every 30 minutes? While it does run every 30 minutes, it may not send data to SCCM every time. When it runs, it gets a list of all Computers and all Mobile Devices using the /subsetbasic/ endpoints in our Customer API. The endpoints only return some of the attributes of each device, along with the timestamp of when the inventory for that device was last updated.

The proxy service then filters the list to only the devices where the inventory has been updated since the last time it ran. So only the devices that have been updated in the last 30 minutes are sent to SCCM. Just wanted to point it out, since running it once a day could end up creating a larger load on your server since there will be more devices to send to SCCM at that time.

warrenpilkingto
New Contributor III

Hi @drhoten

Just reporting back in following a couple of upgrades at our end.

  • The SCCM plugin is working perfectly fine with SCCM 1906. I can see deltas and full inventories reporting back with no issues.
  • The "do not have enough rights" issue - Yes, I use an account part of the local administrators group. Normally a MSI would be clever enough to detect that, but suspect because you may need some of the services to run at system level, it needs a higher elevation level - that might be why it ran into the permissions issue?
  • Having performed some further tests and investigation, the amount of inventory data collected by JAMF Pro collected to SCCM is relatively small - although of course most of us customise the SCCM hardware collection so not every single piece of WMI data is collected. In theory, it'd be fine to increase the frequency, but there's also overnight maintenance tasks running on the server, so didn't want the services to be running at the same time as that. The good thing is that every 1440 mins (once every 24 hours) is working, although maybe for a future release this could be customised to specify "once a day" and at "time"?

drhoten
Contributor II

Thanks for the update @warrenpilkington, glad to hear it's working for you.

We do install the service as "Local System" which requires the higher privileges along with how we are encrypting and storing the password for the API. To immediately run as an administrator, has in the past often required the MSI to be launched from EXE wrapper which is a non-starter for some customers requiring only an MSI to install the plug-in.

Since you are trying to prevent the service from running at times where there is a heavier load on the server, would creating an option for preventing the polling from running at certain times also work for you?

warrenpilkingto
New Contributor III

@drhoten Thanks for your response.

Ideally, an option for preventing polling to run at certain times of day would be a good idea (similar to the SCCM maintenance window concept), and/or be able to configure when the polling takes place - SCCM for example can use a schedule based on hours, days, weeks etc and you can specify when that schedule is. Either or both options would be appreciated.

As the service installs as the local SYSTEM user, it would make sense - you could of course deploy the MSI via SCCM as an application (the installer runs as the SYSTEM user and works), although for most people, the plugin is going to be a one-off install. There should be some way of the installer recognising you're a member of Administrators, and elevating the service install to run as the SYSTEM user accordingly. Other software can do so.

One other thing: the inventory reports back that machines have a SCCM client installed with version 5.00.8325.000 - which is a non-existent version (SCCM 1511 was 5.00.8325.1000). Was there any reason for the inventory record stating that a client was present or is it the way that the inventory reports back? Naturally for systems excluding Macs I can do a collection query in SCCM to exclude any clients with that version (for example if I need a Windows only based collection for limiting purposes) - but may be something to check over in the next release?

drhoten
Contributor II

Hi @warrenpilkington

For scheduling the polling to run only at certain times, would maintaining these settings be acceptable in the settings.xml or would you prefer to see this configurable in the helper application and/or installer?

Would including a bootstrapper application (an exe) be acceptable in your environment? We could possibly solve this by providing folks the option of either running the EXE or the MSI so we could include an application manifest that specifies the correct execution level with the EXE.

I'll have to double-check, but the SCCM client may be specified by default by the SCCM SDK. In this case, would you want to see the version of the SDK being used or perhaps even the version of the plug-in itself?

georgecm12
Contributor III
Even though my account has admin equivalent priveleges, I still had to call the MSI installer (using msiexec /i "Jamf SCCM Plug-in-3.70.0.0.msi") from an admin command prompt, or else I get the "you do not have enough rights" message. The installer should have intelligence to know you are an admin equivalent

For what it's worth, the JSS installer for Windows has always behaved this way as well.

warrenpilkingto
New Contributor III

Just a quick update: I updated SCCM to version 1910 a week and a half ago, which now becomes MECM (Microsoft Endpoint Configuration Manager) - and the plugin works correctly without any issues.

NOTE: For those of you who have the settings for inventory set to be default or frequently (say anything under 2 hours) it is advisable to stop the JAMF SCCM Plugin service before you perform any SCCM upgrade - that way data won't be attempted to be pulled into the SCCM database whilst being upgraded and coming up with the plugin error.

@drhoten For the future plugin releases, having an option for polling during install or by use of tweaking the settings.xml file accordingly (in case you need to change it) is well worth consideration. More than happy to test this in any new release.

drhoten
Contributor II

Thanks for sharing @warrenpilkington, it's really appreciated!

That's a great tip about upgrading and will see what I can do to get that added to our documentation.

Regarding your feature request, are you referring to being able to change the polling interval from the installer and helper form or being able to run the polling on-demand?

warrenpilkingto
New Contributor III

@drhoten I'm referring to the polling interval from the installer / helper form. So you can set it initially during install, and then if you change your mind, you can stop the service, run the helper and change the time, then restart the service.

A stop/start of the service would do an initial poll anyway, so if you wanted to do on demand, there's always that way.

Bear in mind when MECM has its hardware inventory set to once a day, it randomises throughout the whole day to ease load, whereas the polling interval in the Jamf plugin is set to get all deltas from the database. Granted, any delta regardless of platform is smaller, but if you have a considerable number of Macs, you may not want to pull x thousand deltas every hour or so.

drhoten
Contributor II

Thanks for the clarification @warrenpilkington.

By default, the plug-in is configured to run every 30 minutes and will send either a full report or a delta based on whether the plug-in has a copy of the previous report it sent to SCCM. If it does not have a copy it sends a full report, otherwise, it generates a delta.

The original intent behind this was to ensure updates for a large fleet could be distributed over a longer period of time vs. running the plug-in once a day. In your environment what interval are you finding works best for you?

RomanKruglov
New Contributor II

I have a small issue with the Jamf Proxy and SCCM when checking the proxy log all looks good and it is sending the info to the SCCM when checking SCCM the machines are showing up under devices but no other information is coming across, no operating system or primary user or anything else, when checking MP_Hinv.log I see the error: sccm hinv sax: skipping unexpected token at machine. I assume it's not liking something in the xml files that Jamf Proxy is sending, any ideas? Tried working with support and so far 3 days in the row besides sending log files, they have not asked me to even try anything or made any suggestions