Help

intermittent client error when prompting for enrolment - unexpected status (403)

bartreardon
New Contributor III

Posted on ‎04-03-2019 03:18 PM

Hi all,

I'm in the process of getting our legacy systems enroled into Jamf Pro and a number of them (around 10% at this stage) are getting an error "Device Enrolment" installation failed.
The MDM server for your organisation returned an unexpected status (403) 63a1d2fcca9b40169afb1be88a1e0b0f

From a brief search it appears that deleting /Library/Keychains/apsd.keychain and re-booting will fix the issue but given I want to enrol ~6-800 devices this way, actioning 10-15% of those by hand could be tedious.

Taking a look at the server logs I can see these entries which started occurring the same time as I kicked off the enrolment nag.

2019-04-03 05:45:01,290 [ERROR] [lina-exec-6] [dEnrollmentIssuerVerifier] - Unable to validate issuer
java.security.cert.CertPathBuilderException: Certification path could not be validated.
    at org.bouncycastle.jce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at com.jamfsoftware.jss.utils.security.CloudEnrollmentIssuerVerifier.isIssuerValid(CloudEnrollmentIssuerVerifier.java:64)
    at com.jamfsoftware.jss.utils.SecurityUtils.isIssuerValid(SecurityUtils.java:349)
    at com.jamfsoftware.jss.utils.SecurityUtils.validateCMSSignature(SecurityUtils.java:272)
    at com.jamfsoftware.jss.utils.SecurityUtils.verifySignatureForEnrollment(SecurityUtils.java:338)
    at com.jamfsoftware.jss.frontend.enrollment.configurator.ConfiguratorEnrollmentStep.isSignatureLegit(ConfiguratorEnrollmentStep.java:160)
    at com.jamfsoftware.jss.frontend.enrollment.cloud.DEPEnrollmentStep.shouldProcessStep(DEPEnrollmentStep.java:61)
    at com.jamfsoftware.jss.frontend.enrollment.ota.OTAEnrollmentController.doProcess(OTAEnrollmentController.java:97)
    at com.jamfsoftware.jss.frontend.enrollment.ota.OTAEnrollmentController.doPost(OTAEnrollmentController.java:75)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at com.jamfsoftware.jss.frontend.streamlinedenrollment.CloudEnrollmentFilter.doFilter(CloudEnrollmentFilter.java:39)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at com.jamfsoftware.jss.frontend.JSSLoadingFilter.doFilter(JSSLoadingFilter.java:209)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
    at org.apache.catalina.valves.SemaphoreValve.invoke(SemaphoreValve.java:167)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.bouncycastle.jce.exception.ExtCertPathValidatorException: Could not validate certificate: certificate expired on 20180922043724GMT+00:00
    at org.bouncycastle.jce.provider.RFC3280CertPathUtilities.processCertA(Unknown Source)
    at org.bouncycastle.jce.provider.PKIXCertPathValidatorSpi.engineValidate(Unknown Source)
    at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
    at org.bouncycastle.jce.provider.PKIXCertPathBuilderSpi.build(Unknown Source)
    ... 42 more
Caused by: java.security.cert.CertificateExpiredException: certificate expired on 20180922043724GMT+00:00
    at org.bouncycastle.jce.provider.X509CertificateObject.checkValidity(Unknown Source)
    ... 46 more
2019-04-03 05:45:01,299 [WARN ] [lina-exec-6] [DEPEnrollmentStep        ] - Processing device rejection: FAILED_SIGNATURE_CHECK
2019-04-03 05:45:01,299 [ERROR] [lina-exec-6] [onfiguratorEnrollmentStep] - Unable to process signature for enrollment request

I'm thinking this means there is a certificate inside apsd.keychain which is expired and this is why the devices are having issue communicating. Does anyone know of a way to detect or get macOS to refresh these certificates without requiring a reboot? There (so far) doesn't appear to be any obvious combination of hardware, OS version or other software I can check against.

Also, is there a way to match up the above server log with an endpoint (IP address) - I'm running on jamfcloud otherwise I'd triangulate with the apache logs or other access logs but I don't think I have access to those.

New devices, fresh built devices, new OS installs and at least 85% of existing devices don't have an issue at all. Just seems to be a significant handful with the issue.

Labels:
0 Kudos
2 REPLIES 2

bartreardon
New Contributor III

Posted on ‎04-03-2019 03:46 PM

security dump-keychain /Library/Keychains/apsd.keychain gets me some info I might be able to check

update:

/usr/bin/security find-certificate -a -p -Z /Library/Keychains/apsd.keychain | /usr/bin/openssl x509 -noout -enddate| cut -f2 -d=

will get me the expiry date of the apsd certificate - if it's expired, then I can avoid initiating the DEP nag on that device

update 2:
The following script can be used to detect if the certificate in apsd.keychain is valid or not

#!/bin/bash

KEYCHAIN="/Library/Keychains/apsd.keychain"
DATENOW=$(date "+%Y%m%d")

if [ -e ${KEYCHAIN} ]; then  
    CERTDATA=$(/usr/bin/security find-certificate -a -p -Z ${KEYCHAIN})
    CERTEXPDATE=$(echo "${CERTDATA}" | /usr/bin/openssl x509 -noout -enddate| cut -f2 -d=)
    CERTDATECMP=$(date -j -f "%b %d %T %Y %Z" "${CERTEXPDATE}" "+%Y%m%d")

    echo "CERTDATA    = ${CERTDATA}"
    echo "CERTEXPDATE = ${CERTEXPDATE}"
    echo "CERTDATECMP = ${CERTDATECMP}"

    if [[ ${DATENOW} -ge ${CERTDATECMP} ]]; then
        echo "apsd keychain certificate EXPIRED"
    else
        echo "apsd keychain certificate VALID"
    fi
else
    echo "Could not find ${KEYCHAIN}"
fi
0 Kudos

bartreardon
New Contributor III

Posted on ‎04-04-2019 04:39 PM

So thank you to Jesse Peterson over on slack for finding this

MDM spec here https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/iPhoneOTAConfiguration/profile-service/profile-service.html#//apple_ref/doc/uid/TP40009505-CH2-SW4

with this little tidbit:

WARNING:  When device certificates signed “Apple iPhone Device CA” are evaluated their validity dates should be ignored.

so Jamf is being good for checking certificate validity but is actually not following the MDM spec by doing so - also Apple for advising certificate validity checks should be ignored ಠ_ಠ

I'm pretty confident Apple aren't going to change and a bugfix from Jamf probably won't happen in time for me to continue the rollout, but, at least I know what's going on now so I can work around it.

0 Kudos