Uploading audit files via API

[AVmcclint]

We have an annual PCI audit that we have to run on randomly selected Macs once a year. After the Macs have been selected, I scope those Macs to a policy that runs the script below. Everything works up to the point of uploading the resulting .zip file to the computer record on the Jamf server. I keep getting a "curl: (26) Failed to open/read local data from file/application" error. Unfortunately my curl skills are weak and I can't figure out what I'm doing wrong. I think I'm doing something wrong with the -F option or I'm not using the variables correctly in the curl command (line 91).

#!/bin/bash

# this script s used to collect a full system profiler dump in txt format
# as well as user accounts, groups, crowdstrike info, open ports
# it will put all the files in /Library/company/PCI/<computername>/
# for later retrieval and submission for PCI audit

# Set variables for the JAMF API
jamf_url="https://company.jamfcloud.com"
jamf_user="API-only-account"
jamf_password="password_here"
computer_id=$(jamf recon | grep '<computer_id>' | awk -F'[<>]' '{print $3}')
echo $computer_id


SERIAL=`ioreg -l | awk '/IOPlatformSerialNumber/ { print $4;}' | sed 's/"//g'`
echo $SERIAL
PCI_PATH="/Library/company/PCI/$SERIAL"
timestamp=$(date "+%Y.%m.%d")
year=$(date "+%Y")
auditlog="$SERIAL-$timestamp.zip"
echo $auditlog

# if there is already a folder in the target location (probably from an earlier capture) this will delete it
if [[ -d /Library/company/PCI/$SERIAL ]] ; then
	rm -Rf /Library/company/PCI/$SERIAL
fi

# Creates the ./PCI/<serial number> folder
mkdir -p $PCI_PATH

# Local user accounts
# dscl . list /Users | grep -v '_' > /Library/company/PCI/$SERIAL/User.txt

# All accounts (human and system) and Directory with Name
# dscacheutil -q user > /Library/company/PCI/$SERIAL/UserDirectory.txt

# Groups
# dscacheutil -q group > /Library/company/PCI/$SERIAL/groups.txt

# All user and system accounts that do not start with _
dscacheutil -q user | perl -00ne 'print if !/name: _/' > $PCI_PATH/Users.txt
echo "user and system accounts saved
"
# All groups that do not start with _
dscacheutil -q group | perl -00ne 'print if !/name: _/' > $PCI_PATH/Groups.txt
echo "all non invisible groups saved"

# Generates a plain text list of applications and version info
system_profiler SPApplicationsDataType > $PCI_PATH/Applications.txt
echo "list of all applications saved"

# Crowdstrike Status
/Applications/Falcon.app/Contents/Resources/falconctl stats > $PCI_PATH/crowdstrike.txt
echo "Crowdstrike status saved"

# Lists Services in a similar manner to what is listed in the computer record in Jamf

cat << 'EOT' > /Library/company/PCI/$SERIAL/SERVICES_NOTE.txt
the list of services was generated by the following command:
/bin/launchctl list | /usr/bin/awk '{print substr($0, index($0, $3))}' | /usr/bin/sed '1d'
EOT

echo "" > $PCI_PATH/Services.txt
/bin/launchctl list | /usr/bin/awk '{print substr($0, index($0, $3))}' | /usr/bin/sed '1d' >> $PCI_PATH/Services.txt
echo "list of services saved"


# full system profiler dump
system_profiler -xml -detailLevel full > $PCI_PATH/system_profiler_full.xml
echo "The system_profiler_full.xml file can be renamed with .spx extension and can then be opened directly with the System Information.app found in macOS of all versions" > /Library/company/PCI/$SERIAL/SYSTEM_PROFILER_NOTE.TXT
echo "System Profiler dump saved
"
# Open ports listening
lsof -i -P | grep -i "listen" > $PCI_PATH/openports.txt 
echo "list of open ports saved"

# evidence of time sync
touch /var/db/ntp-kod
chmod 666 /var/db/ntp-kod
sntp -sS time.apple.com > $PCI_PATH/timesync.txt
echo "" >> $PCI_PATH/timesync.txt
echo "" >> $PCI_PATH/timesync.txt
echo "commands used to generate this data: touch /var/db/ntp-kod ; chmod 666 /var/db/ntp-kod ; sntp -sS time.apple.com" >>$PCI_PATH/timesync.txt
echo "time sync info saved"

# Zips up all the files generated in the /PCI/<serial number>/ folder
zip -r $auditlog $PCI_PATH 

# Upload file to computer record using JAMF API
curl -X POST -u "${jamf_user}:${jamf_password}" "${jamf_url}/JSSResource/fileuploads/computers/id/${computer_id}" -F name=@auditlog -F file=@$PCI_PATH-$timestamp.zip

echo "Evidence gathered for the $year Audit. Done"

 Can someone a little more curl-savvy point out where I went wrong?

[sdagley]

@AVmcclint Have you tried running the command without the 

 -F name=@auditlog

parameter? The Jamf Pro API may only want the file parameter. (And here's an example of using the API to upload an attachment if you haven't already seen it: https://github.com/kc9wwh/logCollection/blob/master/logCollection.sh)

You should update the script to use Bearer Token auth since simple auth is deprecated.

[mjhersh]

I have a similar script that only uses "name=", not "file=". Like this:

curl -su $JAMF_PRO_USERNAME:$JAMF_PRO_PASSWORD $JAMF_PRO_URL/JSSResource/fileuploads/computers/id/$COMPUTER_ID -X POST -F name=@/tmp/somefile.zip

This script is from a year or so ago, but should still work. You should probably update your scripts to use modern authentication, though, since basic auth will go away at some point.

[AVmcclint]

I know I need to update the authentication, but right now my focus is just figuring out how to upload files.

As for trying it without the name= or file= I have tried both (alternately) with the same error. The name of the zip file that is created will be unique to every Mac the script is run on so that's why I'm trying to use variables. Otherwise I would specify an exact filename.

For experimental purposes I changed it to -F name=@/exact/path/to/file.zip and it still failed with the same error.

I'll have to take a look at that logColleciton.sh script when I have some time.

[AJPinto]

Commenting as I am interested to see how this sorts out :). 

I thought JAMF deprecated the ability to upload files (attachments) directly to JAMF like this? I am reaching but something was mentioned about this in my JAMF 300 a couple of years ago.

[AVmcclint]

Oh I hope you're wrong. There's an Attachments section in the computer record that I presumed was for exactly this purpose.

[mm2270]

Try this variation.

1. Use name=@ instead of file=@
2. Double quoting the zip file variable name (Note the open quote comes after the @ symbol and closes at the end of the variable name)
3. Put the -X POST at the end of the command. This may not make any difference, and it shouldn't, but it's how my own API file upload script is set up, and it seems to work for me.

curl -u "${jamf_user}:${jamf_password}" "${jamf_url}/JSSResource/fileuploads/computers/id/${computer_id}" -F name=@auditlog -F name=@"$PCI_PATH-$timestamp.zip" -X POST

[AVmcclint]

@mm2270 I tried what you suggested and it still failed with the same error. I remembered that I can get help on api calls by going to company.jamfcloud.com/api so I did and looked up the fileuploads section. I removed all the quotes and curly braces and stuff and let the variables sit as $variable in the command. I then ran it as bash -x script.sh so I could watch all the output. It still failed, but the -x option confirmed that my variables are correct... it looks like this: 

curl -k -u API-only-account:password_here https://company.jamfcloud.com/JSSResource/fileuploads/computers/id/1334 -F name=@/Library/company/PCI/SERIALNUMBER/SERIALNUMBER-2023.03.08.zip -X POST 

Which according to the sample command, is formatted exactly how it should be.  However it still gives the same "

curl: (26) Failed to open/read local data from file/application" error.  My searches for that curl error find a wide variety of issues, but I can't seem to find what specifically is the problem here.

[mm2270]

@AVmcclint what version of Jamf Pro are you running? Just curious, as I may try this on a more recent version to check that nothing broke (or changed) with that API resource. I hope it still works to upload file attachments this way, or one of my workflows is going to be broken.

[mjhersh]

I can confirm that the file upload has not been disabled in 10.44.1. I did, however, need to update my script to use modern auth. 

This is my curl call:

curl -X POST -H "Authorization: Bearer $TOKEN" $JAMF_PRO_URL/JSSResource/fileuploads/computers/id/$COMPUTER_ID -F name=@"$FULL_FILE_PATH"

$TOKEN can be pulled using basic auth like this:

TOKEN=$(curl -X POST -H "Content-Type: application/json" -u "$USERNAME:$PASSWORD" "$JAMF_PRO_URL/api/v1/auth/token" | jq -r '.token')

Is it possible your API username or password contains bash-unfriendly characters such as "!"? This has led to confusing error message for me in the past. zsh generally handles string substitution better than bash, but my personal preference is to avoid wonky characters in the first place.

Also worth noting is that I keep my file paths free of spaces and other special characters as well. I'm not entirely sure if curl handles that properly. I assume it would with quotes but I have not tested.

[AVmcclint]

We are running JamfPro 10.44.1.

After bashing my head against this for a while, I have come to the conclusion that I may not even be getting through the front door despite the error referring to "local data". I have never done the token authentication before, so I decided to tackle that. I tried your code, @mjhersh but the jq command is not recognized. I see that is not included with the macOS by default and would need to be a 3rd party install. I need this to work out of the box. I'm not using any special characters and have been very careful to not have spaces in any paths.

I can say that I'm getting closer. I found this article https://developer.jamf.com/jamf-pro/recipes/bearer-token-authorization that tells me how to generate a token.  I can say I'm no longer getting curl errors.  However, when I check the Attachments section of the computer record, there's nothing there.

Here is the final (obfuscated) curl command that is being sent:

curl -s -H 'Authorization: Bearer longencodedgibberishtokentextlongencodedgibberishtokentextlongencodedgibberishtokentext' https://company/JSSResource/fileuploads/computers/id/1334 -F name=@/Library/company/PCI/SERIALNUMBER/SERIALNUMBER-2023.03.09.zip -X POST

There are no errors after that command is sent and there is no sign of success either. 

[mm2270]

Can you, or have you run the command without the -s flag for curl, to see if it's spitting back any specific error that may help diagnose the problem?

I also throw in the silent flag when using curl, because I normally don't want to see all that traffic chatter, but in this case, it may be worth seeing that information. A successful curl command to the API usually returns some information to indicate it worked.

[AVmcclint]

When I remove the -s option, the error comes back. error 26. I guess I'm no closer than I was before. I've run the script in CodeRunner, Terminal, and even threw it into Jamf to run in Self Service - all with the same result.  The permission on the local directory is 755 with root as the owner. The zip file that is created has 644 permissions also with root as the owner. The script is being run as root in all cases.

[AVmcclint]

I tried something else to eliminate the possibility of my path variables being the problem. I changed the 

name=@"$logpath" part to specify an exact file on my desktop: name=@/users/me/desktop/file.txt 

This time I got something!

<html>
<head>
   <title>Status page</title>
</head>
<body style="font-family: sans-serif;">
<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Unauthorized</p>
<p>The request requires user authentication</p>
<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2">here</a>.<br>
Please continue your visit at our <a href="/">home page</a>.
</p>
</body>
</html>

I did some more digging and I figured, let me see what happens if I use MY credentials as a Global Admin. If any account is going to work, it'll be mine. IT WORKED!  While riding that surge of endorphins I figured I'd change the name= back to using the $logpath variable. It failed with that curl error 26. I figured there must be something up with a pure variable name so I added a cp $logpath /tmp/  right after the zip file creation and before the curl command. Then I changed name= to name=@/tmp/$auditlog and it worked! I see the files uploaded to the Attachments section of the computer record. Now the real question is why isn't the server account I created solely for API working? 

The account has Full Access, Custom, Enabled. set under the Account tab.

On the Privileges Tab > Jamf Pro Server Objects:

Advanced Computer search READ

API Integrations CREATE READ UPDATE

Computers READ UPDATE

File Attachments CREATE READ UPDATE

But it still gives that darned "Unauthorized" error. Do I have to do something to the account so it can be allowed to use bearer tokens?

[AVmcclint · Accepted solution]

SOLVED!! It turns out there were a couple things working against me on this.

1. I had some issues with the name= part of the curl command. To simplify it I copied the zip file to a known location "/tmp/" and then used a single variable for the file path. It turned out to be easier than trying to layer multiple variables into the curl command. This solved the curl 26 error.

2. I thought I had issues with using basic authentication so I decided it was time I jump in with Bearer Tokens anyway. Turns out that basic auth wasn't the problem (because I could basic authenticate with my global admin account), but I needed to move forward anyway.

3. I had to do some deep digging to discover that the API-only account I am using needed CREATE rights on Computers in addition to all the other obvious rights. This solved the http error I was getting.

Here is the final working (and obfuscated) script that I verified in CodeRunner, Terminal, and Self Service.

#!/bin/bash

# this script is used to collect a full system profiler dump in txt format
# as well as user accounts, groups, crowdstrike info, open ports
# it will put all the files in /Library/company/PCI/<computername>/
# for later retrieval and submission for PCI audit

# Set variables for the JAMF API
jamf_url="https://company.jamfcloud.com"
jamf_user="APIuser"
jamf_password="APIpassword"

#Variable declarations
bearerToken=""
tokenExpirationEpoch="0"

getBearerToken() {
	response=$(curl -s -u "$jamf_user":"$jamf_password" "$jamf_url"/api/v1/auth/token -X POST)
	bearerToken=$(echo "$response" | plutil -extract token raw -)
	tokenExpiration=$(echo "$response" | plutil -extract expires raw - | awk -F . '{print $1}')
	tokenExpirationEpoch=$(date -j -f "%Y-%m-%dT%T" "$tokenExpiration" +"%s")
}
checkTokenExpiration() {
	nowEpochUTC=$(date -j -f "%Y-%m-%dT%T" "$(date -u +"%Y-%m-%dT%T")" +"%s")
	if [[ tokenExpirationEpoch -gt nowEpochUTC ]]
	then
		echo "Token valid until the following epoch time: " "$tokenExpirationEpoch"
	else
		echo "No valid token available, getting new token"
		getBearerToken
	fi
}

checkTokenExpiration

computer_id=$(jamf recon | grep '<computer_id>' | awk -F'[<>]' '{print $3}')
echo "Jamf computer ID: $computer_id"

# Path and timestamp variables
SERIAL=`ioreg -l | awk '/IOPlatformSerialNumber/ { print $4;}' | sed 's/"//g'`
echo "serial number: $SERIAL"
PCI_PATH="/Library/company/PCI/$SERIAL"
echo "Path to log file: $PCI_PATH"
timestamp=$(date "+%Y.%m.%d")
year=$(date "+%Y")
auditlog="$SERIAL-$timestamp.zip"
echo "log file name: $auditlog"
logpath=/Library/company/PCI/$auditlog
echo "the full path to the zip file is $logpath"

# if there is already a folder in the target location (probably from an earlier capture) this will delete it
if [[ -d /Library/company/PCI/$SERIAL ]] ; then
	rm -Rf /Library/company/PCI/$SERIAL
fi

# Creates the ./PCI/<serial number> folder
mkdir -p $PCI_PATH

#### START OF AUDIT COLLECTION ####
# All user and system accounts that do not start with _
dscacheutil -q user | perl -00ne 'print if !/name: _/' > $PCI_PATH/Users.txt
echo "user and system accounts saved
"
# All groups that do not start with _
dscacheutil -q group | perl -00ne 'print if !/name: _/' > $PCI_PATH/Groups.txt
echo "all non invisible groups saved"

# Generates a plain text list of applications and version info
system_profiler SPApplicationsDataType > $PCI_PATH/Applications.txt
echo "list of all applications saved"

# Crowdstrike Status
/Applications/Falcon.app/Contents/Resources/falconctl stats > $PCI_PATH/crowdstrike.txt
echo "Crowdstrike status saved"

# Lists Services in a similar manner to what is listed in the computer record in Jamf
cat << 'EOT' > /Library/company/PCI/$SERIAL/SERVICES_NOTE.txt
the list of services was generated by the following command:
/bin/launchctl list | /usr/bin/awk '{print substr($0, index($0, $3))}' | /usr/bin/sed '1d'
EOT

echo "" > $PCI_PATH/Services.txt
/bin/launchctl list | /usr/bin/awk '{print substr($0, index($0, $3))}' | /usr/bin/sed '1d' >> $PCI_PATH/Services.txt
echo "list of services saved"

# full system profiler dump
system_profiler -xml -detailLevel full > $PCI_PATH/system_profiler_full.xml
echo "The system_profiler_full.xml file can be renamed with .spx extension and can then be opened directly with the System Information.app found in macOS of all versions" > /Library/company/PCI/$SERIAL/SYSTEM_PROFILER_NOTE.TXT
echo "System Profiler dump saved
"
# Open ports listening
lsof -i -P | grep -i "listen" > $PCI_PATH/openports.txt 
echo "list of open ports saved"

# evidence of time sync
touch /var/db/ntp-kod
chmod 666 /var/db/ntp-kod
sntp -sS time.apple.com > $PCI_PATH/timesync.txt
echo "" >> $PCI_PATH/timesync.txt
echo "" >> $PCI_PATH/timesync.txt
echo "commands used to generate this data: touch /var/db/ntp-kod ; chmod 666 /var/db/ntp-kod ; sntp -sS time.apple.com" >>$PCI_PATH/timesync.txt
echo "time sync info saved"

# Zips up all the files generated in the /PCI/<serial number>/ folder
cd /Library/company/PCI/
zip -rj $auditlog $PCI_PATH 
echo "Zip file created"

# copy the zip file to an easier path to insert in the curl command
cp $logpath /tmp/

echo "uploading zip file to jamf server"
# Upload file to computer record using JAMF API
curl -H "Authorization: Bearer ${bearerToken}" $jamf_url/JSSResource/fileuploads/computers/id/$computer_id -F name=@/tmp/$auditlog -X POST

echo "Evidence gathered for the $year Audit. Done"

I am so glad JamfNation exists. Even if I don't get the exact solution I need many times the replies I get spark something in my brain to get me thinking about other possibilities. I am also glad I have CodeRunner. This app has made script writing so much easier as far as finding where things break. I am glad that bash -x script.sh is a thing to let me see every step of the script in action as it runs. And I am glad removing the -s option for curl lets it tell me what's happening (sometimes).

I am pretty sure there are better ways of accomplishing this task of gathering lots of system information in various text files, zipping them up, and uploading to JamfPro, but this finally works for me.

[mm2270]

Yay! Glad you got it working!

[DonCascone]

Hello,
Sorry to bother on an old post.
First of all glad that you found the solution, it seems really smart and effective. Congrats!
I'm new to this whole Jamf stuff and to coding in general and I'd like to ask you Where these files go when you say "uploading to JamfPro" and how to enable the API to let this work correctly.
Do you know if the user can access in any way to the script above and "find" the API user login info?
Thank you very Much! Have a nice day :)

[AVmcclint]

The Jamf admin can find the files in the computer record > Attachments. 

[nicho2312]

Thank You @AVmcclint this is Great!

Is there a way to modify the script in a way that it will delete the files older than 1 month ad example? 

Both on Device itself and on the Device Page on JAMF portal.

Thanks Mate

[AVmcclint]

In the script, it deletes the local capture folder if it already exists.  As for removing it from the server, I don’t know. That’s why I have the zip file saved with the timestamp in the name so anyone accessing the files in the computer record will know which file is most recent.