Problems with good scripts failing (exit code: 127)

[yellow]

This has been happening off and on and I'm not entirely sure why... a perfectly good script, meaning it will run locally from the command line, will not run as a policy. This is a script that worked fine prior to 9.8 upgrade (we are running 9.8.1). Note, the script continues no reference to the jamf binary.

The error I get is:

Running script RebootNotification_v1.0.sh...
Script exit code: 127
Script result: /bin/sh: /Library/Application Support/JAMF/tmp/RebootNotification_v1.0.sh: No such file or directory

Running it via Casper Remote gives me:

Sending Wake On LAN command...
Opening SSH Connection to xx.xx.xx.xx..
Authenticating...
Successfully authenticated.
Verifying Computer's Identity...
The MAC Address has been verified.
Checking Operating System Version...
Running Mac OS X 10.10.5 (14F27)
Verifying /usr/local/jamf/bin/jamf...
/usr/local/jamf/bin/jamf is current (9.81)
Verifying /usr/sbin/jamf...
/usr/sbin/jamf does not exist.
Verifying /Library/Preferences/com.jamfsoftware.jamf.plist...
Preparing Policy...
Executing Policy 2015-10-09 at 1:18 PM | yellow | 1 Computer...
Mounting Casper to /Volumes/CasperShare...
Running script RebootNotification_v1.0.sh...
Script exit code: 127
Script result: /bin/sh: /Library/Application Support/JAMF/tmp/RebootNotification_v1.0.sh: No such file or directory
Submitting log to https://casper.domain.com:8443/
Finished.

Set up as a ongoing policy.
Running it as a policy fails.
Running it from command line as a -trigger worked ONCE. After that, always fails.

Run as a trigger:

ThisMac:Source yellow$ sudo jamf policy -trigger "runRebootNotification"
Password:
Checking for policies triggered by "runRebootNotification"...
Executing Policy [BETA] Reboot Notification [BETA]...
    Mounting Casper to /Volumes/CasperShare...
Running script RebootNotification_v1.0.sh...
Script exit code: 127
Script result: /bin/sh: /Library/Application Support/JAMF/tmp/RebootNotification_v1.0.sh: No such file or directory

And I do have a tmp folder in my JAMF Application Support folder.

I'm seeing this more and more and it's becoming a problem.

Anyone else seeing this?

(I'd pay no attention to the obvious bug in Casper Remote, in checking for a binary that is no longer there).

[mm2270]

It might help to post the script in full, or a sanitized version if need be, so we can look at that as well. Maybe something in the script would be causing an issue when its run one way out of Casper versus another.

[yellow]

Slight edit to the above to include more info.

Also, before I'd been googling for error 127 + casper.. never without.. but (shame on me) just found this:

"Value 127 is returned by /bin/sh when the given command is not found within your PATH system variable and it is not a built-in shell command. In other words, the system doesn't understand your command, because it doesn't know where to find the binary you're trying to call."

Technically these are bash shell scripts, but nothing special, I know that sh runs them fine locally.

[yellow]

Be gentle, I'm a terrible scripter. :-)

The Script (sanitized a bit):

#!/bin/bash

#  RebootNotification_v1.0.sh

checkIt=`uptime | grep -c "day"`
jHelper="/Library/Application Support/JAMF/bin/jamfHelper.app/Contents/MacOS/jamfHelper"
usIcon="/Library/Application Support/JAMF/Us/SomeLogo.png"

if [ $checkIt != 0 ]; then

myUptime=`/usr/bin/uptime | /usr/bin/awk '/days/ { print $3 }'`

    if [ -e "$usIcon" ]; then

    "$jHelper" -windowType "utility" -title "Extreme Uptime Notification" -description "This Mac has an uptime of ~ ${myUptime} days. You may want to consider rebooting your Mac at the earliest opportunity to regain lost functionality." -icon "$usIcon" -button1 "OK"

 else

    "$jHelper" -windowType "utility" -title "Extreme Uptime Notification" -description "This Mac has an uptime of ~ ${myUptime} days. You may want to consider rebooting your Mac at the earliest opportunity to regain lost functionality." -button1 "OK"
    fi
else
    echo "Uptime Less than 1 day, exiting now."
    exit 1
fi

exit 0

[mm2270]

OK, but based on that error description you found, one of the commands you're calling in it is not being recognized by the shell, so hence why I said you may want to post the script contents itself.
It was already noted on other threads that launchd does not use the same PATH environment variables as the shell. Its mostly the same but may not include all the same paths. Therefore, something in the script isn't being recognized when its running by way of the natural check-in process, since, that's called by the JAMF LaunchDaemon.

[yellow]

Yes, script is above.

But it's almost more like the script isn't being copied to tmp when I look at a verbose output from a policy update:

ThisMac:JAMF yellow$ sudo jamf policy -verbose
 verbose: JAMF binary already symlinked
 verbose: JAMF agent already symlinked
 verbose: Checking for an existing instance of this application...
Checking for policies triggered by "recurring check-in"...
 verbose: Checking for active ethernet connection...
 verbose: Active ethernet connection found...
 verbose: The Management Framework Settings are up to date.
 verbose: Found 1 matching policies.
 verbose: Removing any cached policies for this trigger.
 verbose: Parsing servers...
 verbose: Parsing Policy [BETA] Reboot Notification [BETA] (4371)...
 verbose: Parsing Policy [BETA] Reboot Notification [BETA] (4371)...
Executing Policy [BETA] Reboot Notification [BETA]...
    Mounting Casper (casper.whatever.blah) to /Volumes/CasperShare...
 verbose: Result of mount attempt: Password:
 verbose: Result code of mount attempt: 0
 verbose: Copying script to temp directory...
 verbose: Determining script type...
Running script RebootNotification_v2.0.sh...
Script exit code: 127
Script result: /bin/sh: /Library/Application Support/JAMF/tmp/RebootNotification_v2.0.sh: No such file or directory

 verbose: Removing local copy...
Submitting log to https://casper:8443/
Unmounting file server...
 verbose: Attempting to remove Kerberos ticket for casperinstall

Also, since there's nothing special in there I converted it back to a shell script and the above error reflects the same issue. So it doesn't matter whether it's a bash script or a sh script, apparently.

[yellow]

And I think I might know sort of why....

So I ran it and kept looking at the tmp dir and this is what shows up:

bash-3.2# lf
total 0
drwx------   2 root  admin    68B Oct  9 13:56 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
bash-3.2# lf
total 0
drwx------   2 root  admin    68B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
bash-3.2# lf
total 8
drwx------   3 root  admin   102B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
-rwx------   1 root  admin   316B Oct  9 13:57 3D76953F-DEC3-4F51-A210-D501FE2EB0CA*
bash-3.2# lf
total 0
drwx------   2 root  admin    68B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
bash-3.2# lf
total 0
drwx------   3 root  admin   102B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
-rw-r--r--   1 root  admin     0B Oct  9 13:57 69315.tmp
bash-3.2# lf
total 0
drwx------   3 root  admin   102B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
-rw-r--r--   1 root  admin     0B Oct  9 13:57 69315.tmp
bash-3.2# lf
total 0
drwx------   3 root  admin   102B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:56 ../
-rw-r--r--   1 root  admin     0B Oct  9 13:57 69315.tmp
bash-3.2# lf
total 0
drwx------   2 root  admin    68B Oct  9 13:57 ./
drwxr-xr-x  15 root  wheel   510B Oct  9 13:57 ../

The error is correct, that script isn't found to run, because nothing with that name ever appears there.

[mm2270]

Yeah, that's odd. Offhand I don't see anything glaring in it that would throw an error. The only 2 binaries I see you calling are uptime and grep, both of which are in a path a LaunchDaemon would know about.
The only line I would change for the purpose of troubleshooting is this one:

if [ $checkIt != 0 ]; then

to this:

if [[ "$checkIt" != "0" ]]; then

I've seen sometimes that doing checks like that requires the double bracket method, but I don't always have an explanation for why it can cause errors when doing it the way you have. It really shouldn't. But try that and see what happens.
But you may also be right. Maybe the script just isn't being downloaded at all, which would point to other issues entirely.

[mm2270]

Ah, I posted before I saw your last response. OK, so its not making it down into that tmp folder at all. Well, that explains the error. You'll just have to figure out why. Is your JSS set up so scripts are contained in the database, or are they physical files on a distribution point?

[yellow]

I used to have it with double brackets, I'll go back to that. Just to check my sanity, this script works no problem:

#!/bin/sh

#  HelloWorldTest.sh
#  
#

/bin/echo "Hello World"
/bin/echo "Hello Solar System"
/bin/echo "Hello Galaxy"
/bin/echo "Hello Universe"

exit 0

Our scripts are physical files on a DP.

[bpavlov]

May sound silly but try deleting the script, save and close Casper Admin, then re-add the script, save and close Casper Admin.

[yellow]

Not silly at all... though I change versions so it creates a new file locally and then I copy that up to Casper Admin, so it's a new file every time.

[yellow]

I'm totally stumped. Now it works. On 3 computers where I was testing and it failed 100% of the time on all of them, now it's working on all of them 100% of the time. What changes have I made? I took out whitespace in the bash script... which should make no difference at all. I also moved the "checkIt" declaration down below the other variables. Also shouldn't have made a difference. I also changed it from "After" to "Before". Which also shouldn't make a difference. Especially since I moved it BACK to being an "After" which made no difference. <brain melt>

#!/bin/bash

jHelper="/Library/Application Support/JAMF/bin/jamfHelper.app/Contents/MacOS/jamfHelper"
usIcon="/Library/Application Support/JAMF/Us/SomePic.png"
checkIt=`/usr/bin/uptime | /usr/bin/grep -c "day"`
echo "CheckIt = $checkIt"
if [[ $checkIt != 0 ]]; then
    myUptime=`/usr/bin/uptime | /usr/bin/awk '/days/ { print $3 }'`
    sleep 1
    if [[ -f "$usIcon" ]]; then
    sleep 1
    echo "with logo = $myUptime"
    "$jHelper" -windowType "utility" -title "Extreme Uptime Notification" -description "This Mac has an uptime of ~ ${myUptime} days. You may want to consider rebooting your Mac at the earliest opportunity to regain lost functionality." -icon "$usIcon" -button1 "OK"
    else
    sleep 1
    echo "without logo = $myUptime"
    "$jHelper" -windowType "utility" -title "Extreme Uptime Notification" -description "This Mac has an uptime of ~ ${myUptime} days. You may want to consider rebooting your Mac at the earliest opportunity to regain lost functionality." -button1 "OK"
    fi
else
    sleep 1
    echo "Uptime Less than 1 day, exiting now."
fi
exit 0

[yellow]

OK, now I'm just pissed... this makes NO sense.

I tried to recreate what I did to "make it work" and cannot. I took the same file that was broken and did the same things that "fixed" it. The broken one is still broken, and the "fixed" one is still working. AND BOTH SCRIPTS ARE THE SAME!!

yellow$ diff -s /Volumes/CasperShare/Scripts/RebootNo.sh /Volumes/CasperShare/Scripts/RebootMe.sh 
Files /Volumes/CasperShare/Scripts/RebootNo.sh and /Volumes/CasperShare/Scripts/RebootMe.sh are identical

and yet, when I have both scoped to my Mac and run a policy update, one fails, one works.

yellow$ sudo jamf policy -verbose
 verbose: JAMF binary already symlinked
 verbose: JAMF agent already symlinked
 verbose: Checking for an existing instance of this application...
Checking for policies triggered by "recurring check-in"...
 verbose: Checking for active ethernet connection...
 verbose: Active ethernet connection found...
 verbose: The Management Framework Settings are up to date.
 verbose: Found 1 matching policies.
 verbose: Removing any cached policies for this trigger.
 verbose: Parsing servers...
 verbose: Parsing Policy [BETA] Reboot Test (4377)...
 verbose: Parsing Policy [BETA] Reboot Test (4377)...
Executing Policy [BETA] Reboot Test...
 verbose: Will mount at /Volumes/CasperShare
    Mounting Casper (casper) to /Volumes/CasperShare...
 verbose: Result of mount attempt: Password:
 verbose: Result code of mount attempt: 0
 verbose: Copying script to temp directory...
 verbose: Determining script type...
Running script RebootNo.sh...
Script exit code: 127
Script result: /bin/sh: /Library/Application Support/JAMF/tmp/RebootNo.sh: No such file or directory

 verbose: Removing local copy...
 verbose: Copying script to temp directory...
 verbose: Determining script type...
Running script RebootMe.sh...
Script exit code: 0
Script result: CheckIt = 0
Uptime Less than 1 day, exiting now.

 verbose: Removing local copy...
Submitting log to https://casper:8443/
Unmounting file server...
 verbose: Attempting to remove Kerberos ticket for casperinstall

That doesn't even make any sense to me at all.
Can you detect the keening wail of panic in my words?!

[donmontalvo]

Interesting, we are getting exit 127 on simple postinstall scripts in flat PKG packages.

The flat PKG works fine when invoked in Finder.

JSS 9.65, OS X 10.10.5 (haven't tried other versions of either).

Don

[calumhunter]

am i missing something or is the error just telling you that this file doesn't exist?

/Library/Application Support/JAMF/tmp/RebootNo.sh

If it doesn't exist, you need to find out why not.

What creates that file?

What calls that file?

[yellow]

That is what it's implying, that it doesn't exist, but I don't know why. That's why I'm posting here ;)

The file is a shell script. The jamf binary should be downloading it and running it, but it's not. The worst part is that there are 2 shell scripts that are exactly the same, when called from the same Policy, one works, one does not.

[mm2270]

Hey, the /Library/Application Support/JAMF/tmp/ directory is created by the Casper framework, not by the OP. Its where the scripts get downloaded to and run from. I have the /tmp/ directory on my Mac as well.
So here's something odd I just noticed. I decided to peek inside that folder and I see a dozen or more script files that have run in previous policies just sitting around in there, as well as a bunch of files ending in .tmp. I would think the scripts should be getting deleted after they are run, at least that's what I thought.
This leads me to think that somehow the jamf binary might be leaving failed scripts in that folder.

[tlarkin]

Just a wild guess, but if your scripts still live on DPs and not in the DB, the JAMF Framework could be expecting the script to actually live in the DB. The binary will do a GET request against that script object when it needs to run it and then payload the contents of that object into the shell.

It looks like you are mounting a file share, but then the script expects to be in the /tmp folder in the JAMF app support folder. Just for grins, could we try copy and pasting that script directly into the JSS web UI? That would then put it in the DB and we could see if the framework expects it there.

Thanks,
Tom

[yellow]

But why do 2 scripts, called in the same policy, one fails and one succeeds. They are EXACTLY the same script, just different names.

SomeMac:Source yellow$ diff -s RebootNo.sh RebootMe.sh 
Files RebootNo.sh and RebootMe.sh are identical
SomeMac:Source yellow$ md5 RebootNo.sh 
MD5 (RebootNo.sh) = f13fc1fad33dc635f745c0c22474ce96
SomeMac:Source yellow$ md5 RebootMe.sh 
MD5 (RebootMe.sh) = f13fc1fad33dc635f745c0c22474ce96
SomeMac:Source yellow$

[Fveja]

Yellow,

It seems your JSS is broken, this is why one works and the other doesn't. Whoever set it up should be fired.

Please do what @tlarkin suggested.

Florin

[calumhunter]

^--- Best post ever

[yellow]

@Fveja -- I think that joker already quit. :-)

[tlarkin]

@yellow

Sorry for my delayed response we had our national user conference this week and I got pretty busy. I do not think the issue has anything to do with the scripts. I think the issue lies in a configuration of the JSS. In version 9 we have allowed customers to store their scripts in the JSS database. When this happens a bit is flipped in the database telling the framework we have migrated the scripts from a classic file share to the database. From this point the framework should do a GET request against that script object in the DB when it needs to run it, and that object is downloaded to the tmp folder in /Libray/Application Support/JAMF/tmp which will contain all the XML elements for that object and the actual script contents. Which are then payloaded to the shell and executed.

It is my best guess that the framework thinks you have migrated to the DB and is trying to pull a script from that folder location. This is why I suggested actually putting the script into the JSS to see if it worked.

You will see this flag in your database when you migrate your scripts from a file share into the database via Casper Admin:

 select script_migration_complete from distribution_technology_settings;
+---------------------------+
| script_migration_complete |
+---------------------------+
|                         1 |
+---------------------------+

Does this make sense and could you try tossing that script into the JSS and run that policy? Also I believe we pull objects down by their UUID not their script name because scripts could technically have the same name as names are not unique I do not think. So you seeing "3D76953F-DEC3-4F51-A210-D501FE2EB0CA" in the file system.

thanks,
Tom

[Chris]

I'm currently having the same problem, but only with one DP (Fileshare DP, AFP, Scripts not migrated).
Packages work fine, but all Scripts fail with 127.
Haven't figured it out yet...

Edit: I had a messed up ACL on the Scripts folder

[powellbc]

I have a basic script with a single command:

npm install -g cordova

This script works fine when run locally (he correct package manager is installed already). But like the OP it fails when run through Self Service/the framework with the "command not found"/return code 127. I have both added it natively in the Console and uploaded a separate .sh file. Neither work.

Any ideas? Very frustrating....

[mm2270]

@powellbc I haven't heard of npm before, so I looked it up (https://docs.npmjs.com/getting-started/what-is-npm)
It looks like a custom binary or something to work with javascript. So the question that comes to mind for me is, does it exist on the Mac the script is trying to run on? When we see 'command not found' in script error output, it usually means exactly what its saying. The shell can't locate the command you're calling. I think in this case including the full path to npm may resolve the issue. Remember, using shortened forms of calling binaries, like cp, jamf, awk, etc usually work OK, but in some cases you need to use full paths - /bin/cp, /usr/local/bin/jamf, /usr/bin/awk, etc.
If you can pull it up on the Mac in question with which npm, then use the full path that the command returns in your script and see if that helps fix it.

[powellbc]

@mm2270 As soon as I saw this email come in I was looking at where it was running from. I ended up trying the wrong path but the "which" command got the correct one. The script is now working correctly. Thanks!

[scottb]

Are people still seeing this issue? I am on some of mine with the simple script we use to pull the logged in username.

#!/bin/sh

/usr/local/bin/jamf recon -endUsername $3

Works fine on most - running as an actual script. On maybe 8% though, we get:

/usr/local/bin/jamf: No such file or directory
Error running script: return code was 127.

[yellow]

When I get that, it's universally on devices that are super-old... e.g., running 10.6.8 or before. On those devices, the jamf binary still lives at /usr/sbin/jamf.

You could create a symlink to the old binary in the new location on those old devices. While this is not the best idea in the world, these versions of the OS are unsupported by Apple and according to our policy unsupported by us (at least that's what we tell ourselves).

sudo ln -s /usr/sbin/jamf /usr/local/bin/jamf

But the Error 127... I never did figure out what the root of that was. It wasn't the script. It wasn't the device. It just seemed to be a busted script/package on the DP. TestA would fail. TestB would succeed. TestA and TestB were exactly the same.

[scottb]

Thanks @yellow. I don't care about those old Macs either and we have the same "don't support" thing here LOL.
I'm pretty sure that the failures were not on those old Macs only though, and like you, the fail/success seemed to make no sense...

[sean]

You could run this to see where scripts are running. It will output into the system.log file

#!/bin/bash

logger -t "com.jamf.test" $0

exit 0

[yellow]

It's happening again on a script that had previously worked. Grr...