JSS 9.90 Error 1603 attempting to upgrade from 9.81 on Windows

[BP12C]

Howdy,

I'm receiving an error when attempting to upgrade from 9.81 to 9.90 on a Windows 2012 R2 server.

The following error message is displayed in the logs.

JSS (c) (88:01) [10:08:59:855]: JSS_BACKUP_DIR=C:Program FilesJSSBackupsJSS-20160401-100859 Info 2826. Control Bitmap_1 on dialog CancelDlg extends beyond the boundaries of the dialog to the right by 7 pixels. Exception thrown by custom action: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Exception: There was an unexpected error running 'robocopy.exe "C:Program FilesJSS" "C:Program FilesJSSBackupsJSS-20160401-100859JSS" /b /copyall /R:2 /W:5 /e /XD temp backups logs work /LOG:"C:Program FilesJSSBackupsJSS-20160401-100859JSS-Backup.log"'! ---> System.TimeoutException: Running 'robocopy.exe' timed out after '60' seconds. at JAMF.JSS.InstallUtilities.Utilities.LaunchExe.Run() --- End of inner exception stack trace --- at JAMF.JSS.InstallUtilities.Utilities.LaunchExe.Run() at JAMF.JSS.InstallUtilities.Utilities.FileOperations.RoboCopy(String sourcePath, String targetPath, String args) at JAMF.JSS.InstallUtilities.Backup.JssDirectory() at JAMF.JSS.InstallUtilities.CustomActions.BackupJss(Session session) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Object arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object parameters, Object arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object parameters, CultureInfo culture) at Microsoft.Deployment.WindowsInstaller.CustomActionProxy.InvokeCustomAction(Int32 sessionHandle, String entryPoint, IntPtr remotingDelegatePtr) CustomAction JSS_InstallUtilities_CA_BackupJss returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox) Action ended 10:09:59: JSS_InstallUtilities_CA_BackupJss. Return value 3. Action ended 10:09:59: INSTALL. Return value 3. MSI (s) (84:E4) [10:09:59:952]: MainEngineThread is returning 1603

The Robocopy log does not show any errors.

Any thoughts on what could be wrong?

Can I skip the backup step during the install?

Was excited to see the new installer... for a little while at least ;)

Cheers,
Ben

[ryanstayloradob]

Please tell me you're upgrading your dev environment first. The JSS is notoriously buggy in a Windows server environment. I could be wrong, but my we had nothing but issues at my previous gig with Windows. I'm running Ubuntu and Redhat and we do not have these issues.

[BP12C]

Nope. Production server upgrade. It's a VM so we're only a snapshot away from a quick restore.

We actually had more issues with our JSS on Ubuntu than Windows. Probably more to do with our environment though (mainly proxy issues)

[skeb1ns]

I'm receiving the same error when trying to upgrade from 9.82 to 9.9 on a 2012 R2 machine. And like @bprc, our JSS host is a VM so upgrading right away isn't an issue, snapshotting all the way!

[RLR]

I had this yesterday when upgrading on our Windows VM. I just restarted the server and ran the update again. It worked so not sure what the issue was.

[cvgs]

From the log:

Running 'robocopy.exe' timed out after '60' seconds.

The installer does not wait long enough for the copy job to finish. 60 seconds is a bit on the low side.

[aamjohns]

I had this issue also. Try opening command prompt with right click>run as administrator. cd to the location of your installer msi. At the command prompt enter: msiexec /I "JSS Installer.msi" and hit enter. This is not the only way to run msiexec elevated, just giving you how I do it.

I suspect this has to do with running the installer elevated.

I would be curious if this solves the issue. The error robocopy timed out after sixty seconds is not that they are not giving it enough time, I think that it is hanging on the inability to perform elevated permission actions on files.

[MTFIDjamf]

Just ran this in our dev environment and saw the same error. Using @aamjohns instructions above were the answer. Elevating the installer seemed to do the trick.

[roiegat]

I haven't tried to upgrade our windows VM to 9.9 yet, but in the past I've found it worked better when you stop the apache service before updating. Hopefully that will make it into the documentation someday.

Since this update doesn't have too much OSX based upgrade, we'll probably wait a bit before we upgrade.

[aamjohns]

I usually stop Tomcat service before the upgrade also. Seems to be quicker than the installer stopping it.

[BP12C]

Thanks for the suggestions but still no success. :(

Here's a list of things I've tried.

• Restarted server
• Ran as Admin (you have to do this anyway... the installer bombs if you don't).
• Doubled CPU and RAM on VM.
• Manually stopped the Tomcat service.

Unfortunately, the only suggestion support has offered so far is to re-run the installer and select the "Repair" option. The repair option doesn't exist as it never reaches the install code initially.

The Robocopy script still runs after the installer times out. It seems to take about 90 seconds to complete the backup. Pretty slow for 175MB.

So unless I can skip the backup step it looks like I'm stuck until JAMF update the installer to wait for longer until it times out.

[BP12C · Accepted solution]

So I did a little snooping to see what I could remove from the install directory to speed up the backup.

The TomcatwebappsROOT directory contains about 5,000 tiny files. We don't have customisations so I removed it.

Install went through fine and recreated the ROOT directory.

I'd suggest holding off for a proper fix but if you're desperate this may be an option.

Do it at your own risk.

[sepiemoini]

@bprc I had the same issue crop up during my v9.9 upgrade this morning in my Test JSS running Windows Server 2012 R2 as well. The same steps that you highlighted (modified below in my own words) did the trick for me as well.

Issue: "Invalid Message - The message could not be parsed."
System: Windows Server 2012 R2

1. In your JSS, launch Services.msc and stop the Tomcat service (called Apache Tomcat 7.0 Tomcat7). 2. Navigate to C:Program FilesJSSTomcatwebapps and copy the ROOT folder to your Desktop. 3. Once copied, delete the ROOT folder from C:Program FilesJSSTomcatwebapps. 4. In a browser, navigate to https://my.jamfsoftware.com/products.html and expand the Show JSS installer downloads section. 5. Click on the download button next to JSS Manual Installation. 6. Navigate to the downloaded zip folder and extract all contents. 7. In the extracted folder, navigate to the JSSInstallation>JSS Components directory and copy the ROOT.war file to C:Program FilesJSSTomcatwebapps. 8. Restart the Tomcat service (called Apache Tomcat 7.0 Tomcat7) that was stopped in Step 1. 9. Verify that the ROOT has repopulated. 10. Launch a browser and navigate to your JSS URL and verify that this is starting back up. 11. Once confirmed, please run the "sudo jamf recon" command again from a machine enrolled in this JSS.

Long story short, if you're not in a hurry, wait for the issue to be permanently fixed by JAMF. @Tad is there an ETA on this being permanently fixed for Windows Server 2012 R2 users?

[skeb1ns]

Any news on a permanent fix? I now get an error that the Installer couldn't create the folder "JSS" inside the backup folder when the Installer wants to backup data. I already tried the suggested solutions.

My suggestion would be to at least give the user the choice to backup data or not. I assume that most users are running the JSS as a VM anyway and a restore is just a snapshot away.

[BP12C]

My job was closed off today as my situation has been resolved (using the ROOT folder workaround).

I don't think my helpdesk guy understood the importance of having this fixed. He said he'd "pass this up the line to help improve it further down the track"

The lack of JAMF staff feedback on here is a little concerning.

I'd suggest logging a ticket if you haven't already so they can gauge the severity/impact.

[Sandy]

@bprc, this is a user forum, and while JAMF folk occasionally chime in, if you need or expect official communication with JAMF, your TAM is the route.
Glad you were able to resolve your issue!

[nadams]

I was preparing to do this upgrade myself when I saw this thread. I reached out to my support buddy who said that an engineer is looking into the issue, but we also schedule a WebEx to run through the upgrade with him on the line in case something bombs. JAMF support is great!

[Typhoon_87]

I ran into this yesterday and opened a support case. I sent the tech a link to this thread today.

[powellbc]

I logged a ticket and got a confirmation that it was a defect.

[bpavlov]

@powellbc can you share the defect to this unofficial bug tracker:

https://jamfnation.jamfsoftware.com/featureRequest.html?id=1699#responseChild14176
To view defects peruse through here:
https://goo.gl/zTdvwT

To submit bugs:
http://goo.gl/forms/tEvpXJrZaj

[powellbc]

Sure thing @bpavlov.

[powellbc]

This issue is still not fixed with the 9.91 installer.

[lisamcray]

Holy Crow! How frustrating! I was attempting to upgrade our environment from 9.81 to 9.91 and the installer failed. Log file error: "There was an unexpected error running 'robocopy.exe "C:Program FilesJSS System.TimeoutException: Running 'robocopy.exe' timed out after '60' seconds."

I opened a ticket with JAMF . . . and my reply was to try the installer again. SMH*

Before opening the ticket, I had tried the installer multiple times. Truth of the matter is that the recommendation (to try installing again) only makes the situation worse.

What seems to be happening is if the JSS folder is too large, robocopy times out before the folder can complete copying the pre-install backup.

To compound the issue when the installer attempts the robocopy backup it creates the backup within the JSS folder that it is trying to backup: JSS_BACKUP_DIR=C:Program FilesJSSBackupsJSS-20160511-083832

After five attempted installations, my JSS folder had ballooned an additional 2.5GB in size, because of the numerous backups created each time that I ran the installer. Since there is no mechanism in the installer to delete the failed robocopy backup, the JSS folder continued to grow in size each time the installer runs.

With a JSS folder now 7GB in size I began to work on some clean up. By making a back up of the JSS folder to another volume and then manually deleting log files, back ups and the failed installer backups, I was able to reduce the JSS folder to under 1GB.

With the JSS folder now a manageable size, the installer worked fine and I was able to complete my installs.

I'm glad that I was able to figure this out on my own, and I hope very much that the next release will take this issue into consideration, as this seems to be a rather serious flaw.

[powellbc]

My JSS folder was down to less than 300MB and I am still getting the failures with Robocopy timing out. Very frustrating...

[powellbc]

Still not fixed in 9.92.

[Jens_Mansson]

Seems like migrating to another server platform is the way to avoid these bugs. Cheers for keeping the post updated.

For future troubleshooting.1603 is a "general error" speaking in msi-errors. Check logfiles in %temp% or %windir%/temp on your windows server for more accurate logs when facing this.

[powellbc]

@Jens.Mansson The "real" error is this:

There was an unexpected error running 'robocopy.exe "C:Program FilesJSS" "C:Program FilesJSSBackupsJSS-20160517-134554JSS" /b /copyall /R:2 /W:5 /e /XD temp backups logs work /LOG:"C:Program FilesJSSBackupsJSS-20160517-134554JSS-Backup.log"'! ---> System.TimeoutException: Running 'robocopy.exe' timed out after '60' seconds.

[aamjohns]

I think the issue is other data in the C:Program FilesJSS directory that makes the backup too large. Check to make sure there are not other backups there or that someone did not put files there or do something to inflate the directory.

If there are other backups there, they should be moved to another location outside the C:Program FilesJSS location.

I used to have this issue until I did two things. One, clean up that directory. I had put a backup there myself causing it to be too large, and second, making sure that I run the installer msi from an elevated command prompt. I just right click command prompt and choose 'run as administrator'. Then copy paste the path to where the installer msi is sitting. cd "<path>". Next type msiexec /I "JSS Installer.msi" /log C:some_pathinstall.log and hit Enter.

This works for me now. And if it does not, I can check the log file and see what happened. I do think it is true that if the JSS folder has too much data in it, the robocopy times out and the install will fail. In addition, if the msi is not run elevated, it will fail.

I have a log of a successful install. I don't want to post it but if someone would like assistance, let me know and I can compare your log to my log maybe help figure out what is happening.

I know that occasionally I will run into someone that does not realize you can paste into a cmd window. Just to make sure... it can be accomplished by either clicking the tiny cmd window icon in the upper left of the cmd window, going to Edit>Paste or just right click on the cmd window and it should paste in.

[powellbc]

Some comments:

1. I have removed the entire backup folder. The JSS folder in my test environment is less than 300 MB and it is timing out.
2. I have found this whole business of Jamf saying to run from an elevated command prompt odd. No other installer requires this. I once had a support technician who even advised against executing the installer from PowerShell. Sometimes I wish they would just not support Windows and force my hand than support it in a non-100% way.

Some of this is frustration of course on my part. I am disappointed that after 3 revisions the same error is occurring. I know I can do a manual install but that involves more overhead overall and down the line. I don't think it is unreasonable to expect the Windows JSS installer to work.

[aamjohns]

@powellbc Wow, that is interesting. It is still timing out. I've got the installer msi open in Orca right now trying to see what I can find.

Running elevated is required. It is a part of Windows Server. Essentially all I mean by this is you have to say I am an administrator running this. Being advised to not run an installer from PowerShell is odd. You may be referring to JAMF's recommendations, I did not say PowerShell, I just said a command prompt. And the reason I suggested that is it is easier than going through explaining other ways to add logging the install process. You can right click the MSI, create shortcut, open the shortcut, look at the command line and add log <somepath><somefile>.log or .txt to the command line and then double click your shortcut. Or open a text file and write the command line out and save it as a .bat. The command prompt is not required, I just thought it would be easiest. PowerShell is essential, in this case of running msiexec, just the command prompt. And there is not a thing wrong with it. In the end, it is all the same, msiexec is getting called to install a msi.

Would you mind logging your install and emailing it to me? aamjohns at iu dot edu?

[aamjohns]

By the way, a verbose log would be best for troubleshooting. Here is the command line option for verbose:

msiexec /i <msipath>setup.msi /l*v c:	empmsi.log

/l*v <path><filename.log> is going to produce a verbose log.

Thanks,
AJ.

[BP12C]

TBH this is a really poor effort that this hasn't been resolved yet. I'm happy to do some contract work if you want me to change a "60" to a "300"...

There seems to be some confusion here about the robocopy script. The script excludes a number of folders within the JSS folder. The temp, backups, logs, and work folders are not backed up. The backup size does not increase each time you run it. Perhaps you are getting lucky deleting backups and then successfully running the upgrade.

I still think it's due to the number of files, not size, that is causing the issue. See my earlier post for more info.

Jamf, please fix this rookie mistake!

[aamjohns]

Ben,
I am curious. If you manually stop the TomCat service before running the install, does it make any difference?

[powellbc]

I just looked in Orca too to see if I could modify the timeout but did not see that value. :(

@aamjohns I know you were not saying anything about PowerShell, just brought it up as an example of some of the skittishness I sense around Windows from Jamf support. I did not mean to make it sound like I directed it at you—my apologies.

As I was typing this response I was attempting to generate the verbose log file and on what must be my 20th attempt to install one of the 9.9x releases, it now finally worked on my test server. I did nothing differently other than use the verbose logging command line (and manually calling the msi using msiexec /i instead of just launching it from the command line). I'm happy I have this installed in test but this unreliability in the installer gives me pause before doing it in production.

[BP12C]

AJ, unfortunately, not one bit.

I spent hours on this issue when I first encountered it but I haven't looked at it since. I'll attempt the latest update tomorrow at work (it's 11pm atm) and try a few more workarounds.

[cvgs]

Well, we do not use a timeout value in our "ditto" or "asr" scripts on the Mac – so why would robocopy need one?

Basically, you say "copy this from here to there and come back if you are ready OR give me an error", and that's it. If you insist on having a timeout, use something like 1 hour upwards, because you really want to be sure before aborting a slow, but perfectly fine installation routine.

Trying to upgrade using this installer is like throwing a dice; you can't even properly test it, because the timeout might hit you only in PROD and not in DEV.

So if you haven't already, contact your TAM and upvote that issue.

[aamjohns]

Hi,
I'm happy to hear after that many attempts you finally got it installed. I appreciate the clarification. I found nothing looking at the install through Orca. AJ.

[powellbc]

I have contacted my TAM, I hope everyone else who has this issue has as well.

Also, thanks everyone, especially @aamjohns, for the feedback.

[kentmj]

Same problems here. I cracked the MSI open in Installshield but cannot find any properties to control Robocopy.

[kentmj]

Actually, if you shutdown Tomcat and back up this folder C:Program FilesJSSTomcatwebapps, then go into webapps and delete the entire contents, the install will work and it will just reinstall the contents of that folder. So far I don't see any problems.