NetSUS 5.0.1 not showing latest updates

[carlo_anselmi]

Hello, NetSUS 5.01 here does not show the most recent macOS updates (that is Catalina 10.15.7 and others security updates for Mojave and High Sierra "005" released a couple of days ago)
Do we need to manually change some catalog? If so, how? Any help will be greatly appreciated
Many thanks Have a great weekend

PS
Also installinstallmacos.py now reports UNKNOWN builds and choosing option "6"

 #      ProductID    Version    Build   Post Date  Title
 1      001-15219    10.15.5  19F2200  2020-06-15  macOS Catalina
 2      001-04366    10.15.4  19E2269  2020-05-04  macOS Catalina
 3      041-91758    10.13.6    17G66  2019-10-19  macOS High Sierra
 4      041-88800    10.14.4  18E2034  2019-10-23  macOS Mojave
 5      061-26589    10.14.6   18G103  2019-10-14  macOS Mojave
 6      001-51042    10.15.7  UNKNOWN  2020-09-24  macOS Catalina
 7      001-36735    10.15.6  UNKNOWN  2020-08-06  macOS Catalina
 8      061-86291    10.15.3  19D2064  2020-03-23  macOS Catalina
 9      041-90855    10.13.5   17F66a  2019-10-23  Install macOS High Sierra Beta
10      061-26578    10.14.5  18F2059  2019-10-14  macOS Mojave
11      001-36801    10.15.6  UNKNOWN  2020-08-12  macOS Catalina

here gives this error

Downloading https://swdist.apple.com/content/downloads/20/55/001-51042-A_2EJTJOSUC2/rsvf13iphg5lvcqcysqcarv8cvddq8igek/RecoveryHDMetaDmg.pkm...
Traceback (most recent call last):
  File "/Users/carlo/Desktop/installinstallmacos.py", line 626, in <module>
    main()
  File "/Users/carlo/Desktop/installinstallmacos.py", line 580, in main
    product_info[product_id]['BUILD']))
KeyError: u'BUILD'

Also reported here

[sdagley]

@carlo.anselmi Per MacMule's post in the #netsus channel of the MacAdmins Slack: https://github.com/wdas/reposado/commit/e216c9cb5d54244f6bcadbd6ffebea15c3d34ef5

[dstranathan]

Can Reposado be updated/patched under NetSUS 5.0.1, or do NetSUS customers need to wait for a NetSUS patch?

Example: Can we replace /var/lib/reposado/repo_sync with new/updated code from github on-the-fly?

BTW: According to Reposado logs, the following (16) .dist files appear to be affected:

/srv/SUS/html/content/downloads/25/55/001-39950/9am5cun5prgofnn283uny2ua2rimq4yfxb/001-39950.English.dist
/srv/SUS/html/content/downloads/61/62/001-48327-A_1I6KCAFXDF/yc5m3tmeun8wa93xteu3zprtn2b6tpy03i/001-48327.English.dist
/srv/SUS/html/content/downloads/39/59/001-48382-A_W8PZ5RCN32/6s5mt2degs2ba8xu7g7k4n8d97r8jsde0i/001-48382.English.dist
/srv/SUS/html/content/downloads/31/36/001-48424-A_V8C0CQFUZ2/n8h8dimz2wg1pxo3d97a5iz3ckae4nlrhu/001-48424.English.dist
/srv/SUS/html/content/downloads/05/21/001-48424-A_LWC33T31W9/390vsepb43ifjj55iz4cj9wv8uv5e5ml4l/001-48424.English.dist
/srv/SUS/html/content/downloads/58/14/001-51038-A_EJKE9WBV2D/h7zrh6zf0kv13lyhe9i8qmygds68gicgs5/001-51038.English.dist
/srv/SUS/html/content/downloads/20/55/001-51042-A_2EJTJOSUC2/rsvf13iphg5lvcqcysqcarv8cvddq8igek/001-51042.English.dist
/srv/SUS/html/content/downloads/25/55/001-39950/9am5cun5prgofnn283uny2ua2rimq4yfxb/001-39950.English.dist
/srv/SUS/html/content/downloads/61/62/001-48327-A_1I6KCAFXDF/yc5m3tmeun8wa93xteu3zprtn2b6tpy03i/001-48327.English.dist
/srv/SUS/html/content/downloads/39/59/001-48382-A_W8PZ5RCN32/6s5mt2degs2ba8xu7g7k4n8d97r8jsde0i/001-48382.English.dist
/srv/SUS/html/content/downloads/31/36/001-48424-A_V8C0CQFUZ2/n8h8dimz2wg1pxo3d97a5iz3ckae4nlrhu/001-48424.English.dist
/srv/SUS/html/content/downloads/05/21/001-48424-A_LWC33T31W9/390vsepb43ifjj55iz4cj9wv8uv5e5ml4l/001-48424.English.dist
/srv/SUS/html/content/downloads/13/47/001-51032-A_8AO8HZ42NP/j3mzasosdi7kc008ejfzeg4sh27wlnn3g8/001-51032.English.dist
/srv/SUS/html/content/downloads/51/25/001-51037-A_TXT0LL777V/7h4qoc2a3fmodj7kewu318gcqg9ox7uahc/001-51037.English.dist
/srv/SUS/html/content/downloads/58/14/001-51038-A_EJKE9WBV2D/h7zrh6zf0kv13lyhe9i8qmygds68gicgs5/001-51038.English.dist
/srv/SUS/html/content/downloads/20/55/001-51042-A_2EJTJOSUC2/rsvf13iphg5lvcqcysqcarv8cvddq8igek/001-51042.English.dist

[adhuston]

I made the change to Reposado referenced in the Github code as well, but this doesn't appear to have resolved the issue. I'm seeing the same error messages in the logs and am not seeing the latest updates listed. Is there anything further we need to adjust to make this work?

[dstranathan]

@adhuston What do you change exactly? specific lines of code - or did you replace the whole repo_sync script? Im asking because the last commit to the entire repo_sync python script was just ~3 days ago.

I think that once you make the required changes, you need to locate the .dist files that are appearing with XML errors in the log (/var/log/reposado_sync.log) and delete them - then run repo_sync again. See https://groups.google.com/g/reposado/c/QKXRxtyxk0Q

[adhuston]

Hi @dstranathan,

Thanks for the suggestions! I just made the change to the repo_sync file, but didn't make an other changes. I'll make those changes and see if that fixes my issue.

Andy

[adhuston]

Thanks @dstranathan! That did the trick. I'm back in business. I really appreciate the help!

[Nix4Life]

+1 on @dstranathan post. I Just did it, and it worked straight away. Please mark as solution

[dstranathan]

I dont have git installed on my NetSUS server, so I just backed up the current repo_sync python script and then replaced it with the newest repo_sync code on the Reposado site (main branch). Then I deleted the "bad" .dist files and ran repo_sync manually from cli. Worked great. Im pulling the 10.15.7 update on clients now.

One odd side-effect: I first ran the repo-sync via the cli which worked fine, but once I ran a sync from the NetSUS GUI, the macOS 10.15.7 update now appears as "deprecated" (but Im still able to download it etc).

[stephenb]

Yo. Got a pull request on NetSUS for the fix outlined here. In addition to this, it may be easiest to just SSH to the NetSUS server and run the following:

for DIST_FILE in $(find /srv/SUS/html/content/downloads/ -name "*".dist); do rm -rf $DIST_FILE; done

This should find all *.dist files and remove them; sure it's a bit nuclear, but the next sync will re-download all of the dist files. YMMV though!

[Scott_Watkins]

Posted: 10/1/2020 at 5:14 AM CDT by stephenb Yo. Got a pull request on NetSUS for the fix outlined here. In addition to this, it may be easiest to just SSH to the NetSUS server and run the following: for DIST_FILE in $(find /srv/SUS/html/content/downloads/ -name "".dist); do rm -rf $DIST_FILE; done This should find all .dist files and remove them; sure it's a bit nuclear, but the next sync will re-download all of the dist files. YMMV though!

This solved it for me. I had already updated the reposado files at the start of the week. 10.15.7 was downloaded but none of my clients could see it. Nuked the dist files and then re-synced. Issue solved.

[carlo_anselmi]

@stephenb @dstranathan Many thanks for your help. I am wondering if I should wait for an updated release of NetSUS... ...but also how long will this take?

[dstranathan]

@carlo.anselmi I considered that option too, but the NetSUS (as a stand-alone 'product stack') doesn't get updated/patched very often (at least in my personal experience).

[carlo_anselmi]

@dstranathan Many thanks, NetSUS up and running here too!

[MikeF]

So i got the updated repo_sync file and I ran the command listed to remove the .dist files.

Problem I have now is now it will not download any files

Oct 07 13:54:19 WARNING: did not add product 041-89122 to catalog index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.apple because it has not been downloaded.
Oct 07 13:54:19 WARNING: did not add product 041-89121 to catalog index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.apple because it has not been downloaded.
Oct 07 13:54:19 WARNING: did not add product 041-89120 to catalog index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog.apple because it has not been downloaded.

This is most of the log. Not sure what I did to cause this. Any ideas how to fix this.

Thanks

[afarnsworth]

@MikeF Instead of running the dist command I nuked the entire /srv/SUS/html/content/downloads folder and ran a manual sync and it downloaded everything again and resolved my issue.

Also, if you haven't bounced your server after making the repo_sync changes I would try that as well.

[MikeF]

I blew away the downloads folder and rebooted. Still having the same issue. I re installed netsus and went back and changed that one file again for the compressed files. Still the same. Maybe i'll get lucky and it will sync up overnight. I am not counting on it though.

[MikeF]

What a pain over night it resynced but still did not get Catalina 10.15.7 I went in and this time i just deleted the files @dstranathan showed. I then deleted the purged files and resynced. Now all is working properly.

[ccbell]

I've tried the following:

1. Edit the /var/lib/reposado/repo_sync script as indicated (a comparison with a fresh copy from GitHub confirms this is the only change).

--- repo_sync.20201014.bad      2019-12-02 10:02:01.772874612 -0600
+++ repo_sync   2020-10-14 10:58:37.200501872 -0500
@@ -289,6 +289,7 @@
     # curl so we avoid the problem of URLs showing up in a process listing
     try:
         fileobj = open(curldirectivepath, mode='w')
+        print >> fileobj, 'compress'       # accept and handle compressed files
         print >> fileobj, 'silent'         # no progress meter
         print >> fileobj, 'show-error'     # print error msg to stderr
         print >> fileobj, 'no-buffer'      # don't buffer output

1. Identified the files in /var/log/reposado_sync.log that are bad and removing them.

2. Rebooting the server (I'm not sure why we're doing this step since the repo sync is a nightly cron job so there's nothing to restart).

3. Running /var/appliance/sus_sync.py as root per root's crontab to resync.

This resulted in no change in behavior. The same 16 "bad" files were re-downloaded and erred out.

1. Deleting the /var/SUS/html/content/download directory

2. Reboot the server.

3. Running the /var/appliance/sus_sync.py as root.

This downloads 596 dist files, now all of which err as being "Invalid XML".

1. Deleting the /var/SUS/html/content/download directory

2. Reboot the server.

3. Run /var/lib/reposado/repo_sync directly as root with no command line arguments.

This also downloads 596 dist files, now all of which err as being "Invalid XML".

I'm not sure what to try next. Help?

Can someone who resolved this please post a step by step method of procedure for fixing this? Something detailed (directory paths, diffs, commands run with full path, command line arguments, and invoking user, etc).

Thank you!

[dan-snelson]

@ccbell We're running an OVA of NetSUS 5.0.1 and the following seems to have helped:

sudo cp -v /var/lib/reposado/repo_sync{,-backup-`date '+%Y-%m-%d-%H%M%S'`}

sudo nano /var/lib/reposado/repo_sync

Then, in nano, use Control - and jump to line 291 and copy-paste the following variation of what others have used:

        print >> fileobj, 'compressed'     # accept and handle compressed files

Write Out the edited file with Control O and pressing Return, then, press Control X to exit nano.

Then use @afarnsworth's nuclear option: sudo rm -Rv /srv/SUS/html/content/downloads

Finally, kick-off and monitor a sync:
sudo /var/lib/reposado/repo_sync & tail -f /var/log/reposado_sync.log

[ccbell]

@dan-snelson That's interesting! What you have is different from what's on the Reposado github.

This is a diff between my edited file (the one with the datestamp) and the one pulled right out off of github today:

$ pwd
/var/lib/reposado
$ diff -u repo_sync.20201014 repo_sync
$

Here's the universal format diff of the prior 5.0.1 version of the file, compared against what was pulled from github:

$ diff -u repo_sync.20201014.bad repo_sync
--- repo_sync.20201014.bad      2019-12-02 10:02:01.772874612 -0600
+++ repo_sync   2020-10-14 10:58:37.200501872 -0500
@@ -289,6 +289,7 @@
     # curl so we avoid the problem of URLs showing up in a process listing
     try:
         fileobj = open(curldirectivepath, mode='w')
+        print >> fileobj, 'compress'       # accept and handle compressed files
         print >> fileobj, 'silent'         # no progress meter
         print >> fileobj, 'show-error'     # print error msg to stderr
         print >> fileobj, 'no-buffer'      # don't buffer output
$

But you're saying that the word "compress" should really be "compressed". Is that right?

[dan-snelson]

Correct, @ccbell, compressed vs. compress.

As always, YMMV.

[MikeF]

OK I set mine up last week using compress and mine started working properly.

[ccbell]

Following Dan's advice ('compressed' vs. 'compress') is what worked for me. Perhaps there are some python versioning shenanigans going on? I'm running the application on Ubuntu Server 18.04.5 LTS.

$ lsb_release -idrc
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic
$ /usr/bin/env python --version
Python 2.7.17
$

[dan-snelson]

Glad to hear it's working, @ccbell.

Specs from the NetSUSLP_5.0.1.ova (which is now outdated and EoL):

$ lsb_release -idrc
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:    16.04
Codename:   xenial

$ /usr/bin/env python --version
Python 2.7.12

[ivanlovisi]

@dan-snelson this is the solution for me!
Please mark as solution!
Thanks

[Dalmatian]

Hi @dan-snelson

I followed the steps that you provided, including adding one line with 'compressed' in repo_sync, also did sudo rm -Rv /srv/SUS/html/content/downloads' and run repo_sync in the end to pull/sync from Apple. but there are still some .dist files are affected. So far I could only see updates before 2020-11-05 as attached

There should be an 10.5.7 supplemental update as well these days, but it couldn't load in my Netsus 5.0.1. Not sure whether other folks could load full list in Nov.
@ccbell @ivanlovisi Are you guys able to see updates in Nov?

[dan-snelson]

@Dalmatian You may have better results using compress per e216c9cb5d54244f6bcadbd6ffebea15c3d34ef5.

[Dalmatian]

@dan-snelson Ah, just took a look at our SUS, it took overnight to populate this 10.15.7 supplemental update on the list. It's working. Thanks you