Jamf Pro is down | Jamf Admin's jobs are at Risk

[iamvinaybhatt]

Jamf pro is down from 10 hours and there is no ETA, I hope we will be getting RCA from Jamf otherwise it would be very difficult to make management understand about this blunder.

PS: The employee who did this probably getting a new job soon, I am not sure about me :( and my colleagues @tanuj 

[tanuj]

That employee can join our organization once we also get fired.

[AJPinto]

That person is a level 1 helpdesk tech, they usually don’t know much more than we do, I only opened a ticket for tracking and to request an RCA.

As far as who’s head is on the chopping block, I am wondering on that. This went down during business hours on a Tuesday, I would assume if any patching happened (and of course failed) it would have been failed automation or an accident. However, this took down all their farms which seems intentional as you would have protections in your automation so that does not happen. I’m wondering if this was a security event, and someone yanked the proverbial ethernet cable.

[andrew_nicholas]

Not to put the juju out there but that is my fear/feeling as well. 

[sdagley]

Hopefully it wasn't a Load Bearing Mac mini issue: https://sreally.com/twitter-tales-load-bearing-mac-mini/

[AJPinto]

I needed that story lol.

I have a similar story: Years ago in a previous life I worked in field services, and my employer had a no dual devices policy. I was going through cubes after hours removing unapproved secondary devices and came across a HP 6000 SFF. The inventory notes said it was a failed collection from a device upgrade several months earlier, so I just unplugged it and put it in the lab. In the time between the failed pickup and when I repo’ed the device the person who had it stood up a few production webservers on that workstation, as well as a test environment and a file share all so he did not have to deal with the hoops of our datacenter.

[howie_isaacks]

I don't think my job is at risk because Jamf has an outage. It looks like they're making progress. I hope they explain to us what happened. They owe us that much since this is severely limiting us from doing our jobs. That said, I have a lot to do that does not depend on the Jamf Pro server being available. I have been working with Jamf Pro since 2010 when it was called "Casper Suite". No platform is perfect and I have had to deal with failures several times over the years, both cloud and on-prem servers. Jamf has provided the absolute best support I have ever received form any tech vendor so I'm not quick to throw them under the bus because of an outage.

[sdagley]

@iamvinaybhatt (and anyone else interested in monitoring Jamf Cloud status info) you can subscribe to updates for the JamfStaus page:

https://status.jamf.com/

You can also install the jamfStatus notification tool (https://github.com/jamf/jamfStatus) on your Mac which will provide alerts any time the status page is updated.

[howie_isaacks]

I forgot about the notification tool. Thanks for posting. Using a client ID, I had to right click to paste in my login info. Command + V didn't work for some reason. I got worried that I would have to enter those long strings of text manually 😳

[Samstar777]

Apr 16, 13:18 UTC
Update - All Premium and APAC instances have been resolved. US and EU Standard instances are currently in progress, and some clusters have been resolved within those regions. Please check the status bar for your specific region for more information. We appreciate your patience and apologize for the inconvenience this has caused.Apr 16, 12:07 UTC

[howie_isaacks]

My production server is back online. It's sluggish since all of my Macs are checking in and sending inventories. My test Jamf Pro server is still down.

[taugust_ric]

I hope I'm not misunderstanding this post, but this is the silliest post I've ever read on Jamf Nation.  And I've been here since it was e-mail only discussions.  Unfortunately if your job is at risk because of a vendor outage, then your job is at risk at all times, and you should find a new job.  I know that's easier said then done.

Not making excuses for Jamf but their cloud environment is usually pretty stable.  If I my job was at risk every time Microsoft's services go down, I would be looking for a new job several times a year.

I do hope there is transparency in providing a cause and why it took so long to resolve.  There is a fear it's security related here as well, considering that there should be pretty good redundancy in place since it's run on top of AWS, so I'm interested in what 'infrastructure' was affected.

[kmeck518]

I wonder how many people at their org lost their jobs when AWS went down last year or CrowdStrike blew up.

[taugust_ric]

I would hope very few or none.  Mistakes happen and living in fear of identifying and reporting them, whether you work for the vendor, or work for an organization that uses a service, leads to more issues, not less.  What's more important is having a clear communication plan to mitigate risks and changes, and potential failures when they do occur.  A termination should only occur when there is a large amount of negligence or ignorance to an issue when it does occur.

[MemnochTheRed]

No one is losing their job because of a cloud outage. They happen. Just report to organization and when it is resolved.