Diagnosing kernel panics remotely

[Knighton]

I know the objective of Kernel Panics is generally remediating/preventing them, rather than gathering more information about them, but sometimes it can be helpful to see if multiple panics have the same root cause.

To that end, I built off of mm2270's Panic Counting EA three new things:

• one, a smart group scoped to machines with >0 recent kernel panics. I keep this in my dashboard to alert me to widescale issues. For example, if it goes from 20-30 in one week, to 100+ in the next week, I know something dropped recently that isn't playing nicely.

• two, a policy scoped to machines in the above Smart Group that runs once per day, and copies any .panic logs into a plaintext file in Users/Shared. (There's likely a better way to do this, but I couldn't find one, so... this workaround)

• three, another Extension Attribute that searches the plaintext log for the kernel extension reported in the panic.

• and finally, a search built on Recent Kernel Panics, and sorted by Panic Reason.

In this way, I can see how many machines are having problems, and very quickly determine if they're all having the same problem, and narrow down possible causes. WITHOUT having to reach out to users for their panic logs.

The PanicLog>>Plaintext converter borrowed heavily from Encore Technologies' Kernel Composure project. Very happy with what they're doing. Wish I could have applied it directly to this project without the additional custom coding.

Anyway, here's my python script for locating panic files and converting them to plaintext

#!/usr/bin/env python3

import argparse
import json
import re
import sys
import glob
import sys
import os

#Delete any existing logs
if os.path.exists("/Users/Shared/Panic_Log.txt"):
  os.remove("/Users/Shared/Panic_Log.txt")

#Glob for panic files. Scoped only to machines that have had at least one panic
def paths():
    return(glob.glob("/Library/Logs/DiagnosticReports/*panic*"))  

list = paths()

#Print to file. Appending in case there's multiple panics
sys.stdout = open('/Users/Shared/Panic_Log.txt', 'a')

# Parses a given log file for every match of a specific regex
def parse_log_file(log_file_path, regex):
    match_list = []
    with open(log_file_path, "r") as log_file:
        content = log_file.read().splitlines()

    for line in content:
        for match in re.finditer(regex, line, re.S):
            match_text = match.group()
            match_list.append(match_text)

    return match_list


def main():
    kernel_panic = parse_log_file(log_file_path, regex)

    initial_info = json.loads(kernel_panic[0])

    for key, value in initial_info.items():
        print(f"{key}: {value}")

    panic_json = json.loads(kernel_panic[1])

    print(panic_json["macOSPanicString"])


if __name__ == "__main__":
    regex = r"{(.*?)}"
for item in list:
    log_file_path = item

    main()

And here's the extension attribute for pulling the backtrace information:

#!/bin/sh

Panic_Reason=$(cat /Users/Shared/Panic_Log.txt | grep -A 1 "backtrace")

echo "<result>$Panic_Reason</result>"

[Knighton]

Caveat: This does not work with Big Sur at this time.

I will be working on figuring out why ASAP.