Product Office Hours #4 - Jamf's Engineering AI: Nighthawk - A peek behind the curtain *Q&A in thread*

Product Office Hours recap: Nighthawk Q&A
This week’s Product Office Hours covered Nighthawk, Jamf's internal AI system for engineering work, with Akash Kamath (CTO), Martin Barnard (Senior Product Manager), and Justin Wilke (engineer, and Nighthawk's original builder). Here's every question asked during the session, for anyone who couldn't make it live.

Since each Jamf instance gets its own space in our AWS deployments, how does Nighthawk retain privacy when analyzing multiple customers' data?
Nighthawk doesn't interact with customer pods or customer data unless it's explicitly fed that information directly in a ticket. Information it works with is always filtered, and it doesn't have unfiltered access to customer pods across Jamf's AWS accounts. Network and AWS boundaries are in place specifically to prevent any agent from reaching customer data that isn't explicitly provided.

What can you do with this at scale? Why not just feed all your product issues at once and let it run? Is that a human factor for reviewing, or a limitation on AI capacity? Right now, the binding gate is genuinely humans: every product issue gets auto-triaged and then sits waiting for review. From a pure technical standpoint, there's nothing stopping Nighthawk from scaling out. The real issue is that not all issues are equal. Something like 100 spelling errors could run in parallel easily. But bugs spanning multiple components and workflows don't always get resolved correctly first time, so scaling quality (not just throughput) is the next problem to solve. It's also a cultural adoption curve: as more people get comfortable using Nighthawk for their own work, that frees up time to review the pull requests already waiting.

Follow-up: does the reviewer need a higher skill level to validate AI-generated code, or less? It depends. If acceptance criteria are written clearly, validation is often straightforward: did it solve the customer's problem, and did it not introduce security issues. Whether AI output is "slop" or genuinely good code usually comes down to how well the constraints and documentation feeding it are defined. Teams with strong, well-documented practices tend to get good code out of AI; teams without that groundwork need more manual tweaking. One example shared live: a ticket with no description and no acceptance criteria (just "dismiss all notifications") needed five rounds of triage and five rounds of iteration, versus a clear, well-specified ticket that might need none. The takeaway for anyone reporting a product issue: the more detail you give, the faster the team gets to root cause.

With 900+ issues evaluated, how are code change collisions handled when production code is updated? If a branch is off main and main gets updated, does the branch get re-evaluated for new conflicts? Standard engineering controls still apply here. Git raises merge conflicts as normal, and that gets resubmitted to Nighthawk with instructions to merge in main and resolve what's come up. It may ask the engineer a few clarifying questions on the conflict, then goes back through the full review and testing process again.

How can end users utilize APIs and Jamf's internal AI model to review their own issues? How might we perform similar environment reviews ourselves? This isn't unique to Jamf, plenty of other companies are adopting similar practices. At a high level, it's about using AI at cloud scale to do rapid triage across large codebases, driven through MCPs and APIs. The current bottleneck is the human review gate, and the team is being deliberately conservative about ever skipping that step, even as test coverage improves. If you want to start down this path yourself: begin with documentation. Write everything in markdown that an AI agent could actually read, with clear best practices and good test coverage in your repos. That foundation is what makes the rest possible.

Has implementing AI in this process affected headcount either way? No job impact. The team's view is that every job is changing rather than disappearing: engineers are still needed at every level, from entry to principal, but how each role works is shifting fast, so skill sets and job expectations are being refreshed accordingly. Team composition has changed in places (a team of three engineers might now be two, with the others freed up to work on other product issues or features), but there's been no overall headcount reduction.

What were the technical challenges of building this internally? What technologies and LLMs are you using? It started as a personal side project without enterprise security or scale in mind. Getting it secure and compliant across the whole business meant a lot of conversations and sign-off. On the stack: it runs on AWS with a containerized Kubernetes environment, using a mix of cloud providers for the AI models themselves, and built on the Claude Code SDK in Python.

A lot has changed in macOS binaries. Can Nighthawk review the Extension Attributes GitHub repo to check they're all still valid and functional? This refers to an older Jamf GitHub repo with extension attribute templates for Jamf Pro that hasn't been kept on a regular update cadence, and some binaries in it may no longer exist or may function differently. The team's follow-up: yes, they can have Nighthawk scan it, and they'll look into it.