Apple saves the Preference Pane binaries in /System/Library/PreferencePanes. You can try to make an app restriction for the binary name and block it which is what we do for the Apple Intelegence Pane as apple does not tell users its being managed when you are managing it. Just be cautions of unintended blast radius of blocking things you don’t want to block.
