Block installing new applications for admin users

Do the devices have to be owned by the organization and in MDM for Restricted Software to work?

I know if I BMOD to work and they blocked installs, I may not be happy about that and stop bringing it...

Which Apps are you trying to block?

They do not need to be owned by the organization. Organization, in this case, bought MacBooks, but they are initialized by employees. And yes, devices have to be in MDM for Restricted Software to work. 
Restricted Software works fine, but providing a list of restricted software is not an option for us. For example, Watsapp, Viber, any kind of torrent, etc... The list could be endless.
We must have some kind of mechanism to whitelist software that we need and to forbid everything else.
Also, installing the software that is needed at the start, and forbidding any new installation by the end-user would also work.

In your original post, you mentioned BYOD, which implies that the computers are employee owned. Thanks for the clarification.

Maybe this thread would help...?

https://community.jamf.com/t5/jamf-pro/show-evidence-standard-users-are-unable-to-install-applications/m-p/253148

@zlatko Google's Santa tool will allow you to restrict what applications can run: https://github.com/google/santa

The entire concept of BYOD devices is that ownership and management remains in control of the end user. That includes their privilege to install whatever apps they choose. Your management system will never have total control.

You do have control over "managed apps" that your MDM installs and with managed apps you can control things like "open in" or "open with". This would allow you, for example, to specify that email attachments sent to an organization's email system can only be copied to DropBox and not Messages. Unfortunately, that's more iOS than macOS.

If you need more control over a device, your organization needs to instead own it and manage it starting with Automated Device Enrollment where you'll have the most control available to you.

Hmm. Maybe a silly question, but if your org needs to control these Macs to that degree, why are the users local admins? You should be revoking their admin rights, or just not making them admins from the get-go. There is a maxim in this industry - as soon as users have admin rights, all bets are off on what you can control.

Making them standard accounts won't 100% solve the issue, but it will make a serious dent in it. Without local admin rights, all installers using a pkg format will be off limits. You'll still need to deal with apps that distribute on a DMG, because most times those can be copied to the user's Desktop and launched without issue.

If you have to control that, you can do a couple of other things. First, you can add the disk image helper application to the Restricted Software list. This prevents casual double clicking of disk images and mounting them, since when you do this in the Finder it launches the Disk Image Helper app. If you block it, they won't be able to open the DMGs. That being said, anyone with enough general knowledge can figure out how to mount disk images using Terminal, so it's more of a deterrent than a bullet proof block.

Second option is, you can go down the path of using Application folder whitelisting and blacklisting in a Config Profile under the Restrictions payload. This is a more complex option to use, and often entails a lot of trial and error, and sometimes continuous tweaking, to get it working satisfactorily. But it will allow you to say that only applications in /System/, /Applications/, and /Library/Application\\ Support/ for example, can launch, but other locations are blocked. So even if they manage to download an app and copy it to their desktop, they won't be able to launch it.

Just fair warning that this option isn't the easiest to set up and use, and you may need to field a decent amount of complaints of apps that won't work correctly until you add in extra whitelisting paths for example.

Edit: Correction on the above. It's Disk Image Mounter (DiskImageMounter), not Disk Image Helper as I stated incorrectly before.

You may be confusing the term ownership with usage. If your org has purchased the Macs, then your org "owns" the Macs and they don't belong to the end users using them. If that's the case, then if you haven't done so already, look into getting your org signed up with Apple Business or School Manager so they can be enrolled in Apple's Automated Device Enrollment process. This makes the Macs Supervised, so you have some extra control over them, not to mention auto enrollment is a nice thing to have.

If these are in fact real BYOD Macs, as in the end users own them, then needing to lock them down to the level you're describing means they aren't a good fit for the company. It doesn't sound like the users own them though, based on your post above, so my guess is the first description.

Wow! A lot of useful answers! Did not expect this much! Thank you guys A LOT for your time trying to help me! I really appreciate it SO MUCH!
As I can conclude, I misunderstood the term BYOD. I assumed that, if the company bought Macs and gave them to the employees to set it up with admin rights, it is BYOD. I definitely confused the term ownership with usage. Thanks for the clarification!
The majority of our laptops are for developing software because we are a SaaS business platform. Programmers on start are given rights to install whatever they need because there could be differentiates based on role (frontend, backend, python, java, AWS DevOps, and so on). But after ISO 27001 certification, one of the minor disconformities was the Watsapp application on the employee's laptop, and MDM was suggested as a solution.
At this moment, what we have in the plan is to leave admin rights as it is, and just to prevent future installations. For example, if the user has already installed package manager for Python, he will be able to install any new python package without asking for permission. We are mainly concerned about non-technical users (marketing, sales, for example) because they are more likely to install malicious software. So allowing DMG to be installed only through mount disk images using Terminal is totally acceptable for us. Also, we have security audits once in a while, and we can control which applications are installed, to be sure that everyone is respecting the computer usage policy which is accepted through the onboarding process.

Due to a fact that I am new to the whole MDM thing (and Mac thing also, I am a Linux guy :)), I discovered Apple Business or School Manager just recently, and we today (believe it or not :)) got a confirmation that our request is confirmed. Will see what capabilities are. 
@awoodbury, thanks. That looks exactly what I was looking for. The option with restricting paths for Disk Image Mounter and Disk Images UI Agent looks more elegant to me, and I just tried it - But it does not work. :/ I tested with the micro torrent dmg file and also with the firefox dmg file. Both are mounted and installed without any problems.
I am testing soon the first option mm2270 mentioned here (same as in your post), but it will take more time and I will postpone it for tomorrow. Will keep you informed. 
Once again, thanks for your effort and valuable informations!

Ah, developers. That kinda makes sense now why they have admin rights. Just so you know though, it is possible in many cases to have developers as standard users and still be able to do their jobs. But it's definitely harder, no doubt about it. I live this every day since I work in a financial institution. No-one outside of IT has local admin rights, and many of our Mac users are also developers. It's a pain for sure, and I've had to make heavy use of Self Service and advanced scripting for them to be able to perform their work. So I feel for what you're dealing with.

In terms of blocking mounting of disk images, it's definitely possible to block the Disk Image Mounter process using Restricted Software. I'm posting 2 images of how I tested it out some months back. These are confirmed working titles. I am not actually using this in our environment, but I explored the option and confirmed it did indeed block disk images from mounting. You have to block 2 processes in my testing for it to work.

One other thing to keep in mind when testing this. Not sure if you know, but enabling a new Restricted Software title doesn't take effect instantly. The Macs in scope have to check in to Jamf Pro in order to get the new settings. You can force the check in by doing a sudo jamf policy on the Mac and it should pull down the new settings. Just wanted to mention that in case that was the issue with your last test.

:) 
Thanks for letting me know! We will move from admin accounts in the future, we are just a little bit in a rush at this moment because of the ISO deadline. So, the mentioned solution seemed like the easiest and most effective at the moment. 
About this issue, I found those screenshots on the awoodbury post he mentioned here, and I followed it. But I did not help.
Because I do not have permission to send video, I will describe the process of opening the Firefox application on my Macbook:

I downloaded the dmg file and double-clicked it, then prompted to drag and drop it to the Applications folder. After I drag and drop it, it is shown in the applications folder, but not in the Launchpad. 
After I right-click Firefox in the application folder and click Open, the system warns me:

“Firefox” can’t be opened because it was not downloaded from the App Store. Your security preferences allow the installation of only apps from the App Store. After I click Ok and repeat the Open process, I get:

“Firefox” is not from the App Store. Are you sure you want to open it?
Clicking Open opens it without any problems.

Am I missing something? At which exact point disk image mounter should show effect? 

If this does not work, I will have to check the second option, to populate Config Profile under the Restrictions payload, but I more liked this one. :) Was so disappointed when it did not work. 

That's odd. Are you sure the machine you're testing on has received the updated restricted software setting? Did you run a sudo jamf policy on it? You can also try running sudo jamf manage to ensure it's received all the latest management settings.

I'm going to go back and test mine out again to make sure I'm not overlooking something. Just so I know, what operating system version are we talking about here? Big Sur? Monterey? Or something else.

Ok, tested and I can confirm it's working on both an Intel MacBook Pro running 11.6.3, and a brand new M1 Pro MacBook Pro running 12.3. In both cases, after applying the Restricted Software setting and confirming it's in place, I downloaded the latest googlechrome.dmg file and tried opening them in the Finder and it will not mount. I should point out that in my case, my settings don't pop up a message on screen, so the block is silent. It just doesn't run disk image mounter and therefore cannot mount the disk image.

It's still possible to open it using the command line, like I mentioned, since that uses a different process not affected by those restricted software titles. In fact, the jamf binary uses the same shell driven process, so it doesn't affect deploying DMGs from Jamf, which is kind of nice, in case you happen to use that format at all.

I would suggest going back to double check the spelling and everything in your Restricted Software titles matches up, and then make sure to force both a check in and sudo jamf manage (just to be sure) on your test Mac and try it again.

Hey, @mm2270 ! Sorry for the late replay. 
My MacBook is M1 Pro, Monterey 12.2.1. 
I did everything that you suggested ( sudo jamf policy, sudo jamf manage, double-checked spelling, etc.)
And I can confirm that blocking Chrome is working. Nothing happens. It is blocked silently. Same for Watsapp, and Skype. So, I assume that policy is enforced.
But for Viber, microtorrentWeb, Firefox, the installation prompt is opened and applications are installed without any problems.
Could you, please, confirm that block is working for you for Firefox or any other application that does not work for me?

Ok that’s very weird. Restricted software settings should not work “selectively” like that. 

Sure, I’ll try with Firefox and some of those other products that aren’t working for you. Will let you know what happens. 
In the meantime, this sounds odd to ask, but have you rebooted the Mac since the setting was applied, just to rule out anything funky like that?

Thanks a million!
Not odd to ask at all. :) Yes, I tried that. I also just tried to shut it down, but the same applies. Nothing changes, unfortunately.