Just posting in case helpful. Inherited an environment where Mac's have always RDP's to Windows and gotten the "The certificate couldn't be verified back to a root certificate..." warning and it's been ignored.
Note: Windows > Windows RDP should be using Kerberos authentication, which is why you're not seeing the error there.
I believe the below is the proper way to fix (with the upside of working on all machines you're RDP'ing to, as long as they have the GPO). As opposed to just marking the cert it can't verify as trusted. At least this is what myself and our server guy came up with after putting our heads together. We no longer get those certificate warnings.
1. Have AD CS configured and a Root CA.
2. Create a certificate template to be the RDP cert / cert used to authenticate an RD Session Host server.
3. GPO on Windows machine you are RDP'ing to - Windows Components/Remote Desktop Services/Remote Desktop Session Host/Security - Server Authentication Certificate Template - pointing to template mentioned above.
4. (Group policy update on each VM/machine is needed before warning will go away upon connecting.)
5. Get your Root CA cert, import it onto your Mac, mark it trusted in keychain access, then export.
6. Create a config profile in Jamf with that cert and push out.
7. Ensure you are connecting by FQDN in your macOS RDP client (and not regular domain name or IP address) or you'll still get the error.
8. There should no longer be a warning upon initiating the RDP connection.
