The Access Policies have been configured correctly and I was owner of the subscription.
Unfortunately, the KeyVault requires explicit 'owner' permissions on KeyVault level to make this work.
Error
Get-AzKeyVaultSecret : Operation returned an invalid status code 'Forbidden'
Solution
Grant owner permissions (temporarily) to your AD-User on KV-Level.
Cheers