I added the credentials to my spark configuration and it worked. So I am not passing credentials to data catalog, just making them available at the cluster level.