Getting password prompt after logging in with Github

I’m using Sentry with Team plan and authentication is taken care via Github. I am prompted for password when trying to do changes to projects. I don’t remember setting a password, as I’ve used the Github authentication mechanism.
I’ve also seen similar issues with other people (but with SSO rather than Github): Getting password prompt after logging in with SSO

Any help on this?