To use with an ssh key, you need a git@github url. So if you have e.g.
https://github.com/johnallsup/javascript-games
you need to
git remote set-url origin git@github.com:johnallsup/javascript-games.git
Then add your id_rsa.pub (or other public key) to github via settings.