I don't use passphrases nor ssh-agent; most of my coworkers who do end up having issues. It's finicky. So all your tests are run within the same terminal, right? I would try to find out which ssh ...