OpenSSH_9.0p1 cannot connect to repository

[angelodicrescenzo]

I've installed the OpenSSH 9.0p1 on my MacOS Monterey (12.5.1) via brew but I can't connect to my bitbucket repositories. I haven't any issues with OpenSSH_8.6p1 that is installed in the OS.

Does anyone have this issue and knows how to solve it?

Any help will be appreciated.

These are the log when I try to connect:

ssh -vvv git@bitbucket.org                                                   

OpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022

debug1: Reading configuration data /Users/angelo/.ssh/config

debug1: /Users/xxxx/.ssh/config line 5: Applying options for bitbucket.org

debug1: /Users/xxxx/.ssh/config line 9: Applying options for *

debug1: /Users/xxxx/.ssh/config line 12: Ignored unknown option "usekeychain"

debug1: Reading configuration data /usr/local/etc/ssh/ssh_config

debug1: /usr/local/etc/ssh/ssh_config line 47: Applying options for *

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/xxxx/.ssh/known_hosts'

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/xxxx/.ssh/known_hosts2'

debug2: resolving "bitbucket.org" port 22

debug3: resolve_host: lookup bitbucket.org:22

debug3: ssh_connect_direct: entering

debug1: Connecting to bitbucket.org [104.192.141.1] port 22.

debug3: set_sock_tos: set socket 5 IP_TOS 0x48

debug1: Connection established.

debug1: identity file /Users/xxxx/.ssh/id_rsa type 0

debug1: identity file /Users/xxxx/.ssh/id_rsa-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_9.0

debug1: Remote protocol version 2.0, remote software version conker_55bb0d2b3c a2da21f73e47

debug1: compat_banner: no match: conker_55bb0d2b3c a2da21f73e47

debug2: fd 5 setting O_NONBLOCK

debug1: Authenticating to bitbucket.org:22 as 'git'

debug3: record_hostkey: found key type RSA in file /Users/xxxx/.ssh/known_hosts:12

debug3: load_hostkeys_file: loaded 1 keys from bitbucket.org

debug1: load_hostkeys: fopen /Users/xxxx/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory

debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

ssh_dispatch_run_fatal: Connection to 104.192.141.1 port 22: Operation timed out

[Theodora Boudale · Accepted solution]

Hi @angelodicrescenzo,

I have installed OpenSSH 9.0p1 on a MacOS Monterey (12.5.1) and I see no issues connecting when I run ssh -vvv git@bitbucket.org or when I clone/push to Bitbucket repos via SSH.

The "Operation timed out" error usually indicates a network issue. Do you get this error consistently when you run this command?

Is your computer behind a corporate firewall?

Can you connect from your Mac to a different network (e.g. a 4G network) and check what output you get from ssh -vvv git@bitbucket.org then?

Kind regards,
Theodora

[John Smith]

Similar connection issues today https://community.atlassian.com/t5/Bitbucket-questions/Cannot-perform-any-remote-Git-commands-to-BB/qaq-p/2144475

[angelodicrescenzo]

Hi @Theodora Boudale

I can exclude connection issues because when I use the ssh client that is installed natively on the Operative System (OpenSSH_8.6p1) it works like a charm.

I'm not behind any firweall and I get the connection error consistently.

I'll trying with a cellular network as you suggest but I don't expect it will work for the reason I mentioned above.

Many thanks for your support.

Kind regards,

Angelo.

[Theodora Boudale]

Hi Angelo,

Please let me know how it goes when you have the chance to test this with the same Mac, OpenSSH 9.0p1, and a different network. When you do, please let me know the outcome and what IP bitbucket.org is resolving to in the output (e.g. on the output you shared with your current network it is resolving to 104.192.141.1)

A second thing I would suggest testing is using the same Mac and also the same network, adding temporarily in your /etc/hosts file the entry

18.205.93.0 bitbucket.org

and then running the command ssh -vvv git@bitbucket.org again, and let me know what output you get. This is to narrow down whether the issue may have to do with Anycast Shift, detailed here:

• https://community.atlassian.com/t5/Bitbucket-articles/Diagnosing-and-Troubleshooting-TCP-Resets-from-Bitbucket-Cloud/ba-p/1562132

You mentioned

when I use the ssh client that is installed natively on the Operative System (OpenSSH_8.6p1) it works like a charm

Is this with the same Mac and before you installed OpenSSH 9.0p1 a week ago? Or are you using a different Mac with OpenSSH_8.6p1 now?

Kind regards,
Theodora

[angelodicrescenzo]

Hi @Theodora Boudale

I've run the command ssh -vvv git@bitbucket.org without adding any entry in the /etc/hosts file and using a cellular connection and it worked fine. I've used the OpenSSH 9.0p1 version

I also tried to perform the same test above after adding the entry

18.205.93.0 bitbucket.org

in the /etc/hosts file and it also worked fine

Unfortunately using my internet connection it doesn't work, even if I add the entry in the /etc/hosts file. 

But if I use OpenSSH that is included in macOS Monterey 12.6 (OpenSSH_8.6p1, LibreSSL 3.3.6) with my internet connection it works without any issues.

These are the logs:

nslookup bitbucket.org                          

Server: xx.xx.xx.xx
Address: xx.xx.xx.x#53


Non-authoritative answer:

Name: bitbucket.org

Address: 104.192.141.1

/usr/bin/ssh -vvv git@bitbucket.org            

OpenSSH_8.6p1, LibreSSL 3.3.6

debug1: Reading configuration data /Users/xxx/.ssh/config

debug1: /Users/xxx/.ssh/config line 5: Applying options for bitbucket.org

debug1: /Users/xxx/.ssh/config line 9: Applying options for *

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files

debug1: /etc/ssh/ssh_config line 54: Applying options for *

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/xxx/.ssh/known_hosts'

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/xxx/.ssh/known_hosts2'

debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling

debug1: Connecting to bitbucket.org port 22.

debug1: Connection established.

debug1: identity file /Users/xxx/.ssh/id_rsa type 0

debug1: identity file /Users/xxx/.ssh/id_rsa-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_8.6

debug1: Remote protocol version 2.0, remote software version conker_55bb0d2b3c 0cc9ef731626

debug1: compat_banner: no match: conker_55bb0d2b3c 0cc9ef731626

debug3: fd 5 is O_NONBLOCK

debug1: Authenticating to bitbucket.org:22 as 'git'

debug3: record_hostkey: found key type RSA in file /Users/xxx/.ssh/known_hosts:12

debug3: load_hostkeys_file: loaded 1 keys from bitbucket.org

debug1: load_hostkeys: fopen /Users/xxx/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c

.... Output Omitted ...

So you was right when you thought an ISP connection issue but I can't understand why it works with the OpenSSH_8.6p1, LibreSSL 3.3.6. Do you have any idea?

Many thanks for your support.

Kind regards,

Angelo.

[Theodora Boudale]

Hi Angelo,

Thank you for sharing that info.

After examining and comparing the output of ssh -vvv git@bitbucket.org with OpenSSH_8.6p1 and the one with OpenSSH_9.0p1, we can see the following line in the output of the latter one (which is not present with OpenSSH_8.6p1):

debug3: set_sock_tos: set socket 5 IP_TOS 0x48

During the handshake process, OpenSSH sets a Quality of Service flag in the IP headers. There are some NAT routers that don't work well when this is set to anything other than 0x00. Can you try adding the following in your ~/.ssh/config file and let me know if you can then connect with OpenSSH_9.0p1?

Host *
    IPQoS 0x00

Kind regards,
Theodora

[angelodicrescenzo]

Hi @Theodora Boudale

after adding the IPQoS flag in the config file I was able to connect to the repo.

Thank you very much for you support.

Kind regards,

Angelo.

[Theodora Boudale]

That's good to hear Angelo and you are very welcome, I'm glad to have helped.

Please feel free to reach out if you ever need anything else!

Kind regards,
Theodora