Can not fetch from VPS in Vultr

[Chain Zhang]

fetching from remote shows error below

Forbidden
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I don't know why only the Vultr has this problem, but they say bitbucket has the issue of accepting the request. I tried same repository in local environment and on AWS, it works without any problem.

Is there any network issue happen or just my account is blocked?

[Daniil Penkin · Accepted solution]

Hello @Chain Zhang ,

I'm not familiar with Vultr, so please tell me a bit more on the configuration. Have you configured authentication against Bitbucket in Vultr? What kind of authentication it is: basic auth with your regular credentials, app password, access key or your personal SSH key?

Cheers,
Daniil

[Chain Zhang]

Thanks for replying, the authentication I used was personal SSH Key, by the way, the app password works fine (this the method our jenkins used).

[Daniil Penkin]

Can you configure SSH to log verbose? One of the ways to achieve this is by setting environment variable on the machine

GIT_SSH_COMMAND="ssh -vvv"

Is that feasible in Vultr? This way we'll see all the steps it takes to authenticate and then verify which key you're using.

Cheers,
Daniil

[Chain Zhang]

This is the output, can you figure out the problem? It seems the authentication was succeeded.

ecouser@ecoese-web-1:/usr/local/app/ecoese$ GIT_SSH_COMMAND="ssh -vvv" git fetch
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /home/ecouser/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "bitbucket.org" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to bitbucket.org [18.205.93.0] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/ecouser/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version conker_e350bda95a app-169
debug1: no match: conker_e350bda95a app-169
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/ecouser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ecouser/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-dss,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,arcfour256,arcfour128
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug3: hostkeys_foreach: reading file "/home/ecouser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ecouser/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: hostkeys_foreach: reading file "/home/ecouser/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/ecouser/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from 18.205.93.0
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/ecouser/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: chainzhang@ChainnoMacBook-Pro.local (0x564b1a9ac8b0), agent
debug2: key: /Users/chainzhang/.ssh/id_rsa (0x564b1a9af680), agent
debug2: key: /Users/chainzhang/.ssh/in2_dev_id_rsa (0x564b1a9af700), agent
debug2: key: /home/ecouser/.ssh/id_rsa ((nil))
debug2: key: /home/ecouser/.ssh/id_dsa ((nil))
debug2: key: /home/ecouser/.ssh/id_ecdsa ((nil))
debug2: key: /home/ecouser/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:6N2mg856hnrnGer4jplsRyddSbEK0JINskIOAKGbL2E chainzhang@ChainnoMacBook-Pro.local
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 407
debug2: input_userauth_pk_ok: fp SHA256:6N2mg856hnrnGer4jplsRyddSbEK0JINskIOAKGbL2E
debug3: sign_and_send_pubkey: RSA SHA256:6N2mg856hnrnGer4jplsRyddSbEK0JINskIOAKGbL2E
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org ([18.205.93.0]:22).
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env LESSOPEN
debug3: Ignored env MAIL
debug3: Ignored env GIT_SSH_COMMAND
debug3: Ignored env SSH_CLIENT
debug3: Ignored env USER
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env OLDPWD
debug3: Ignored env SSH_TTY
debug3: Ignored env LOGNAME
debug3: Ignored env _
debug3: Ignored env TERM
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env PATH
debug3: Ignored env XDG_RUNTIME_DIR
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env LS_COLORS
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env SHELL
debug3: Ignored env LESSCLOSE
debug3: Ignored env PWD
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env XDG_DATA_DIRS
debug1: Sending command: git-upload-pack 'contributor-app/ecoese-server.git'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd ext data 10
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: obuf_empty delayed efd 7/(10)
Forbidden
debug2: channel 0: written 10 to efd 7
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: send packet: type 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 3084, received 1880 bytes, in 0.4 seconds
Bytes per second: sent 7755.7, received 4727.9
debug1: Exit status 1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[Daniil Penkin]

debug1: Offering public key: RSA SHA256:6N2mg856hnrnGer4jplsRyddSbEK0JINskIOAKGbL2E chainzhang@ChainnoMacBook-Pro.local

Hm, I can't find any SSH key with that fingerprint associated with your user or access key for the repository you're trying to access. I assume that SSH key is registered with another Bitbucket user which doesn't have access to the repository in question, or as an access key for some other repository.

Can you ran the following command on the same machine:

ssh -v -T git@bitbucket.org

This should output which user you logged in with, see this documentation page.

Please let me know what you find.

Cheers,
Daniil

[Chain Zhang]

Hi, Daniil, thanks for your support. Yes, I found that I logged in as the wrong account cause this problem. It works after removing the key from the agent.

Thanks a lot.