Debugging SSH

Debugging ssh is monotonous shit ‘cos you get reams of messages which don’t tell you why you can’t connect.

E.g.

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED

Actual error message should be:

YOU'VE PROBABLY REPLACED YOUR HOST AND YOUR EXISTING KEY IN ~/.ssh/known_hosts DOES NOT MATCH

Delete your key on line 293.

Permission denied (publickey).

Your keys aren’t on the server. i.e. your Public Key isn’t in the ~/.ssh/authorized_keys file of the user you’re trying to login with.

Use ssh -v to debug. Ignore the 20 odd lines of useless information that get output and focus on:

debug1: Offering public key: RSA SHA256:hash /Users/snowcrash/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Offering public key: RSA SHA256:hash /Users/snowcrash/.ssh/another_key
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
snowcrash@1.2.3.4: Permission denied (publickey).

 

AWS: add ssh key, check fingerprint and add to Terraform

1. Generate key

ssh-keygen -t rsa -b 4096 -C "<email address>"

File name: /home/dir/.ssh/file-name_id_rsa

 

2. Upload

AWS Dashboard > EC2 > Key Pairs > Upload

 

You can check the fingerprint with:

openssl rsa -in path_to_private_key -pubout -outform DER | openssl md5 -c

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#verify-key-pair-fingerprints

It’s important to use the correct openssl command. There are 2 separate commands – one for an AWS generated key and the second for a key you upload.

 

3. add the key_name to Terraform

e.g. a launch configuration:

https://www.terraform.io/docs/providers/aws/r/launch_configuration.html

 

4. ssh in with

ssh -i ~/.ssh/<new-key> ec2-user@<public ip>

If you’re unable to connect make sure you’ve got port 22 open on the EC2 instance Security Group.

E.g. Inbound rule:

e.g.
 

Ansible: Proxy using ProxyCommand

https://stackoverflow.com/questions/28553307/ansible-using-custom-ssh-config-file

and https://stackoverflow.com/questions/38651791/is-it-possible-to-add-ansible-ssh-common-args-in-inventory-file

https://www.cyberciti.biz/faq/linux-unix-ssh-proxycommand-passing-through-one-host-gateway-server/

Will restarting sshd disconnect you?

Say you’re on a bastion host and want to restart sshd with:

 

Is it going to disconnect you?

No.

What you will see if you’re tailing /var/log/auth.log is:

https://serverfault.com/questions/141205/restart-ssh-on-a-machine-where-ssh-is-the-only-mode-of-access

 

Ansible: running through a bastion host

E.g. in inventory create:

ssh-config.yml

with:

ansible_ssh_common_args: "-o ProxyCommand='ssh -W %h:%p {{ AWS_IAM_ID }}@hostname' -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no"

Then make sure you’ve got both your private keys (bastion and destination) added via ssh-add.

E.g. ssh-add ~/.ssh/id_rsa

etc…

Debugging

  1. use ansible -vvvv to get ssh output
  2. run this ssh command
  3. I was getting

ec2-user@<ip>: Permission denied (publickey).

4. ssh to bastion and check you can access that host

5. on the bastion check the sshd logs

tail -f /var/log/auth.log

which revealed nothing.

i.e. the first line shows the successful ssh connection to the jumpbox. The second shows the disconnect.

Note: setting sshd logging to verbose did not help. e.g.

6. checking the destination box

Note this had a different sshd log at:

/var/log/secure

A successful connection (i.e. directly from the bastion host) would show:

Accepted publickey for ec2-user

but via the proxy command I’d get:

Connection closed by <bastion> [preauth]