How to stole ssh session when you’re root

It happen to me all the time that one of developers notifies me about some kind of problem that I can’t confirm from my account. Sometimes it was because of bad ssh keys configuration, other times file permissions, mostly such stuff. It’s sometimes convenient to “enter into someone’s shoes” to see what’s going on there.

If you’re root on machine you may do that like this:

su developer -

Easy one but that’s not enough for all cases. When you use bastion host (or similar solutions) sometimes users have connection problems and it’s harder to check. When such user have ForwardAgent ssh option enabled you may stole this session to check login problems. After you switch to such user, you may wan’t to hide history (it’s optional πŸ˜‰ ) – disable history like that:

export HISTFILESIZE=0
export HISTSIZE=0
unset HISTFILE

Now you may stole ssh session, but first check if you have your dev is logged on:

$ ls -la /tmp/ | grep ssh
drwx------   2 root     root          4096 Apr 27 20:56 ssh-crYKv29798
drwx------   2 developer developer    4096 Apr 27 18:03 ssh-cVXFo28108

Export SSH_AUTH_SOCK with path to developer’s agent socket:

SSH_AUTH_SOCK=/tmp/ssh-cVXFo28108/agent.28108

Finally you may try to login via ssh as developer and see with his eyes what’s now working.

pip – uninstall package with dependencies

Virtualenvs in python are cheap but from time to time you will install something with pip on your system and when time comes removing all this crap could be difficult. I found this bash snippet that will uninstall package with all dependencies:

for dep in $(pip show python-neutronclient | grep Requires | sed 's/Requires: //g; s/,//g') ; do sudo pip uninstall -y $dep ; done
pip uninstall -y python-neutronclient

Source: http://stackoverflow.com/a/32698209/4828478

Use bastion host with Ansible

When you deploy your application in cloud you don’t need and don’t want your hosts exposed via SSH to the world. Malware scans whole network for easy SSH access and when find something will try some brute force attacks, overloading such machines. It’s easier to have one exposed, but secured host, that doesn’t host anything and is used as proxy/gateway to access our infrastructure- it’s called bastion host.

Ansible is quite easy to integrate with bastion host configuration. We will need custom ansible.cfg and ssh_config file. So let’s start with ssh_config:

Host bastion
  Hostname ip.xxx.xxx.xxx.xxx.or.host.name
  User ubuntu
  IdentityFile ~/.ssh/id_rsa
  PasswordAuthentication no
  ForwardAgent yes
  ServerAliveInterval 60
  TCPKeepAlive yes
  ControlMaster auto
  ControlPath ~/.ssh/ansible-%r@%h:%p
  ControlPersist 15m
  ProxyCommand none
  LogLevel QUIET

Host *
  User ubuntu
  IdentityFile ~/.ssh/id_rsa
  ServerAliveInterval 60
  TCPKeepAlive yes
  ProxyCommand ssh -q -A ubuntu@bastion nc %h %p
  LogLevel QUIET
  StrictHostKeyChecking no

Now I will describe what most important options mean. For bastion:

  • User – I’m using Ubuntu kickstarted on cloud as bastion host with it’s default user. Never use root here – you don’t need that
  • ForwardAgent yes – we want to forward our ssh keys through bastion to destination hosts,
  • ServerAliveInterval 60 – this is like keepalive connection, ssh will send small ping/pong packets every 60 seconds so your connection won’t hung/terminate after long time,
  • ControlMaster auto – we will open one connection to bastion host and multiplex other ssh connections through it, connection will be opened for ControlPersist time,
  • ControlPath – this have to be configured same way like in ansible.cfg,
  • ProxyCommand none – we’re setting ProxyCommand for all hosts but we need it disabled for bastion,

Default hosts configuration:

  • ProxyCommand ssh -q -A ubuntu@bastion nc %h %p – this is what makes all magic, it will pipe your ssh connection via bastion to destination host,
  • StrictHostKeyChecking no – this options shouldn’t be there for production but it’s useful at beginning when you create and destroy machines few times before you test everything. Normally this will cause notifications about ssh key changes, but you’re aware of that – you just recreated those machines.

I’ve found examples without netcat but was unable to get them working – this one worked for me really well.

To test if connections work fine use this configuration like:

ssh -F ssh_config bastion
ssh -F ssh_config other.host.behind.bastion

And now ansible.cfg:

[defaults]
forks=20

[ssh_connection]
ssh_args = -F ./ssh_config -o ControlMaster=auto -o ControlPersist=5m -o LogLevel=QUIET
control_path = ~/.ssh/ansible-%%r@%%h:%%p
pipelining=True

Most important section here is in ssh_args where we’re pointing to ssh_config file in current dir with -F option. I also have to reenter configuration for multiplexing here – it wasn’t working with ssh only configuration. control_path option have to use same paths like ssh_config (% signs are escaped with %%).

You should be able to run ansible/ansible-playbook commands normally now – all traffic will be forwarded through bastion.

It’s good time now to install fail2ban on bastion and maybe reconfigure it to run ssh on crazy high port πŸ™‚

Source:
http://alexbilbie.com/2014/07/using-ansible-with-a-bastion-host/ http://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/
https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing

List octal file permissions in bash

Sometimes it’s easier to use octal file permissions but they’re not so easy to list. I caught myself few times that I didn’t remember how to list them – so this is a reason for that note.

$ stat -c "%a %n" *
755 bin
755 games
755 include

Yes, it’s that easy πŸ™‚
And here also with human readable attributes:

$ stat -c '%A %a %n' *
drwxr-xr-x 755 bin
drwxr-xr-x 755 games
drwxr-xr-x 755 include

Automatically build after file change

I’m playing a lot with Docker lately. Building images, and then rebuilding, and then building again… It’s pretty boring. To automate this task a little I used inotify to build automatically after I changed any file. This trick could be used in many different situations.

You will need inotify-tools package:

    sudo apt-get install -y inotify-tools

Then run something like this:

while inotifywait -e modify -r .; do docker-compose build; done

This commands will rebuild my Docker images after any file change in current directory. Use Ctrl+c to exit from loop.