It happen to me all the time that one of developers notifies me about some kind of problem that I can’t confirm from my account. Sometimes it was because of bad ssh keys configuration, other times file permissions, mostly such stuff. It’s sometimes convenient to “enter into someone’s shoes” to see what’s going on there.
root on machine you may do that like this:
su developer -
Easy one but that’s not enough for all cases. When you use bastion host (or similar solutions) sometimes users have connection problems and it’s harder to check. When such user have
ForwardAgent ssh option enabled you may stole this session to check login problems. After you switch to such user, you may wan’t to hide history (it’s optional 😉 ) – disable history like that:
Now you may stole ssh session, but first check if you have your dev is logged on:
$ ls -la /tmp/ | grep ssh
drwx------ 2 root root 4096 Apr 27 20:56 ssh-crYKv29798
drwx------ 2 developer developer 4096 Apr 27 18:03 ssh-cVXFo28108
SSH_AUTH_SOCK with path to developer’s agent socket:
Finally you may try to login via
ssh as developer and see with his eyes what’s now working.
Virtualenvs in python are cheap but from time to time you will install something with pip on your system and when time comes removing all this crap could be difficult. I found this bash snippet that will uninstall package with all dependencies:
for dep in $(pip show python-neutronclient | grep Requires | sed 's/Requires: //g; s/,//g') ; do sudo pip uninstall -y $dep ; done
pip uninstall -y python-neutronclient
When you deploy your application in cloud you don’t need and don’t want your hosts exposed via SSH to the world. Malware scans whole network for easy SSH access and when find something will try some brute force attacks, overloading such machines. It’s easier to have one exposed, but secured host, that doesn’t host anything and is used as proxy/gateway to access our infrastructure- it’s called bastion host.
Ansible is quite easy to integrate with bastion host configuration. We will need custom
ssh_config file. So let’s start with
ProxyCommand ssh -q -A ubuntu@bastion nc %h %p
Now I will describe what most important options mean. For bastion:
User – I’m using Ubuntu kickstarted on cloud as bastion host with it’s default user. Never use root here – you don’t need that
ForwardAgent yes – we want to forward our ssh keys through bastion to destination hosts,
ServerAliveInterval 60 – this is like keepalive connection, ssh will send small ping/pong packets every 60 seconds so your connection won’t hung/terminate after long time,
ControlMaster auto – we will open one connection to bastion host and multiplex other ssh connections through it, connection will be opened for
ControlPath – this have to be configured same way like in
ProxyCommand none – we’re setting
ProxyCommand for all hosts but we need it disabled for bastion,
Default hosts configuration:
ProxyCommand ssh -q -A ubuntu@bastion nc %h %p – this is what makes all magic, it will pipe your ssh connection via bastion to destination host,
StrictHostKeyChecking no – this options shouldn’t be there for production but it’s useful at beginning when you create and destroy machines few times before you test everything. Normally this will cause notifications about ssh key changes, but you’re aware of that – you just recreated those machines.
I’ve found examples without netcat but was unable to get them working – this one worked for me really well.
To test if connections work fine use this configuration like:
ssh -F ssh_config bastion
ssh -F ssh_config other.host.behind.bastion
ssh_args = -F ./ssh_config -o ControlMaster=auto -o ControlPersist=5m -o LogLevel=QUIET
control_path = ~/.ssh/ansible-%%r@%%h:%%p
Most important section here is in
ssh_args where we’re pointing to
ssh_config file in current dir with
-F option. I also have to reenter configuration for multiplexing here – it wasn’t working with ssh only configuration.
control_path option have to use same paths like
% signs are escaped with
You should be able to run
ansible-playbook commands normally now – all traffic will be forwarded through bastion.
It’s good time now to install
fail2ban on bastion and maybe reconfigure it to run
ssh on crazy high port 🙂
Sometimes it’s easier to use octal file permissions but they’re not so easy to list. I caught myself few times that I didn’t remember how to list them – so this is a reason for that note.
$ stat -c "%a %n" *
Yes, it’s that easy 🙂
And here also with human readable attributes:
$ stat -c '%A %a %n' *
drwxr-xr-x 755 bin
drwxr-xr-x 755 games
drwxr-xr-x 755 include
I’m playing a lot with Docker lately. Building images, and then rebuilding, and then building again… It’s pretty boring. To automate this task a little I used inotify to build automatically after I changed any file. This trick could be used in many different situations.
You will need
sudo apt-get install -y inotify-tools
Then run something like this:
while inotifywait -e modify -r .; do docker-compose build; done
This commands will rebuild my Docker images after any file change in current directory. Use
Ctrl+c to exit from loop.