Thursday, April 19, 2012

Mosh, NAT & LXC containers

I've been using mosh for a few days now and find it very promising.  There are a few aspects of it that are not the most convenient, but these are balanced out by other features.  In particular the following points need patience or workarounds:
  1. Lack of SSH agent forwarding.  This is a pretty big deal and a source of much frustration.
  2. No IPv6 support yet.
  3. No configuration file similar to SSH's ssh_config or ~/.ssh/config
  4. No graceful handling of NAT'd servers.
The first point about SSH agent forwarding I'm hoping will just need some patience.  There appear to be enough people feeling the pain so I suspect this will be resolved soon.

The lack of IPv6 support is frustrating since there is the lack of SSH agent forwarding and the inability for mosh to easily traverse NAT's.  I expect this too should be an easy win and I hope it will be coming sooner than later.

Regarding point three, I find myself wanting a configuration file much like SSH for various reasons.  For example to create shortcut entries for hosts, or to handle some sort of ProxyCommand, etc.  However my motivation for this post relates more to the need to configure a port for a particular host, which leads me to the NAT related point.

I have LXC containers running on bare metal hardware.  Frequently these LXC's are behind a NAT like this:

                                     +--- (eth0) lxc-guest-1
                                     |    10.1.1.1
WAN --- (eth0) lxc-master-1 (br0) ---+
       1.2.3.4            10.1.1.254 |
                                     +--- (eth0) lxc-guest-2
                                          10.1.1.2

In this case you wouldn't normally be able to establish a mosh connection to either of the guest servers, however with some ugly setup it can be done.  This requires iptables rules and some client-side magic.

With SSH there are a couple of ways to handle this scenario.  My preferred approach is to make use of the ProxyCommand through the NAT'ing master, however this doesn't work with mosh:
Host lxc-guest-1
    ProxyCommand ssh lxc-master-1 /bin/nc -q 8 -w 3 lxc-guest-1 22

For mosh you need to get far more involved, setting up port forwarding through the NAT and finally having custom command lines to connect.

First, to get this NAT working the standard easiest approach is to:
iptables -A POSTROUTING -o eth0 -s 10.1.1.0/24 -j MASQUERADE 

Then you need to set up port forwarding for ssh:
iptables -A PREROUTING -i eth0 -p tcp --dport 22001 -j DNAT --to-destination 10.1.1.1:22
iptables -A PREROUTING -i eth0 -p tcp --dport 22002 -j DNAT --to-destination 10.1.1.2:22

And finally port forwarding for mosh.  It seems that by default mosh will start listening at the low end of the ports, so each LXC container will listen on port 60001 by default first.  This will quickly overlap between host and guests.
iptables -A PREROUTING -i eth0 -p udp --dport 60010:60019 -j DNAT --to-destination 10.1.1.1
iptables -A PREROUTING -i eth0 -p udp --dport 60020:60029 -j DNAT --to-destination 10.1.1.2

With all the relevant ports being forwarded you can now set up your SSH config in ~/.ssh/config
Host lxc-guest-1
    Hostname lxc-guest-1
    Port 22001

Host lxc-guest-2
    Hostname lxc-guest-2
    Port 22002

And finally you should be able to establish a mosh connection with a defined port:
mosh -p 60010 lxc-guest-1

With this config you can establish up to 5 total connections to the each guest.  This is because mosh uses the requested port for one direction of the communication, and the port+1 for the return communication.  Thus for each additional connection you'll need to increment your port number by 2.

Since you've made it this far hopefully this is working for you too now.

Since there could easily be many ports with many hosts I've made a very simple script to handle the port config for the first connection.  Put this in one of your PATH folders.  I've saved this script at /usr/local/bin/moshi
#!/bin/bash

source ~/.ssh/mosh_config
eval opts=\$${1//-/_}
mosh $opts $1

And as you can see the script expects a rudimentary mosh config file at ~/.ssh/mosh_config.  For the above server setup the config file contains the following:
lxc_guest_1="-p 60010"
lxc_guest_2="-p 60020"

Now to connect to either guest I use:
moshi lxc-guest-1

As you see this is horribly messy, but for now working with several dozen servers at the other side of the world it's making my life slightly easier.  If I find it useful enough I might even create a chef recipe for it, but I sincerely hope this won't be necessary and that both SSH agent forwarding and IPv6 support will be implemented soon.

Wednesday, April 11, 2012

Mosh - the great new SSH replacement

I'm regularly connecting to servers at the other side of the planet, frequently with latencies of 280-300ms.  While you do get used to the slow response of such connections any improvement is welcome, to the extent that I was pleased to hear that a 60ms improvement would be on the cards soon.

I heard about the mosh project today and made time to get it working this evening.  The initial impression is great.  In addition to the improvements in the feeling of latency it's fantastic to be able to switch between wired and wireless connectivity without losing the terminal session.

I can definitely see this becoming an essential tool.

Installation on Mac OSX

On my OSX 10.7 client installation was as simple as:
sudo port install mosh

Installation on Debian Squeeze

Update: It appears mosh has arrived in Debian squeeze backports: https://github.com/keithw/mosh/issues/75#issuecomment-5067129

On the Debian Squeeze server it was a little more involved.  I tried to use the debian testing repository with apt preferences for Pin-Priority.  Unfortunately trying to install mosh wanted to upgrade around 20 packages to the testing versions.  Not something I want to do on production servers.

Fortunately building and install mosh was easier than expected using the standard squeeze packages:
sudo aptitude install build-essential autoconf protobuf-compiler libprotobuf-dev libboost-dev libutempter-dev libncurses5-dev zlib1g-dev pkg-config
git clone https://github.com/keithw/mosh
cd mosh
./autogen.sh
./configure
make
sudo make install

Following this it was necessary to open up the relevant ports in the firewall:

$ sudo iptables -A INPUT -p udp -m multiport --dports 60000:61000 -j ACCEPT

Locale configuration

The Mosh developers have decided to only go down the path of supporting UTF-8, which doesn't seem like such a bad idea, however it does mean you'll have to ensure both ends of your connection properly support the UTF-8 locales.  To do this the following conditions need to be met.

On the server ensure you have the following line in your sshd_config.  On debian this is located at /etc/ssh/sshd_config:

Server /etc/ssh/sshd_config
...
AcceptEnv LANG LC_*
... 

OSX client ~/.ssh/config or /etc/ssh_config. Thanks to srmadden for this snippet.
...
Host *
    SendEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
  SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
  SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
...

Now confirm the locales are correct.  Initially running locale on both client and server reported the locales were all correctly UTF-8, however checking the server locale via the ssh one liner "ssh remotehost locale" was returning POSIX. After adding the above config the correct locales were returned.

On the local workstation:
user@workstation $ locale  
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8" 

And a regular connection to the remote host.
user@workstation $ ssh remotehost locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

Now go ahead and enjoy the low latency "feeling" and the ability to seamlessly move between connections.

Last but not least a big thank you to all the people that made the mosh project a reality!

Thursday, March 29, 2012

Opscode Chef client within an rbenv environment

Ruby dependencies and versions on distro's aren't always maintained to our liking. I've had problems deploying Chef on CentOS appliances running ruby on rails apps that break when the packages for Chef are installed. I know the problem should be fixed so dependencies can be handled correctly, but that's not always going to happen as quickly as we'd like. So I adapted the default CentOS bootstrap script to install Chef within an rbenv environment.

Hopefully I won't need to ever do this for Debian.

Since this is limited to CentOS I have the following hard-coded condition in the recipe[chef-client::service]

*** service.rb.orig 2012-03-29 22:36:48.000000000 +0100
--- service.rb 2012-03-18 14:29:20.000000000 +0000
***************
*** 50,55 ****
--- 50,60 ----
    init_content = IO.read("#{node["languages"]["ruby"]["gems_dir"]}/gems/chef-#{chef_version}/distro/#{dist_dir}/etc/init.d/chef-client")
    conf_content = IO.read("#{node["languages"]["ruby"]["gems_dir"]}/gems/chef-#{chef_version}/distro/#{dist_dir}/etc/#{conf_dir}/chef-client")
  
+   # We're always using rbenv on CuntOS, so ensure the service does too
+   if platform?("centos") then
+  conf_content = "#{conf_content}\nexport PATH=\"$HOME/.rbenv/bin:/usr/local/bin:$PATH\"\neval \"$($HOME/.rbenv/bin/rbenv init -)\"\nrbenv shell $(rbenv versions | tail -n 1| grep -Eo '\w+\.\w+\.\w+-\w+')\n"
+   end
+ 
    file "/etc/init.d/chef-client" do
      content init_content
      mode 0755

Adapt the following bootstrap script as needed, and use at your own risk.

bash -c -x -e '
<%= "export http_proxy=\"#{knife_config[:bootstrap_proxy]}\"" if knife_config[:bootstrap_proxy] -%>

#
# EXAMPLE BOOTSTRAP LINE
#
# knife bootstrap -N your_host -E development -r 'role[base-server]','role[lsb]' --template-file ~/scm/git/chef/bootstrap/centos5-rbenv.erb your_host_fqdn

#
# CONFIGURATION
#
export RBENV_VERSION="1.9.3-p125"

#
# MAIN
#
[ -f /usr/bin/git ] || yum -y install git
if [ ! -d ~/ruby-build ]; then
 cd ~/
 git clone git://github.com/sstephenson/ruby-build.git
 cd ~/ruby-build
 ./install.sh
fi
if [ ! -d ~/.rbenv ]; then
 cd ~/
 git clone git://github.com/sstephenson/rbenv.git .rbenv

 echo "export PATH=\"\$HOME/.rbenv/bin:/usr/local/bin:\$PATH\"" >> ~/.bash_profile
 echo "export PATH=\"\$HOME/.rbenv/bin:/usr/local/bin:\$PATH\"" >> ~/.zshenv

 echo "eval \"\$(\$HOME/.rbenv/bin/rbenv init -)\"" >> ~/.bash_profile
 echo "eval \"\$(\$HOME/.rbenv/bin/rbenv init -)\"" >> ~/.zshenv

 echo "source \$HOME/.bash_profile" >> ~/.bashrc
 echo "rbenv shell $RBENV_VERSION" >> ~/.bashrc

 echo "source \$HOME/.zshenv" >> ~/.zshrc
 echo "rbenv shell $RBENV_VERSION" >> ~/.zshrc
fi

export PATH="$HOME/.rbenv/bin:/usr/local/bin:$PATH"
eval "$($HOME/.rbenv/bin/rbenv init -)"

if  $HOME/.rbenv/bin/rbenv versions | grep -q $RBENV_VERSION && [ -x $HOME/.rbenv/shims/ruby ] ; then
 eval "`$HOME/.rbenv/bin/rbenv sh-shell $RBENV_VERSION`"

else
 export CONFIGURE_OPTS="--disable-install-doc"
 #export MAKEOPTS="-j$(cat /proc/cpuinfo | grep ^processor | tail -n1 | awk \"{print $3 + 2}\")"
 rm -rf /tmp/ruby-build*
 rbenv install $RBENV_VERSION &
 while [ ! -d /root/.rbenv/versions/1.9.3-p125/lib/ruby/1.9.1 ]; do
  sleep 5
  echo "Waiting for ruby-build to complete."
 done
 $HOME/.rbenv/bin/rbenv sh-shell $RBENV_VERSION
fi

if [ ! -f /usr/bin/chef-client ]; then
 rpm -qa epel-release | grep -q epel-release || {
  wget <%= "--proxy=on " if knife_config[:bootstrap_proxy] %> http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
  rpm -Uvh epel-release-5-4.noarch.rpm
 }

 [ -f /etc/yum.repos.d/aegis.rep ] || wget <%= "--proxy=on " if knife_config[:bootstrap_proxy] %>-O /etc/yum.repos.d/aegis.repo http://rpm.aegisco.com/aegisco/el5/aegisco.repo
 
 yum install -y gcc gcc-c++ automake autoconf make
fi

RBENV_BIN="$HOME/.rbenv/versions/$RBENV_VERSION/bin"
GEM_OPTS="--no-rdoc --no-ri"

gem update $GEM_OPTS --system
gem update $GEM_OPTS
[ -x $RBENV_BIN/ohai ] || gem install ohai $GEM_OPTS --verbose
[ -x $RBENV_BIN/chef-client ] || gem install chef $GEM_OPTS --verbose <%= bootstrap_version_string %>

mkdir -p /etc/chef

for x in chef-client chef-solo knife ohai shef; do
 [ -h /usr/bin/${x} ] && rm -f /usr/bin/${x}
 ln -s $RBENV_BIN/${x} /usr/bin/${x}
done

(
cat <<'EOP'
<%= validation_key %>
EOP
) > /tmp/validation.pem
awk NF /tmp/validation.pem > /etc/chef/validation.pem
rm /tmp/validation.pem

(
cat <<'EOP'
<%= config_content %>
EOP
) > /etc/chef/client.rb

(
cat <<'EOP'
<%= { "run_list" => @run_list }.to_json %>
EOP
) > /etc/chef/first-boot.json

<%= start_chef %>'

Monday, February 27, 2012

RabbitMQ startup and "Too short cookie string"

While setting up a RabbitMQ cluster through the Opscode Chef cookbookI ended up in a situation where rabbitmq-server wouldn't start even without a config file.  The error logs showed:

> /var/log/rabbitmq/startup_err
Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})

> /var/log/rabbitmq/startup_log
{error_logger,{{2012,2,27},{18,8,41}},"Too short cookie string",[]}
{error_logger,{{2012,2,27},{18,8,41}},crash_report,[[{initial_call,{auth,init,['Argument__1']}},{pid,<0.19.0>},{registered_name,[]},{error_info,{exit,{"Too short cookie string",[{auth,init_cookie,0},{auth,init,1},{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]},[{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},{ancestors,[net_sup,kernel_sup,<0.9.0>]},{messages,[]},{links,[<0.17.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,24},{reductions,822}],[]]}
{error_logger,{{2012,2,27},{18,8,41}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{"Too short cookie string",[{auth,init_cookie,0},{auth,init,1},{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},{offender,[{pid,undefined},{name,auth},{mfargs,{auth,start_link,[]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]}
{error_logger,{{2012,2,27},{18,8,41}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,shutdown},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2012,2,27},{18,8,41}},std_info,[{application,kernel},{exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}}"} 

The solution was simple. Checking in the data folder I saw the .erlang.cookie was of size zero:
# ls -al /var/lib/rabbitmq
total 272
drwxr-xr-x  3 rabbitmq rabbitmq   4096 Feb 27 18:05 .
drwxr-xr-x 28 root     root       4096 Feb 27 18:04 ..
-r--------  1 rabbitmq rabbitmq      0 Feb 23 16:45 .erlang.cookie
-rw-r-----  1 rabbitmq rabbitmq 213540 Feb 27 18:08 erl_crash.dump
drwxr-xr-x  4 rabbitmq rabbitmq   4096 Feb 23 16:44 mnesia

Simply remove the cookie file and start rabbitmq-server again and it succeeds.
# rm /var/lib/rabbitmq/.erlang.cookie