Saturday, February 2, 2008

Xen VM's & NAT Bridging

I have a dedicated box at an ISP that provide me a base IP for the box and a /29 range. I'm using Xen to create VM's on this box and I wanted to maximize the use of IP's for my hosts.

The default configuration one would use would result in 2 IP's being assigned to the Xen host.

Lets say I have a default IP of 1.2.3.4 on the main box, and the ISP has issued me 5.6.7.24/29, leaving me with 5.6.7.25-29 as usable IP's. One of the /29 range has to be applied to the base box to serve as a gateway, meaning that one IP is wasted (or you could say even 2 are wasted if you include the 1.2.3.4 address).

I didn't want to loose these 2 valuable IP's, so I used iptables with SNAT/DNAT to make use of all the assigned IP's.

The first steps were simple when I found this page: http://grml.org/xen/, so the bulk of the following code is copied from there.

Using a debian system, I entered the following extra config into my /etc/network/interfaces to make the Xen host listen to my public IP's and perform DNAT/SNAT to private IP addresses.
auto xenintbr
iface xenintbr inet static
pre-up brctl addbr xenintbr
post-down brctl delbr xenintbr
address 10.1.1.1
netmask 255.255.255.0
bridge_fd 0
bridge_hello 0
bridge_stp off

auto eth0:25
iface eth0:25 inet static
address 5.6.7.25
netmask 255.255.255.248
post-up iptables -t nat -A POSTROUTING -o eth0 -s 10.1.1.25 -j SNAT --to 5.6.7.25
post-up iptables -t nat -A PREROUTING -d 5.6.7.25 -j DNAT --to-destination 10.1.1.25


Next make xend set up the bridge correctly in /etc/xen/xend-config.sxp. This tells xend to set up the bridge between the guest and the bridge setup above, but instead of using the default network-bridge, we simply do routing, and let the iptables rules above perform the routing to the outside world.
(network-script    network-route)
(vif-bridge xenintbr)
(vif-script vif-bridge)


And in the guest machine configure the network on the private subnet.
auto eth0
iface eth0 inet static
address 10.1.1.25
netmask 255.255.255.0
gateway 10.1.1.1


You can check the status of the iptables rules with:
iptables -L -vn
iptables -t nat -L -vn

Wednesday, January 16, 2008

IPMI Reference

You can find the IP address of the ilom using the ipmitool on many systems:

# modprobe ipmi_si && modprobe ipmi_devintf
# ipmitool lan print 1

Sunday, January 6, 2008

Resizing Xen disk images

If you're in a situation where you're using disk images instead of LVM then you may occasionally need to resize your disks. It's really simple when you know how ;)

dd if=/dev/zero of=disk.img bs=1MB count=0 seek=20480
e2fsck -f -y disk.img
resize2fs -p disk.img


This will resize an existing image file to 20Gb.

Thursday, December 20, 2007

Vim modelines

Regularly I've wondered why vim modeline settings don't take effect when they're embeded in a file.

Historically modelines have opened up vim to security exploits, so the default is for modeline parsing to be disabled. You can check the status in vim with

:set modeline? modelines?
nomodeline
modelines=0


This is the default setting and can be overridden in your ~/.vimrc file like this

:set modeline modelines=20


Now a modeline appearing in a file will be parsed, so something I find useful for python scripts is:

# vim:ts=4:sw=4:sts=4:expandtab

Tuesday, December 4, 2007

mkfs.xfs tweaks

A couple of tweaks I'm using these days for XFS are:

mkfs.xfs -l size=64m /dev/md3


And in /etc/fstab mount the partition with the following options:

/dev/md3    /srv    xfs    noatime,nodiratime,logbufs=8    0 0


For a discussion of why these options see the bearded man.

Screen and backspace

Aarrrgghh! The backspace in screen doesn't work out of the box!

It's really simple to fix, but rather annoying. I'm using Debian and Ubuntu and SSH to my boxes from OSX.

The steps to clean this are:

echo 'bindkey ^? stuff ^H' >> /etc/screenrc


You may also want to know you're in screen from the prompt. Add this to the beginning of your PS1:

$(test $TERM == 'screen' && echo '{screen}')


Finally, if you have a PROMPT_COMMAND variable, you may want that to be run from within screen too.

Friday, October 19, 2007

HowTo Gentoo on Bytemark Xen host

Setting up a gentoo instance on a Bytemark xen system is simplified by the tools and image that Bytemark (www.bytemark.co.uk) have provided. There are a couple of catches however.

First thing to do is log into your Xen master and create your DomU:

xen-create-image --hostname=DomU_Hostname --ip=DomU_IP --dist=gentoo --passwd --boot


The installation will stop giving feedback for a few minutes. Be patient and the DomU will start to boot.

When the image boots you'll see your problem: the Gentoo init scripts will try to mount /proc and /sys, however these are already mounted by Xen.

Due to the problems you will not have been given the opportunity to set your root password, so you can't log in for maintenance. The solution is to add the following line to the xenmachine.cfg for your DomU:

extra = "init=/bin/sh"


Then boot the DomU and you'll be presented directly with a shell prompt. Now you can fix the /proc and /sys boot issue.

First remount the root as rw:

mount -o remount,rw /


Then edit /sbin/rc and comment out the code that mounts /proc and /sys.

When you've saved the file you need to shutdown gentoo and remove the "extra.." line you added to the xenmachine.cfg, then start your DomU again and it'll boot to the login prompt.

Sunday, September 23, 2007

Debian Sarge from VMWare to Xen DomU

Now I have a good Gentoo Xen instance, I want to migrate my VMWare environments to Xen DomU's.

My mobo is the Via Epia EN12000 with a Via C7 Esther CPU, and only has the cmov CPU flag, so no chance of getting HVN working. I had to build the standard Gentoo Xen Linux Sources at version 2.6.20. That's better than the stock Xen Kernel which is currently 2.6.18, but it still doesn't have specific support for the Via C7. Not a problem though.

I have an old Debian Sarge that hasn't been updated in a while. To get it working I had to add the sarge-backports source to /etc/apt/sources.list. Once that was done it was still a bit of a problem getting a Xen kernel installed. In the end I figured out the dependencies to work.

Aptitude didn't give very helpful details so I had to get the info I needed by using "dpkg -i" on packages. Eventually the following commands got me where I wanted to get:

aptitude install klibc-utils
aptitude -V -f install udev=0.105-4~bpo.1
aptitude -V -f install initramfs-tools
aptitude -V -f install linux-image-2.6-xen-686


The key to this seems to have been the packages removed and installed by klibc-utils. After that you could probably skip to installing linux-image-2.6-xen-686.

For reference, the main error from aptitude that was killing me was:

The following packages have unmet dependencies:
initramfs-tools: Depends: udev (>= 0.086-1) but it is not installable


even though udev-0.105-4~bpo.1 was installed.

Once the kernels were installed I tried to boot the domain in Xen and faced another problem. The linux-image-2.6.18-4-xen-686 in Sarge Backports now supports PAE, so I had to rebuild my dom0 kernel and apps-emulation/xen with PAE support.

Tuesday, September 11, 2007

Creating a filesystem on a RAID array

I have a linux software RAID5 array. I want to make sure I create an optimized filesystem on the array.

The array details are:

# mdadm --misc -D /dev/md3
/dev/md3:
Version : 00.90.03
Creation Time : Sat Oct 21 19:32:23 2006
Raid Level : raid5
Array Size : 735334848 (701.27 GiB 752.98 GB)
Used Dev Size : 245111616 (233.76 GiB 250.99 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 3
Persistence : Superblock is persistent

Update Time : Tue Sep 11 19:18:42 2007
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : 6f6b9978:683a7ada:109f3c19:b8abb3ae
Events : 0.53048

Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 1 3 active sync /dev/sda1


Here we see a Chunk Size of 64K and 4 disks in the array.

Based on the information in this discussion thread, this means that the stride is 16 based on a Chunk Size of 64k accross 4 disks with a filesystem with a 4k block size.

So a good mkfs.ext3 command would be
mkfs.ext3 -b 4096 -E stride=16 /dev/vg/lvol01

Monday, September 3, 2007

Gentoo GCC-3.3.6 and mplayer or xine

Got a new VIA EPIA EN12000E mobo over the weekend and plan to set it up as a set top box and file server, among other things. It's a pretty nifty little gadget and best of all has no fan.

I'm building a new gentoo instance on it. So far I've built the core OS on a mirror RAID pair of drives and plan to move 4 SATA drives to it when all the services are completely built and it works with my 1080i HDTV. For now I'll probably just stick to using the VGA interface, hoping that'll be easiest.

I've had a few hurdles to get through during the build. For example, I wanted to do a network boot and install, but the gentoo network boot and install instructions weren't clear. So I reverted to an Ubuntu network boot I set up a while ago and went through the rescue interface.

I set up a RAID1 pair for the root partition, and LVM over RAID1 for a few other partitions. Considering my first gentoo installation is still going strong since being built in 2002, I figure I'll make sure this one lasts too.

It took a while to get going. I generally remember how to get started with the gentoo install since I started doing it years ago when we would start from stage1 (if fanatic enough). Unfortunately the Ubuntu 7.04 rescue boot doesn't provide SSHD so I had to get the stage3 and portage installed and chrooted before I could get SSH to the box.

Once that was done I tweaked the CFLAGS and USE flags for the hardware and did the bootstrap.sh to make sure it's fully optimized for the hardware. I then built a custom kernel (2.6.22.6) to enable most of the embeded hardware that I'll be using to start with.

When I booted the new kernel none of the LVM partitions mounted. Duh! I forgot to emerge the lvm2 package. Luckily LVM was compiled in the kernel, so I just copied the static vgscan and vgchange binaries from my other box and started the LVM partitions up.

Next I diff'd the /var/lib/portage/world file to see what I want from my old box and started working through the USE flags. I noticed a dependancy for gcc-3.3.6. The core system already came with gcc-4.1.2 so why should i put an old gcc-3 on it? It turns out that all that is needed is to 'emerge libstdc++-v3'. Note that the package is not simply libstc++.

I'm not sure why libstdc++-v3 isn't included in the 2007.0 stages since they're packaged with gcc-4. I eventually found the answer on an IRC log (http://dbforms.oxxus.net/dependency-question-appemulationwine-world-corefonts/index.php), which references the Gentoo GCC Upgrade Guide.

Next I have ccache and distcc set up to speed up the builds and I must wait a couple of days for everything to install. I hope the configs won't be too complicated and I'm looking forward to seeing the performance of this system.