Grow linux md raid5 with mdadm --grow

Growing an mdadm RAID array is fairly straight forward these days. There's a few limitations, depending on your setup, and I strongly recommend you read the mdadm man page in addition to the notes here.

A couple of the limitations include:

• raid arrays in a container can not be grown, so this excludes DDF arrays
• arrays with 0.9x metadata are limited to 2Gb components - the total size of the array is not affected though
  

Before you start it's a good idea to run a consistency check on the array. Depending on the size of the array this can take a looong time. On my 3 x 1Tb RAID5 array this usually takes around 10 hours with the default settings. You can explore tweaking the settings, though I haven't done this for checks yet. We will see how we can tweak the settings for the reshape later on.

Running a consistency check is done as follows. I don't have the sample mdstat output at this time but have included the command for consistency.

# echo check >> /sys/block/md4/md/sync_action
# cat /proc/mdstat

You'll see if any errors were corrected in the array parity in the dmesg output and/or kernel logs.

Once the check is complete you should be safe to grow the array. First you have to add a new device to it so there is a spare drive in the set.

mdadm --add /dev/md3 /dev/sdc1

The event will appear in the dmesg output and the spare will show up in mdstat:

# dmesg
md: bind

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sdc1[s] sdb1[0] sda1[2] hdd1[1]
      1953519872 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

Now the spare is there - you can give the command to grow the array:

# mdadm --grow /dev/md3 --backup-file=~/mdadm-grow-backup-file.mdadm --raid-devices=4

The array now starts reshaping. You can monitor progress:

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sdc1[3] sdb1[0] sda1[2] hdd1[1]
      1953519872 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  reshape =  3.3% (33021416/976759936) finish=1421.0min speed=10341K/sec

In dmesg you should see something like this:

# dmesg

RAID5 conf printout:
 --- rd:4 wd:4
 disk 0, o:1, dev:sdb1
 disk 1, o:1, dev:hdd1
 disk 2, o:1, dev:sda1
 disk 3, o:1, dev:sdc1
md: reshape of RAID array md3
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
md: using 128k window, over a total of 976759936 blocks.

Before tweaking the speed settings, now is a good time to edit your /etc/mdadm.conf file with the new ARRAY changes so it's recognized and started on your next reboot.

Now we can tweak the speed settings to speed up the reshape time. I played around with a few settings and found the following to be good for my own system.

# echo 8192 >> /sys/block/md3/md/stripe_cache_size
# echo 15000 >> /sys/block/md3/md/sync_speed_min
# echo 200000 >> /sys/block/md3/md/sync_speed_max

On my system this cut about a third off the predicted finish time:

# cat /proc/mdstat

Personalities : [raid1] [raid6] [raid5] [raid4]
md3 : active raid5 sdc1[3] sdb1[0] sda1[2] hdd1[1]
      1953519872 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  reshape =  4.0% (39983488/976759936) finish=957.6min speed=16303K/sec

It seems values greater than 8192 for stripe_cache_size were more harmful than beneficial on my system. It's not clear to me if this is CPU bound or bandwidth to the drives, though looking at older posts I suspect both can play a roll.

Also note that reducing the stripe_cache_size may not occur immediately when you echo a smaller value to the file. I had to echo smaller values several times before the value was adopted. This was on kernel 2.6.32.7.

You can monitor the stripe_cache_active file to see how filled the cache is:

# cat /sys/block/md3/md/stripe_cache_active
7136

When the reshape is complete you will still need to grow the file system (or volume group if you use LVM) contained in there. I'll document that tomorrow when my reshape is complete ;)

Virtual Inbox in Thunderbird

Right - I've finally figured out how to set up a virtual inbox in ThunderBird3 that centralizes messages from multiple accounts and folders into a single location.

1. Select an exsting folder on an existing IMAP account
2. From the menu select File > New > Saved Search
3. Decide where you want to keep the Virtual Inbox. It can't be top-level, so I choose Local Folders as the parent.
   
4. Name the folder, such as Virtual Inbox.
5. Use the Choose button to select the source folders of your messages
6. Select "Match all messages"
7. Done

Hack OTA installation of BlackBerry applications

Finally found a way to install BB apps when the data plan doesn't allow the BB browser to work properly.

I wrote a little bash script to do the dirty work for me. Works a treat for me :) But I can't guarantee it won't hose your phone :p

The script can be retreived from github at http://github.com/jinnko/get-bb-ota-install/blob/master/get-bb-ota.sh

Hard disk upgrade quick reference

Howto copy entire system to a new disk


echo cp -a `/bin/ls -1Ab | egrep -v '^(new|dev|proc|vols|sys)$'` /new/


Speeding up Firefox on OSX

Wow! There's two very simple actions that will speed up firefox.

After some time of using firefox the address bar can slow down considerably. SpeedyFox is an extension that claims to speed up the browser, but there's no need for an extension - it's simple enough to do manually.

Here's how you do it.

1. Quit firefox.

2. Open your favorite Terminal.app or iTerm.app

3. Change to your profile directory
cd ~/Library/Application Support/Firefox/Profiles/your_profile_code.default

4. Issue the command to vacuum your sqlite db's
for x in *.sqlite; do echo 'VACUUM;' | sqlite3 ${x}; done

That's it! Next time you start firefox it'll be faster.

And the second tweak you can make is to reduce the page render delay. I guess firefox has this since it assumes pages take a while to come in - but I don't see the point.

1. Open up the about:config area:

2. Edit the print.print_pagedelay config item, and set it to 0 (zero). This tells firefox to start drawing the page immediately instead of waiting half a second.

git svn checkout

git-svn checkouts are pretty simple, but here's a reference anyway:

Checking out the SVN tree within GIT

$ mkdir git-svn-project
$ cd git-svn-project
$ git svn init https://svn.server.url/repos/svn-project
$ git svn fetch

Staying up-to-date with upstream SVN commits

$ git svn rebase

Commiting back to SVN

$ git commit -a
$ git svn dcommit

git-svn and failed svn commit hooks

The workflow of using GIT with SVN can be a steep learning curve. After the easy part of getting an SVN checkout within GIT, you could end up in a situation where an SVN pre-commit hook fails, leaving your in a bit of a muddle.

The solution to this may not be immediately obvious, so it's necessary to understand what GIT is doing under the hood.

When you have a git-svn checkout, git maintains the svn repository state in a remote branch. You work within your local branch and commit changes locally. When you're ready to commit back to SVN you'd normally just do a "git svn dcommit".

If you make some local git changes and commit locally, then do the dcommit that is rejected by a pre-commit hook, then normally in SVN you would fix the error locally and re-try your commit. In GIT the commits have been separately checked-in locally, so GIT will try to commit each change separately to SVN - but this won't work because the first local commit was rejected by SVN.

The solution is to create a new git branch, forked from the current branch just before your rejected commit. To do this you would first want to look at "git log" to get the commit hash from git, then create your new branch and change to it:

$ git log
  # find the commit hash you're after
$ git branch temporary_branch 68b8976ba0944
$ git checkout temporary_branch
$ git merge --squash dd7354fcd397 7f5e78f7e737ed4 5e07bf3c6f5c15
$ git svn dcommit

You'll then want to go back to your local master git branch, and likely clean up after yourself:

$ git checkout master
$ git svn rebase
$ git reset --hard HEAD
$ git branch -D temprorary_branch

Disclaimer: You'll want to check the last "reset --hard" and "branch -D" commands do lose anything before running them. I.e. don't do anything here without understanding it first!

Locale problems

This has happened too many times - so here's a note: http://www.peterryan.net/2007/11/05/perl-locale-is-corrupted-after-gutsy-upgrade/

EDIT: The target site appears to break, so here's the fix:

 sudo locale-gen
sudo dpkg-reconfigure locales

Thanks to armware on December 12th, 2006, 01:00 PM for identifying this.

Changing root users' shell in Mac OSX Leopard 10.5

Leopard changes the way users are managed on Leopard. Some online resources say you have to enable root, then you can change the users' shell via the System Preference, however that didn't work for me.

Directory Utility never connected to any directory, and thus didn't work as a solution for Enable Root. The solution here is to use dsenableroot on the command line.

Then the obvious sledgehammer for editing users accounts, vipw, didn't work. The next obvious tool for editing the shell was chsh and that did the job.

Xen and keeping time: NTP, DomU's and hwclock

There's plenty of discussion out there on keeping time in Xen DomU's. Unfortunately there is no one definitive source of information.

After much research I've come to the conclusion that the following is the best way to keep all Dom clocks synchronized:

• Set independant_wallclock=0 on all Dom's
• Run NTP in Dom0 only (not OpenNTP, and not in DomU's)
• Cron "hwclock --systohc" to run daily or twice daily, but never more than once every 55 minutes.
  

Here is a more in depth explanation of the possible ways of keeping time, and the reasoning behind using the above solution.

1. The default method for keeping time is for the DomU's to get their time directly from Dom0.
2. An alternative methods involves setting each DomU to have an independant wallclock, then to run NTP in the DomU's

I used the second method fairly successfully for a long time, but then some flaws in this solution became apparent. Primarily, Dovecot IMAP/POP3 would constantly complain that TimeWentBackwards. This highlighted to me that using an independant wallclock like this wasn't working well. While other processes didn't complain, I can imagine that a changing clock could easily confuse several things.

I then set the DomU's back to the default independant_wallclock=0, disabled NTP on the DomU's and only ran NTP in Dom0. Unfortunately there was still a significant drift in the DomU's even though Dom0 was correct, however the DomU's all seemed to have very similar drive relative to the Dom0. There are a couple of reasons for this.

OpenNTP doesn't set the Dom0 clock in the same way that NTPd does, and so NTPd works out of the box much better. But this still didn't solve the problem. It seems that something changed between ntpd and/or kernel versions.

I believe ntpd used to make use of the "11 minite mode" outlined in the hwclock man page. After some investigation it turned out this was not the case, and can be confirmed by using "adjclockx --print". If the status value isn't 64, or doesn't have the 64 bit set, then "11 minute mode" is on. Despite various sources indicating that ntpd would automatically turn on "11 minute mode", it didn't. Further investigation seems to indicate that "11 minute mode" is not a Good Thing(tm), so I'm guessing that's why it doesn't get set when ntpd is started. A greater discussion about this can be found here among many other places.

So we now know that NTPd is prefered over OpenNTP, and that using hwclock is prefered over "11 minute mode". And that's why I use the settings outlined above.

Convert cue bin files to iso on linux

When faced with cue/bin files the first reaction is to convert them to something you can use.  Essentially this isn't necessary.

If the contents of the cue/bin files is a movie or audio, mplayer can read the files natively:

mplayer "cue:///path/to/image.cue:track"

If you do need to access the contents of the raw image, then cdemu is your friend.  Once installed you need to start the cdemud daemon, then use the cdemu tool to expose your raw image on a new cdrom or dvd device, such as /dev/dvd2.

root@ # /etc/init.d/cdemud start
user@ $ cdemu -b system load 0 "/path/to/image.cue"
user@ $ mount -t iso9660 /dev/dvd2 /mnt/dvd2

Like this you have no need to perform any time consuming conversions with tools such as bchunk or bin2iso which proved to be unreliable for me.

If you want to burn the cue/bin files to disk - cdrdao should be able to do that, though I haven't tested it.

Opsview between i386 and amd64

This is something you should never do!  But somehow I got myself into this situation, what a PITA.

My master server is i386, and the slave is amd64. 

These are the step to get it working - and have to be done each time you send updates to the slave.  So basically this is temporary until I have time to migrate my master to amd64.

• Make sure all opsview apps are not running on either server
• Install ia32libs on the slave
• Upgrade to the latest opsview on the master.
  
• Upgrade the slave to the latest version.
• At this point your dpkg --configure -a will have failed - do the following
• Edit /var/lib/dpkg/info/opsview-core.postinst and comment out the restart actions at the bottom
• On the slave - go into /usr/local/nagios/bin/
• chmod 700 nd02db; cp nd02db ndo2db.ARCH
• Back on the master run dpkg --configure -a.  Just after it finishes copying the tar.gz to the slave - do this on the slave: cp ndo2db.ARCH ndo2db
• Now your master will think it's succeeded finally.
• Go to the slave and aptitude reinstall opsview-core opsview-base opsview-perl.
• Again the dpkg --configure -a will fail.  Edit /var/lib/dpkg/info/opsview-core.postinst as before.
• Run dpkg --configure -a on the slave and you should be fine
• Make sure all your services are up and running.  Slaves should have opsview and opsview-agent running.

OSX Automator, USB Drive Insertion or Mounted Volumes & Backup

I was looking for a way to trigger an event when a volume is mounted on my system when I came accross this auto detect mounted volumes post. I'm completely new to automator, so all I want it to do is kick off a shell script.

Now the shell script runs whenever any volume is mounted. But what happens to stdout & stderr when the script is triggered by automator? Well - it vanishes, so you need to redirect it somewhere useful. Early on in my script I redirect stdout and stderr to a log file:

exec > ~/Library/Logs/mount-event.out 2>&1

The script will not receive any incoming content or variables and the current working directory will be they users' home, so you just need to start from scratch.

MacOSX syslog & asl.db system slowdown

I noticed that if Firefox goes apeshit it chew up memory and log repeated errors to syslog that would cause the system to compete for disk I/O.

Aside from Firefox just being crap and loosing it, I found that I could prevent syslog from logging too much crap by reducing the logging level from NOTICE (5) to ERROR (3) - see man 1 syslog for other levels. These changes were inspired by a CodeSnippets post, though the instructions there appear to be far too complex than is needed.

Here's how to do it:

sudo /usr/libexec/PlistBuddy -c "Delete :ProgramArguments" /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo /usr/libexec/PlistBuddy -c "Add :ProgramArguments array" /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo /usr/libexec/PlistBuddy -c "Add :ProgramArguments:0 string '/usr/sbin/syslogd'" /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo /usr/libexec/PlistBuddy -c "Add :ProgramArguments:1 string '-c'" /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo /usr/libexec/PlistBuddy -c "Add :ProgramArguments:2 integer 3" /System/Library/LaunchDaemons/com.apple.syslogd.plist

Then you just need to restart syslog:

sudo launchctl stop com.apple.syslogd
sudo launchctl start com.apple.syslogd

Debian Etch, mod_fcgid & php4

There are a fair few howto's out there with instructions on getting php4 & fcgid working together, but it seems none of them have been tested properly. PHP5 works easily following the instructions, but not PHP4 - so here's my extra notes to get it working.

I had to strace apache a few times to figure out exactly what was going on since it fcgid doesn't provide any useful logging. This can easily be done with "strace -f -p {pid_of_the_apache_process_answering_the_request} -e read=all -e write=all -o strace_output.txt" - this will give you something to look for when debugging.

The main difference between most of the existing howto's is the exec line in the php-fcgi-wrapper. Although "apt-get install php4-cgi" provides you with /usr/bin/php4-cgi - and it seems you would use this, you _don't_ use it. You need to use /usr/lib/cgi-bin/php4. You should be able to figure out the rest with the existing howto's.

ClusterSSH in Leopard

Unfortunately the DarwinPorts way of installing ClusterSSH doesn't work, so you gotta do it manually. Good news is you don't have to re-invent the wheel. Someone has kindly posted the procedure at: http://lists.macosforge.org/pipermail/macports-users/2008-January/008336.html

EpiaWiki > EpiaSound > Setting up Alsa

The following content is captured from web.archive.org because the original domain no longer exists. I copy the content here for reference as I found this useful when setting up my own Via EPIA EN12000 S/PDIF 5.1 system with Gentoo linux.

The content below is provided as-is, and is unchanged from the original text. If it doesn't work for you, then your guess is as good as mine as to why.

---

3.1 Setting up Alsa

Make sure your /usr/src/linux link points to the correct kernel, as Alsa and other packages use this to get the correct headers and modversions. Run emerge alsa-utils to get required alsa tools and the init script. If you are using the 2.4 kernel, you will need to run "ALSA_CARDS=via82xx ACCEPT_KEYWORDS=~x86 emerge alsa-driver" to get the drivers (2.6 includes the drivers). It is helpful to add ALSA_CARDS="via82xx" to your /etc/make.conf and "media-sound/alsa-driver ~x86", etc to your /etc/portage/package.keywords so you don't have to specify them every time.

Please refer to the official Gentoo ALSA guide (cache) if you have any questions.

3.1.2 Configuring Alsa

Now we need to configure it to auto start.
Here are some relavent via82xx module options:

ac97_clock  - AC'97 codec clock base (default 48000Hz)
dxs_support - support DXS channels,
             0 = auto (defalut), 1 = enable, 2 = disable,
             3 = 48k only, 4 = no VRA
             VIA8233/C,8235 only

The default of dxs_support=0 usually results in wierd sounding non-48Khz files so try one of the other options. Now we need to edit /etc/modules.d/alsa it should look something like this:

# ALSA portion
alias char-major-116 snd
alias snd-card-0 snd-via82xx
options snd-via82xx dxs_support=3

# OSS/Free portion
alias char-major-14 soundcore
alias sound-slot-0 snd-card-0

alias sound-service-0-0 snd-mixer-oss
alias sound-service-0-1 snd-seq-oss
alias sound-service-0-3 snd-pcm-oss
alias sound-service-0-8 snd-seq-oss
alias sound-service-0-12 snd-pcm-oss

alias /dev/mixer snd-mixer-oss
alias /dev/dsp snd-pcm-oss
alias /dev/midi snd-seq-oss

options snd cards_limit=1

Then run modules-update after which, you can type /etc/init.d/alsasound start to insert the alsa modules. rc-update add alsasound boot will load that every time you reboot.

3.1.3 Unmuting Alsa

Alsa is muted by default. Run alsamixer to view the mixer levels. Move left and right with the arrow keys, and press 'M' to unmute Master and PCM channels. Up and Down will adjust the volume levels. Press 'Esc' to exit. The alsasound boot script will restore mixer settings on reboot.

3.1.4 6 Channel Output (5.1)

Now how to switch to 5.1 out...

Once you switch the sound card to output 5.1, the functions of the jacks change.
Jack : Mode 1/Mode 2
Blue : Line-In/Rear-Out
Green : Line-Out/Front-Out
Pink : Mic-In/Center&Subwoofer-Out

3.1.5 Using the S/PDIF out

The EPIA-M can output S/PDIF through the RCA jack on the back. First you need to change a jumper on the motherboard before you can use this, it is near the RCA port, see your manual for details.

The S/PDIF output is controled through the IEC958 Playback AC97-SPSA mixer channel you can set the different options with the command amixer set 'IEC958 Playback AC97-SPSA' 0. The S/PDIF always outputs at maximum volume, the main and pcm mixers have no affect on it. Here are the different output options for IEC598 Playback:

0. PCM1
1. PCM2,PCM1 (rear)
2. Center and LFE
3. PCM3,Modem,Dedicated S/PDIF

Most people want to use 0.

To get AC3 passthrough with mplayer, use mplayer -ao alsa -ac hwac3

Also, if you haven't set the above mixer setting, then you should be able to use mplayer -ao alsa9:iec958 to get s/pdif output still.

3.2 OSS Kernel Module

The OSS sound driver is included in the kernel. Enable "VIA 82C686 Audio Codec" under sound in the menuconfig. The module will be called via82cxxx_audio.

3.3 VIA Combo Audio Driver

http://www.viaarena.com/?PageID=294 (cache) offers a oss driver that supports all of VIA's sound chips. It supports six channel audio and spdif out.

You can get viaaudiocombo for 2.6.1 here:
http://www.shipmail.org/~thomas/via/audio/ (cache)

this link is dead :-( however, you can still look at the cached copy and from there download the file

Xen VM's & NAT Bridging

I have a dedicated box at an ISP that provide me a base IP for the box and a /29 range. I'm using Xen to create VM's on this box and I wanted to maximize the use of IP's for my hosts.

The default configuration one would use would result in 2 IP's being assigned to the Xen host.

Lets say I have a default IP of 1.2.3.4 on the main box, and the ISP has issued me 5.6.7.24/29, leaving me with 5.6.7.25-29 as usable IP's. One of the /29 range has to be applied to the base box to serve as a gateway, meaning that one IP is wasted (or you could say even 2 are wasted if you include the 1.2.3.4 address).

I didn't want to loose these 2 valuable IP's, so I used iptables with SNAT/DNAT to make use of all the assigned IP's.

The first steps were simple when I found this page: http://grml.org/xen/, so the bulk of the following code is copied from there.

Using a debian system, I entered the following extra config into my /etc/network/interfaces to make the Xen host listen to my public IP's and perform DNAT/SNAT to private IP addresses.

auto xenintbr
iface xenintbr inet static
  pre-up brctl addbr xenintbr
  post-down brctl delbr xenintbr
  address 10.1.1.1
  netmask 255.255.255.0
  bridge_fd 0
  bridge_hello 0
  bridge_stp off

auto eth0:25
iface eth0:25 inet static
  address 5.6.7.25
  netmask 255.255.255.248
  post-up iptables -t nat -A POSTROUTING -o eth0 -s 10.1.1.25 -j SNAT --to 5.6.7.25
  post-up iptables -t nat -A PREROUTING -d 5.6.7.25 -j DNAT --to-destination 10.1.1.25

Next make xend set up the bridge correctly in /etc/xen/xend-config.sxp. This tells xend to set up the bridge between the guest and the bridge setup above, but instead of using the default network-bridge, we simply do routing, and let the iptables rules above perform the routing to the outside world.

(network-script    network-route)
(vif-bridge        xenintbr)
(vif-script        vif-bridge)

And in the guest machine configure the network on the private subnet.

auto eth0
iface eth0 inet static
  address 10.1.1.25
  netmask 255.255.255.0
  gateway 10.1.1.1

You can check the status of the iptables rules with:

iptables -L -vn
iptables -t nat -L -vn

IPMI Reference

You can find the IP address of the ilom using the ipmitool on many systems:

# modprobe ipmi_si && modprobe ipmi_devintf
# ipmitool lan print 1

Resizing Xen disk images

If you're in a situation where you're using disk images instead of LVM then you may occasionally need to resize your disks. It's really simple when you know how ;)

dd if=/dev/zero of=disk.img bs=1MB count=0 seek=20480
e2fsck -f -y disk.img
resize2fs -p disk.img

This will resize an existing image file to 20Gb.