michaelanthoni.com

So I’ve been using the Ubuntu 8.04 LTS Desktop Edition on my laptop since its release and I must say that its a huge improvement since their last LTS release which I was running prior. I was previously on a dual boot configuration but I have been so satisfied with the current state of Ubuntu that I have not felt the need to boot into Windows at all. Anything I need to do on Windows has been far and few between lately and terminal services is just a click away, not to mention wine 1.0, wow has that come along way. Sure this version of ubuntu has its bugs but what doesn’t, and its not necessarily the bugs that bug me, but its the annoyances that get under my skin, sometimes caused by bugs or just design issues or lack of features. The previous Ubuntu LTS was filled with so many annoyances that it drove me crazy sometimes, such as WPA support out the box, buggy suspend/hibernation modes,ipod recognition, audio/video/codec annoyances and numerous other minor issues that just left me with a bad taste in my mouth. Their has been very little that has annoyed me with Ubuntu 8.04 LTS and I have never been a huge fan of Ubuntu, but it was the lesser of three evils. I still haven’t gotten to the point where I feel I need to migrate all my debian installations to ubuntu, but this desktop lts version might persuade me to at least give the ubuntu server version another shot. I had a horrendous experience with the last ubuntu server version and I havent looked back since. I dont remember what the exact issues were but debian was just so much less of a headache. And with the lenny branch frozen with a pending release in september this may be a good time to start evaluating my options that will be runnin on my servers for the next couple of years. So thanks to the Ubuntu/Debian teams for releasing a great product. Its been a pleasure.

So its been a few weeks since most of us have patched our vulnerable dns servers, but I hadn’t noticed this little bonus until today which actually made me laugh. You see a few years ago I had noticed an annoying little behavior with the way Microsoft’s DNS Server handles outgoing client connection for domains/servers that are listed under the Forwarders tab. We use this Forwarders tab to list frequently queried domains in which we host a copy of the zone file in rbldnsd so as to not go to the internet to find the answer to. This gives us the benefit of returning an answer to a dns query much faster and saves us the extra bandwidth. This is highly beneficial to our mail systems which process on average 100 million messages per month, mostly spam of course. So back when we had implemented the rbldnsd system, we had placed Linux Virtual Server in front of the rbldnsd to load balance the traffic accross 8 or so machines. After pointing the Forwarded domains to the LVS VIP, I had expected hundreds, even thousands of connections to get sprayed accross the rbldnsd farm, but uh-uh, nope. There were only two connections listed to two of the backend servers, however all the queries were getting answered.

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 1
-> server2.domain.com:domain Route 2 0 0
-> server3.domain.com:domain Route 2 0 0
-> server4.domain.com:domain Route 2 0 1
-> server5.domain.com:domain Route 2 0 0
-> server6.domain.com:domain Route 2 0 0
-> server7.domain.com:domain Route 2 0 0
-> server8.domain.com:domain Route 2 0 0

This had me scratching my head at first and then after a few packet captures later, I realized that Microsoft was opening 1 socket connection and pushing all the forwarded queries through it. Gee Wiz Microsoft! Why would you do such a thing? I figured that opening and closing socket connections carries an overhead and could also potentially exhaust all available udp ports in a very short amount of time, I can understand why Microsoft would implement it in this way. However this is exactly the insufficient socket entropy that is described in Dan’s advisory as flawed, and from my perspective I hated it as I couldn’t load balance all the forwarded dns queries across each machine that had rbldnsd running on them. Luckily rbldnsd wasn’t the primary service on those machines that we were load balancing so I had decided to just let it be after a spending a few minutes of looking for a workaround then banging my head on my desk out of frustration. Availability was still guaranteed and rbldnsd being as fast and memory efficient as it was, was performing fine in this configuration, so I let it be. I had bigger fish to fry at the time. Fast forward a few years later and a Dan Kaminsky patched Microsoft DNS Server, and wallah, this is what I noticed today…

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 264
-> server2.domain.com:domain Route 2 0 258
-> server3.domain.com:domain Route 2 0 256
-> server4.domain.com:domain Route 2 0 252
-> server5.domain.com:domain Route 2 0 250
-> server6.domain.com:domain Route 2 0 252
-> server7.domain.com:domain Route 2 0 252
-> server8.domain.com:domain Route 2 0 252

and this is with a modified udp timeout of 10 seconds…

me@director:~$ sudo ipvsadm -L --timeout
Timeout (tcp tcpfin udp): 60 10 10

Awesome, entropy, security, and load balancing :). Thanks Dan!

If you need access to any of the ctype functions such as ctype_alnum() in any of your php applications and you are running gentoo, make sure you add the ctype use flag or you will get this lovely error :).

Well I’ve been using cfengine for a number of years now and thought I had paid my dues already when I initially took on its steep learning curve… Well today I had a little run in with cfengine that made me feel as frustrated as when I was a newbie to this software, but I guess it was a newbie mistake that Im sure I learned years ago that I just happened to forget over the years when adding a cluster of new hosts to the mix - reverse dns.
The issue came about when I was configuring a new group of servers. I was on the final one when I simply installed cfengine on the host, scp’ed over cfagent.conf, cfservd.conf, and update.conf from a host that I had just been successful with. But after running “cfagent -v” i ran into the familiar “Can’t stat /var/lib/cfengine… in copy” which struck me as odd because it had just worked on all the other hosts. After checking the usual suspects such as the grant: function in the cfservd.conf to make sure permissions were explicitly granted on the server side, the hostname and domain name configured on the client, typos, cfkeys, cfservd started ?, etc, nothing seemed to work and adding the debug options -d seemed to frustrate me even more. As a last resort I took a packet capture to see what was going on between the client-> server for both the system that was failing and one that was working. I didnt think it would help much but sure enough after crawling through the capture packet by packet I seen the issue in one of the packets data field that looked something like this…

CAUTH IP IP user - non-working host
CAUTH IP hostname user - working host

This is when the little cfl lightbulb went off in my head and I decided to have a look at reverse dns. Sure enough all the hosts had reverse dns configured but this last one.

Although other functions such as directories,files,editfiles seemed to authenticate and run fine without reverse dns it seemed the copy function was failing because authentication under cfservd and the grant directive is based on the domain *.domain.com and not the IP… sheesh… it seems the parameter SkipVerify can be applied globally here and workaround hosts not having reverse dns, however I decided not to use this option since we control the reverse dns and it really should of been configured, not sure why it was not…

as soon as I added the reverse dns for the host cfagent ran without a hitch…

So i came across this the other day while trying to configure ldirectord to load balance pop3 services.

Can’t locate Mail/POP3Client.pm

It seems there is a missing dependency that is specific to debian etch I believe. I was a little disappointed as I’ve had little other issues with LVS and ldirectord but the fix was easy enough and I was able to find bug #421415 in Debian’s bug tracking system so Im sure I was not the first or the last to run into this. If you run into this just run apt-get install libmail-pop3client-perl and you should be good to go.

Slackware has been a staple in the linux community since the beginning and its great to see another release as this distribution evolves thanks to the hard work by Patrick J. Volkerding. Slackware 12.1 was released yesterday and boasts a 2.6.24.5 kernel, and of course the simplicity, stability and security we’ve always expected and recieved from this distro.

The rsync folks just recently released rsync 3.0 last month with a bug-fix release and a security release earlier this month. Unfortunately, after upgrading one of my critical systems that feeds a cluster of about 10 machines, I ran into an ugly little bugger that prevented my cluster’d nodes from successfully pulling there data from the central rsync machine resulting in stale files on the clustered nodes. Heres the error I seen when running my rsync manually:

$ rsync -t 10.9.8.7::module/* /dest
rsync: link_stat “/*” (in module) failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(1515) [receiver=3.0.2]

Instead of using the wildcard I tried one file specifically and that seemed to work just fine so I knew something was up with the wildcard thrown in there…After a little searching I confirmed my suspicions…

https://bugzilla.samba.org/show_bug.cgi?id=5388

Unfortunately this required a manual patch as the current version remains unpatched at the time of this writing and unavailable via package managers such as apt-get, portage, ports, yum, etc. Luckily this was easy enough as the patching and compilation was very smooth as I would expect…

#cd /usr/local/src/
#wget http://samba.org/ftp/rsync/src/rsync-3.0.2.tar.gz
#wget http://samba.org/ftp/rsync/src/rsync-patches-3.0.2.tar.gz
#tar -zxvf rsync-3.0.2.tar.gz
#tar -zxvf rsync-patches-3.0.2.tar.gz
#cd rsync-3.0.2
#patch util.c patches/util.c
#./configure
#make
#make install
#cp /usr/local/bin/rsync /usr/bin/rsync
#/etc/init.d/rsync restart

And once again my cluster’d nodes are happy again :).

The rsync folks just recently released rsync 3.0 last month with a bug-fix release and a security release earlier this month. Unfortunately, after upgrading one of my critical systems that feeds a cluster of about 10 machines, I ran into an ugly little bugger that prevented my cluster’d nodes from successfully pulling there data from the central rsync machine resulting in stale files on the clustered nodes. Heres the error I seen when running my rsync manually:

$ rsync -t 10.9.8.7::module/* /dest
rsync: link_stat “/*” (in module) failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(1515) [receiver=3.0.2]

Instead of using the wildcard I tried one file specifically and that seemed to work just fine so I knew something was up with the wildcard thrown in there…After a little searching I confirmed my suspicions…

https://bugzilla.samba.org/show_bug.cgi?id=5388

Unfortunately this required a manual patch as the current version remains unpatched at the time of this writing and unavailable via package managers such as apt-get, portage, ports, yum, etc. Luckily this was easy enough as the patching and compilation was very smooth as I would expect…

#cd /usr/local/src/
#wget http://samba.org/ftp/rsync/src/rsync-3.0.2.tar.gz
#wget http://samba.org/ftp/rsync/src/rsync-patches-3.0.2.tar.gz
#tar -zxvf rsync-3.0.2.tar.gz
#tar -zxvf rsync-patches-3.0.2.tar.gz
#cd rsync-3.0.2
#patch util.c patches/util.c
#./configure
#make
#make install
#cp /usr/local/bin/rsync /usr/bin/rsync
#/etc/init.d/rsync restart

And once again my cluster’d nodes are happy again :).

Now, Im one of those admins who despises running certain open source software packages on Windows. It just bugs me. Just because a certain application runs on Windows, or is reported to run on Windows (if you follow the 101 step procedure to the t and don’t stumble on any gotchas because your trying to accomplish something slightly different then what is documented), doesn’t mean that it should be run on Windows, especially for production applications. Take the LAMP stack for example Apache HTTP Server, PHP/PERL, MySQL. All these run on the Windows platform and thats great and all. It makes it very convenient to set up a development environment on windows machines. But in my very humble opinion, these projects were made to be ran in a *nix environment. Just look at the configuration files and you can easily see the developers have intended these to be run under a *nix platform. Installation is usually a breeze with these apps under *nix. Now I must admit that the Apache Group, MySQl, and ActiveState have done a very great job with there windows installation packages, but have you ever installed PHP on IIS? Tell me that wasn’t a headache the first time you tried that? Didn’t it seem kind of hackish? And the performance, how was it for you? Now, I haven’t tried the PHP “friendly” IIS 7 but I’m hoping this has improved some. Now, the lamp stack is just my example, but theres a multitude of open source software out there that just should not be run under windows. Installation and configuration usually involve some sort of two-bit hack and integration and/or troubleshooting is typically dreadful and time-consuming. And it goes both ways. I would never try to install IIS or MSSQL on a *nix platform but good luck to you if you dare! Let me know how that goes for you. And did someone say emulators? Yah, I “hear” wine has made some great strides in the past few years, especially thanks to the folks at google, but if your gonna run linux as your desktop, cant you make the full commitment and just run OpenOffice or Gimp? And please don’t even mention the word “cygwin” to me!

But “business requirements” are “business requirements” after all, aren’t they.

Heres a quick little script for automating the creation of Dom-U’s and there corresponding configuration files. Nothing to fancy, but it helps for users who may be unfamiliar with the process or just need to fire up a new Dom-U quickly without going thru an installation process.

This script assumes that when you first build a Dom-U either by d’bootstrapping or windows installation or whatever it may be, you shut the Dom-U down and move the “template” to /home/xen/templates and that your running Dom-U’s are running from /home/xen/domains . Lastly make sure the template configuration file has the <> strings so sed can replace them with your intended variables. This can be extended as you see fit, if you need to change values for networking, kernel, etc… Happy cloning.

./clone.sh

#!/bin/bash

function syntax {
           echo "Syntax: ./clone.sh source_image destination_image memory(mb)"
           echo " where source_image is the name of the directory in this folder                                                              "
           echo "  destination_image is the name of your new DomU "
           echo "  and memory(mb) is the ammount of memory you want to provision                                                              ."
           echo ""
           echo "Example: ./clone.sh debian_etch_x86 v-debian-etch-x 512"
           echo "Example: ./clone.sh winxp_pro_x86 v-winxp-pro-x 512"
}

if [ $1 ]; then
        if [ $2 ]; then
                if [ $3 ]; then
                        echo "Cloning Template $1 to DomU $2 with $3 MB of memor                                                              y."

                        echo "Copyig Disk Images... This will take a few minutes                                                              ..."
                        cp -R /home/xen/templates/$1 /home/xen/domains/$2

                        echo "Creating Configuration file."
                        sed 's/<domU>/'"$2"'/' /home/xen/templates/$1/$1.cfg > /                                                              home/xen/domains/$2/$2.cfg.tmp
                        sed 's/<domU-mem>/'"$3"'/' /home/xen/domains/$2/$2.cfg.t                                                              mp > /home/xen/domains/$2/$2.cfg
                        echo "Cleaning up temporary files..."
                        rm /home/xen/domains/$2/$2.cfg.tmp
                        rm /home/xen/domains/$2/$1.cfg
                        echo "All Done."
                else
                        syntax
                fi
        else
                syntax
        fi
else
        syntax
fi

./template.cfg

kernel = ‘/boot/vmlinuz-2.6.18-5-xen-vserver-amd64′
ramdisk = ‘/boot/initrd.img-2.6.18-5-xen-vserver-amd64′
memory = ‘<domU-mem>’
root = ‘/dev/sda1 ro’
disk = [ ‘file:/home/xen/domains/<domU>/disk.img,sda1,w’,
‘file:/home/xen/domains/<domU>/swap.img,sda2,w’ ]
name = ‘<domU>’
dhcp = ‘dhcp’
vif = [ ” ]
on_poweroff = ‘destroy’
on_reboot = ‘restart’
on_crash = ‘restart’