michaelanthoni.com

So its been a few weeks since most of us have patched our vulnerable dns servers, but I hadn’t noticed this little bonus until today which actually made me laugh. You see a few years ago I had noticed an annoying little behavior with the way Microsoft’s DNS Server handles outgoing client connection for domains/servers that are listed under the Forwarders tab. We use this Forwarders tab to list frequently queried domains in which we host a copy of the zone file in rbldnsd so as to not go to the internet to find the answer to. This gives us the benefit of returning an answer to a dns query much faster and saves us the extra bandwidth. This is highly beneficial to our mail systems which process on average 100 million messages per month, mostly spam of course. So back when we had implemented the rbldnsd system, we had placed Linux Virtual Server in front of the rbldnsd to load balance the traffic accross 8 or so machines. After pointing the Forwarded domains to the LVS VIP, I had expected hundreds, even thousands of connections to get sprayed accross the rbldnsd farm, but uh-uh, nope. There were only two connections listed to two of the backend servers, however all the queries were getting answered.

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 1
-> server2.domain.com:domain Route 2 0 0
-> server3.domain.com:domain Route 2 0 0
-> server4.domain.com:domain Route 2 0 1
-> server5.domain.com:domain Route 2 0 0
-> server6.domain.com:domain Route 2 0 0
-> server7.domain.com:domain Route 2 0 0
-> server8.domain.com:domain Route 2 0 0

This had me scratching my head at first and then after a few packet captures later, I realized that Microsoft was opening 1 socket connection and pushing all the forwarded queries through it. Gee Wiz Microsoft! Why would you do such a thing? I figured that opening and closing socket connections carries an overhead and could also potentially exhaust all available udp ports in a very short amount of time, I can understand why Microsoft would implement it in this way. However this is exactly the insufficient socket entropy that is described in Dan’s advisory as flawed, and from my perspective I hated it as I couldn’t load balance all the forwarded dns queries across each machine that had rbldnsd running on them. Luckily rbldnsd wasn’t the primary service on those machines that we were load balancing so I had decided to just let it be after a spending a few minutes of looking for a workaround then banging my head on my desk out of frustration. Availability was still guaranteed and rbldnsd being as fast and memory efficient as it was, was performing fine in this configuration, so I let it be. I had bigger fish to fry at the time. Fast forward a few years later and a Dan Kaminsky patched Microsoft DNS Server, and wallah, this is what I noticed today…

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 264
-> server2.domain.com:domain Route 2 0 258
-> server3.domain.com:domain Route 2 0 256
-> server4.domain.com:domain Route 2 0 252
-> server5.domain.com:domain Route 2 0 250
-> server6.domain.com:domain Route 2 0 252
-> server7.domain.com:domain Route 2 0 252
-> server8.domain.com:domain Route 2 0 252

and this is with a modified udp timeout of 10 seconds…

me@director:~$ sudo ipvsadm -L --timeout
Timeout (tcp tcpfin udp): 60 10 10

Awesome, entropy, security, and load balancing :). Thanks Dan!

Well I’ve been using cfengine for a number of years now and thought I had paid my dues already when I initially took on its steep learning curve… Well today I had a little run in with cfengine that made me feel as frustrated as when I was a newbie to this software, but I guess it was a newbie mistake that Im sure I learned years ago that I just happened to forget over the years when adding a cluster of new hosts to the mix - reverse dns.
The issue came about when I was configuring a new group of servers. I was on the final one when I simply installed cfengine on the host, scp’ed over cfagent.conf, cfservd.conf, and update.conf from a host that I had just been successful with. But after running “cfagent -v” i ran into the familiar “Can’t stat /var/lib/cfengine… in copy” which struck me as odd because it had just worked on all the other hosts. After checking the usual suspects such as the grant: function in the cfservd.conf to make sure permissions were explicitly granted on the server side, the hostname and domain name configured on the client, typos, cfkeys, cfservd started ?, etc, nothing seemed to work and adding the debug options -d seemed to frustrate me even more. As a last resort I took a packet capture to see what was going on between the client-> server for both the system that was failing and one that was working. I didnt think it would help much but sure enough after crawling through the capture packet by packet I seen the issue in one of the packets data field that looked something like this…

CAUTH IP IP user - non-working host
CAUTH IP hostname user - working host

This is when the little cfl lightbulb went off in my head and I decided to have a look at reverse dns. Sure enough all the hosts had reverse dns configured but this last one.

Although other functions such as directories,files,editfiles seemed to authenticate and run fine without reverse dns it seemed the copy function was failing because authentication under cfservd and the grant directive is based on the domain *.domain.com and not the IP… sheesh… it seems the parameter SkipVerify can be applied globally here and workaround hosts not having reverse dns, however I decided not to use this option since we control the reverse dns and it really should of been configured, not sure why it was not…

as soon as I added the reverse dns for the host cfagent ran without a hitch…

OpenBSD 4.3 was publicly released on schedule last week with astonishing amount of improvements, new features, and bug fixes. Hats off to the OpenBSD Developers that are putting out some great work to make a stable and reliable product.

However, I must say, I did notice some bug fixes in the changelogs that I’ve actually came across and been caught up on that were fixed in this release. These issues had been referenced in the openbsd mailing lists by other users but were never acknowledged by the developers leaving me frustrated and at a dead end without getting into the code. Its good to see these issues finally acknowledged in the changelog.

With that said… (see previous post)

I use Windows XP Pro as my desktop operating system of choice and haven’t seen the reason to make the move to Vista yet, although I will be investigating more in the near future with Windows Server 2008 looming around the corner. I find myself to be more efficient and comfortable, in a windows desktop environment than I do in either GNOME or KDE, although I feel right at home in a GNOME enviornment. Although Mac was probably the first OS I ever used, I grew up using W indows and it feels like home to me. However, I do tend to run unix-like virtual machines on top of my desktop to allow me some flexibility. Also, I run a dual booting, Windows XP/linux configuration on my laptops. Never know when your gonna need what. I like to keep my options open.

I have run a MacOSX laptop in the past but at the present I have not had the luxury.

On the server side of things, I run both Windows and Unix-Like, depending on the situation, client, customer, requirements, etc. On the Windows side I run Windows Server 2003, of course, in its different flavors. On the Unix-Like side my Linux OS of choice for production is Debian, however I am a linux hobbyist and have ran most major Linux flavors including Ubuntu, Slackware, Gentoo, Suse, and Redhat based operating systems at one point or another. My BSD of choice for servers is FreeBSD, but I use OpenBSD for the network infrastructure such as routers and firewalls.

Again, I have closed commenting on this page as to not insite a flame war. This is what I run.