michaelanthoni.com

So its been a few weeks since most of us have patched our vulnerable dns servers, but I hadn’t noticed this little bonus until today which actually made me laugh. You see a few years ago I had noticed an annoying little behavior with the way Microsoft’s DNS Server handles outgoing client connection for domains/servers that are listed under the Forwarders tab. We use this Forwarders tab to list frequently queried domains in which we host a copy of the zone file in rbldnsd so as to not go to the internet to find the answer to. This gives us the benefit of returning an answer to a dns query much faster and saves us the extra bandwidth. This is highly beneficial to our mail systems which process on average 100 million messages per month, mostly spam of course. So back when we had implemented the rbldnsd system, we had placed Linux Virtual Server in front of the rbldnsd to load balance the traffic accross 8 or so machines. After pointing the Forwarded domains to the LVS VIP, I had expected hundreds, even thousands of connections to get sprayed accross the rbldnsd farm, but uh-uh, nope. There were only two connections listed to two of the backend servers, however all the queries were getting answered.

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 1
-> server2.domain.com:domain Route 2 0 0
-> server3.domain.com:domain Route 2 0 0
-> server4.domain.com:domain Route 2 0 1
-> server5.domain.com:domain Route 2 0 0
-> server6.domain.com:domain Route 2 0 0
-> server7.domain.com:domain Route 2 0 0
-> server8.domain.com:domain Route 2 0 0

This had me scratching my head at first and then after a few packet captures later, I realized that Microsoft was opening 1 socket connection and pushing all the forwarded queries through it. Gee Wiz Microsoft! Why would you do such a thing? I figured that opening and closing socket connections carries an overhead and could also potentially exhaust all available udp ports in a very short amount of time, I can understand why Microsoft would implement it in this way. However this is exactly the insufficient socket entropy that is described in Dan’s advisory as flawed, and from my perspective I hated it as I couldn’t load balance all the forwarded dns queries across each machine that had rbldnsd running on them. Luckily rbldnsd wasn’t the primary service on those machines that we were load balancing so I had decided to just let it be after a spending a few minutes of looking for a workaround then banging my head on my desk out of frustration. Availability was still guaranteed and rbldnsd being as fast and memory efficient as it was, was performing fine in this configuration, so I let it be. I had bigger fish to fry at the time. Fast forward a few years later and a Dan Kaminsky patched Microsoft DNS Server, and wallah, this is what I noticed today…

me@director:~$sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP w.x.y.z:domain wrr
-> server1.domain.com:domain Route 2 0 264
-> server2.domain.com:domain Route 2 0 258
-> server3.domain.com:domain Route 2 0 256
-> server4.domain.com:domain Route 2 0 252
-> server5.domain.com:domain Route 2 0 250
-> server6.domain.com:domain Route 2 0 252
-> server7.domain.com:domain Route 2 0 252
-> server8.domain.com:domain Route 2 0 252

and this is with a modified udp timeout of 10 seconds…

me@director:~$ sudo ipvsadm -L --timeout
Timeout (tcp tcpfin udp): 60 10 10

Awesome, entropy, security, and load balancing :). Thanks Dan!

If you need access to any of the ctype functions such as ctype_alnum() in any of your php applications and you are running gentoo, make sure you add the ctype use flag or you will get this lovely error :).

Well I’ve been using cfengine for a number of years now and thought I had paid my dues already when I initially took on its steep learning curve… Well today I had a little run in with cfengine that made me feel as frustrated as when I was a newbie to this software, but I guess it was a newbie mistake that Im sure I learned years ago that I just happened to forget over the years when adding a cluster of new hosts to the mix - reverse dns.
The issue came about when I was configuring a new group of servers. I was on the final one when I simply installed cfengine on the host, scp’ed over cfagent.conf, cfservd.conf, and update.conf from a host that I had just been successful with. But after running “cfagent -v” i ran into the familiar “Can’t stat /var/lib/cfengine… in copy” which struck me as odd because it had just worked on all the other hosts. After checking the usual suspects such as the grant: function in the cfservd.conf to make sure permissions were explicitly granted on the server side, the hostname and domain name configured on the client, typos, cfkeys, cfservd started ?, etc, nothing seemed to work and adding the debug options -d seemed to frustrate me even more. As a last resort I took a packet capture to see what was going on between the client-> server for both the system that was failing and one that was working. I didnt think it would help much but sure enough after crawling through the capture packet by packet I seen the issue in one of the packets data field that looked something like this…

CAUTH IP IP user - non-working host
CAUTH IP hostname user - working host

This is when the little cfl lightbulb went off in my head and I decided to have a look at reverse dns. Sure enough all the hosts had reverse dns configured but this last one.

Although other functions such as directories,files,editfiles seemed to authenticate and run fine without reverse dns it seemed the copy function was failing because authentication under cfservd and the grant directive is based on the domain *.domain.com and not the IP… sheesh… it seems the parameter SkipVerify can be applied globally here and workaround hosts not having reverse dns, however I decided not to use this option since we control the reverse dns and it really should of been configured, not sure why it was not…

as soon as I added the reverse dns for the host cfagent ran without a hitch…