Remember back in September 2009, when I posted this blog entry on the importance of automation?  No?  (I don’t really blame you.  I only remembered that I’d posted something about automation; I had to re-read it to recall exactly what I’d written.)  My point was that automating as much as possible of the process of administering DNS was about to get even more critical with broader adoption of DNSSEC.

With DNSSEC-signed zones, there are all sorts of tasks to remember:  Re-sign your zone after you make any change to zone data.  Re-sign your zone to refresh signatures before they expire.  Roll your Zone-Signing Key over monthly.  Roll your Key-Signing Key over annually.  Oh, but be sure to wait the requisite amount of time during the rollover for old keys to time out of remote caches or for new keys to propagate.  Then multiply all of these tasks by the number of signed zones you have, and add the complexity of maintaining different re-signing and rollover timers per zone.  That way madness lies.

Read more...

Waaaay back when I ran hp.com, I had what I only now realize was an enviable position:  I was HP’s hostmaster (the somewhat-ceremonial title given to the person responsible for a zone) but not much else.  I dabbled in NTP and ran a big mail relay, but the bulk of my responsibility was DNS.  From when I got to work in the morning to when I left in the evening, I could concentrate on DNS.

At the time, I didn’t realize what a luxury that was.  I figured every big company probably had a person dedicated to DNS.  And in those days, some did. Partly, this was because we hostmasters could get away with it.  DNS was such a black art that you could simply assert that it took up most of your time and your management wouldn’t know any better.

How the times have changed.  I’ve had the opportunity to meet the folks responsible for DNS at many big companies, but I hesitate to call them “hostmasters”—not because they don’t deserve the customary title, but because it sells them short.  These people run routers, switches, firewalls, mail servers, and more.  Almost no one has the luxury of specializing in DNS any more.  The economic climate dictates that we all take on more responsibilities to make our employers more competitive.

Read more...

An hour or so ago, I tried to check a Wikipedia entry and my browser told me it couldn't find en.wikipedia.org.  Surely that's wrong, I thought, but pushed "Check Wikipedia" onto the stack and went on to something else.  Then, coincidentally, while searching for DNS-related news articles to inspire my next blog entry, I ran across this one from PC Magazine.  Turns out Wikipedia's European data center had an overheating problem that caused many of their servers to shut down in an act of self-preservation.  To shunt European traffic to their servers in Florida, they enacted their failure procedure, which modifies their DNS records.

Unfortunately, that failover mechanism was broken (they didn't specify how), and broken so badly that it interrupted DNS resolution for all Wikimedia sites globally.  While they quickly recognized and fixed the problem, it took as long as an hour for the corrected data to propagate because of TTLs.

Read more...

...especially at the expense of the excellent folks who run .SE, but wasn't I just writing last month about everything that can go wrong in a manually administered DNS environment?  In fact, didn't I specifically say:

"Use a trailing dot to prevent the origin from being appended to a domain name.  After editing a zone data file, increment the serial number and reload. Forget any one of those and you've caused an operational issue, maybe even an outage."

Well, it looks like .SE had one of those very problems.

Read more...

Over the past few years--I can't remember exactly when, which is part of the problem--I've become alarmingly forgetful.  I'll get up, walk across the building to do something, and forget completely what it was that I intended to do.  Talk to Julie about upcoming roundtables?  Ask Eric or Arlen a question about UI design?

That's a nuisance for me around the office, but it would be downright dangerous if anyone still let me manage a production zone or name server.

Even in the simplest DNS environments, there's a lot to remember:  An SOA record has seven RDATA fields.  Use a trailing dot to prevent the origin from being appended to a domain name.  After editing a zone data file, increment the serial number and reload. Forget any one of those and you've caused an operational issue, maybe even an outage.

Read more...