Whenever I have a client call me up to tell me that a service that’s been working fine for ages has suddenly stopped, two prime culprits immediately spring to mind. One is very subtle, the other not so much.
When first logging in to the afflicted system, I do `df -h` to check available disk space on all volumes. It’s amazing how many of these incidents come down to a disk running out of space. Usually, a monitoring system warns when disk space hits 90%. Sometimes, though, that last 10% can be churned through very quickly, catching even the monitoring service off guard. In a number of cases, I’ve had monitoring systems miss full disk conditions because the full disk condition stopped the agent from accurately reporting the current state (a topic for another entry). Bottom line: check your disk space.
DNS issues are much more subtle and can manifest themselves in different ways. Often, you’ll see a sudden decrease in performance for some inexplicable reason. Things that used to work suddenly break. In many cases, this is because systems that refer to hosts by name can either no longer find those hosts or there’s a sudden delay in the process.
The `host` command is your friend here:
chris@utc-pdc1:~$ host www.dollmont.net
www.dollmont.net is an alias for dollmont.net.
dollmont.net has address 216.86.156.11
This tells me that www.dollmont.net points to dollmont.net, which in turn has an ip address of 216.86.156.11.
chris@utc-pdc1:~$ host 216.86.156.11
11.156.86.216.in-addr.arpa domain name pointer server1.fusednetwork.com.
Doing a `host` command on the IP address will tell me if reverse DNS is working. In this case, it’s pointing me at a Fused Network server, which is correct—Fused Networks does my site hosting. In a perfect world, I would have a DNS PTR record for the IP address that pointed back to dollmont.net, but this is more than good enough. Forward and reverse DNS both resolve.
If there’s an error in either one of these checks, it means that DNS is broken somewhere down the line. Trouble shooting that can be a nightmare, and is beyond the scope of this article.
Disk space and DNS are the first two things you should look at whenever things start behaving in a wildly different manner. Often the problem will be one or the other and usually involves a quick fix.
