Today’s escapade in server administration reminded me of the absolutely most difficult and frustrating job in IT. That would be “Debugging time dependent or time sensitive applications.”
Here is a couple of real world cases to ponder upon:
1) Once I had a predictive dialer based call center stop functioning because of time differences. The asterisk server and the database server had a one minute (yes that is a single 1) time difference between them. After a full hectic hour of reading PERL code in a frenzy I found out that there was a magic stanza that read: if you have more than 30″ time difference in status updates, assume that the agents are off line. Cute uh ?
2) A newly minted SQUID server stopped serving cache hits because that new fangled version does time/date comparisons differently. So it validated the page’s expiration time the web server sent back and decided that it should NOT cache the query because it was expired. Half a workday lost! Cute uh?
So all you admins, don’t be cute, do NOT play loose with a system’s clock , more things than you can imagine might depend on date/time synchronization.
Manage Not, Engage! 