First day of Seventh Month

Yesterday was the first day of the Lunar Seventh Month, popularly known as the "Ghost Month" for the Chinese.

And yesterday, it was very 'exciting' in office.

Early in the morning, the Exchange mail server was flooded by thousands of emails originating from a test. We stopped the email from sending out from the culprit server, but were hopeless to stop those that were already stuck in the mail server. The mail server was down almost the entire work day and was only working properly around 5.30pm, after a reboot of the server which cleared away the dead emails.

Then around 4:30pm, the Internet server was behaving strangely and users were not able to access any of the websites at all. Seems like the Unix server encountered a major core dump, which hogged the disk space and also killed the java processes that were running in it. So before we could delete any logs, we need to get the log files out for forensic purposes. With the files amounting to 3G, it took us almost 40 min to ftp the files out. As no advisory can be sent to the users to notify this outage immediately (becos of the mail server issue), we had to send 2 consecutive advisory, first to advise of the outage and the second one for it's resolution.

Two major happenings in a day.. that's too much excitement for me.. especially when the team is left with 3 engineers (including me) to hold the fort. So I think I need to go and pray, so I can have a smooth double 7th month..