probably the same thing happened here. Vibration will cause latency in reading disk. If the hd platter are jumping around the needle will have a hard time reading it. That probably explains the spike. Time to buy some SSDs ;p
Why did page load time go up during the quake? Was there a fiber fault that took a few seconds to be routed around? Did the vibrations cause HDDs to temporarily suspend?
This could have some interesting data behind it, but as it is the article doesn't even have conclusive proof that the earthquake did cause this outage.
As far as the reason, you'd have to ask who works on AWS. For obvious reasons, we don't have direct access to the Amazon datacenter.
As far as conclusive proof is concerned, yes, we can't guarantee there wasn't a gravitational singularity that affected response time, but it's very likely that this was the case.
Yeah, latency is most likely due to the vibrations affecting the various rack components, as commented here. Actually, earthquake-proofing datacenters is a big business in places like the West-Coast USA and Japan: http://www.datacenterknowledge.com/archives/2007/07/17/earth...
"Over here at SeatGeek, we were excitedly discussing the tremor when Mike, our trusty sysadmin, realized that our Amazon AWS servers were all in Virginia, right near the epicenter. Did it impact the service at all?"
Presumably the earthquake caused a spike in social network usage, microblogging of various popular types, and reload-mashing on cnn.com and similar. If any of those is hosted on AWS then they might steal some cycles from other AWS users.
Purely anecdotal, but here in downtown DC, cell networks were fine during and immediately after the quake, but were completely overwhelmed 5-10 minutes later by everyone pulling out their phone at once.
Another thing that comes to mind is that some computers have accelerometers in them that stop the harddisks if the machine experiences sudden acceleration. If AWS has the same system in place, that might affect their servers' responsiveness.
Here at RadioReference.com had a MySQL Master server which is hosted on AWS East in the N. Virginia data center inexplicably crash on us right after the earthquake. The server uses a RAID-0 Stripe across 4 EBS instances and has been running for over a year without a reboot.
And, we were featured on CNN live right after the quake as a source for breaking news information.
We're scaled to handle a traffic floods because we get them occasionally when something big happens public safety wise, but I'm really wondering whether or not this crash was due to a huge influx of people or some hardware anomaly during the quake (frozen disk, network problem etc)
A reboot of the server and an INNODB recovery fixed the issue, and all is fine now.
There are probably less than 50,000 people living inside the yellow circle and there are no cities.
Amazon's data centers are in northern Virginia. This earthquake did not happen in northern Virginia, it happened in central Virginia, between Richmond and Charlottesville, about 60-90 miles away from northern Virginia.
I know, but it doesn't sound right if you live in Virginia. Just like saying "Los Angeles is near San Francisco" probably only sounds right to east coasters. And if you look at the shake map, it makes a big difference whether you are within 20 miles of the epicenter or 100 miles.
I saw that much, but I'd love to see a write-up on their implementation. We primarily use Munin with a couple of custom plugins. It's fine for the sysadmin side, but we were thinking of pushing some app data stats to a customer facing interface. Tools like GeckoBoard look much better than Munin graphs.
http://www.youtube.com/watch?v=tDacjrSCeq4