But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm."
It's basically Test-Driven Development: if you cannot test it, don't write it.
It is hard to test emergent behavior in large distributed systems, you pretty much have to actually run the tests live to see what is going to happen and see if it aligns with your predictions.
It's basically Test-Driven Development: if you cannot test it, don't write it.