Designing these kinds of systems in the 50s and 60s must have been one of the all-time peak engineering experiences. Nearly everything on paper and drafting film, stacks of databooks, nomograms, slide rules, electromechanics everywhere, stratospheric budgets, hand wiring, manually machined parts and just generally making up and discovering things, from machining to physics, as you go along that we now consider fundamentals.
While deep diving into the history of how digital computers evolved from the initial ENIAC era of the late 1940s to the IBM 360 and DEC PDPs of the early 60s, I was surprised by just how much fundamental 1950s tech engineering was driven by the massive effort (and budgets) to create the technology needed to enable the national air defense network NORAD would later manage.
While a lot of the basic research existed on paper, turning that into functional 24/7 systems at mega-scale was an almost unimaginable engineering challenge. The fragments which already existed were little more than one-off prototype computers like ENIAC, none of which were remotely close to big enough, fast enough or reliable enough. So they were faced with creating pretty much the full stack from transistors to storage to networking to displays - all of which had to be several generations bigger, faster and better than anything that had ever been shown to even sort-of work on a lab bench. While the budgets were huge, the requirements were equally insane and there was huge pressure to deliver it ASAP. And "it" had to actually work and then be built, deployed and operated daily across dozens of locations. It ultimately involved dozens of huge companies, hundreds of sub-contractor firms and tens of thousands of people. Pretty much everything I'd ever heard about in 1960s computing, when I dug into where the fundamental tech came from, ended up tracing back to enabling something the air defense network needed in the 50s.
I came away realizing the mid-60s scalable commercial computing industry I think of as my ancestral 'up-line' in computing, the 360/PDP systems which led to the 1970s 8-bit microprocessors which led to the 80s home computers anyone could own, would have all been at least a decade later without the crazy mad dash in the 1950s to enable the air defense build-out.
It is more than the computer tech. Norad's tasking to the computer people was itself pulled from the air combat tech of the time: fast planes with very short-ranged weapons. With the threat of incoming supersonic bombers, NORAD needed to get interceptors on the tail of those bombers within, perhaps, a one-mile margin if they were to employ the missiles/gun available at the time. At supersonic speeds, and fuel reserved measured in seconds, delay or calculation error was unacceptable.
Any system of human controllers talking to human pilots would not be up to the task. Even a one-second delay would have meant a missed intercept. They needed machines to make the calculations and issue the orders. Those same machines eventually even controlled the interceptors directly, interacting with onboard radar/autopilot systems to remotely steer fighters into position.
Things would be/are very different today. Long-range air-to-air missiles/radar mean fighters do not need sub-second navigation accuracy to intercept fast targets. Air-to-air refueling also means fighters can loiter in position rather than panic-launch from the ground. Pilots today would balk at the idea of having a computerized ground controller drive "their" aircraft. It is now an unnecessary and alien concept.
Had missile and airborne radar tech advanced a little faster, maybe a little less than a decade, NORAD may not have asked for all those computers. Who knows what the world would look then like today. Get the sparrow (Aim-7) missile ready in 52 instead of 58 and maybe the computer revolution happens much more slowly.
He got his start because he had a security clearance and knew how to install and repair ccd camera systems do he did so for rocky flats and Cheyenne mountain. Then they asked him to build a box that had a film video camera that would take images of a crt display for record keeping. This was before it was possible digitally. So he did, he made 2. Then they needed a lift to bring down the Hughes projectors they used in Cheyenne mountain, the old kind with 3 color crt projectors. It needed to drop down the projector so they could change it out in less time than it took a nuke to come from Russia, so he did. He founded a company to keep making them, they still make them to this day.
So many companies were set up on the back of this cutting edge stuff, as well as Mercury and Apollo, satellites, nuclear reactors, submarines, jet liners, the SR-71, the Heavy Presses, it just goes on and on. Everything from one guy in a barn up to huge corporate labs. Obviously the war already boosted a lot of stuff, but the Atomic Age atmosphere must have felt like an unstoppable industrial whirlwind.
Even though the recent AI hype has been pretty feverish, it doesn't seem like we have had the same kind of top-to-bottom hopeful dynamism.
The redundancy systems in Cheyenne Mountain are fascinating from a modern distributed systems perspective. When designing our real-time services, we often struggle with 4-9s uptime, yet NORAD achieved remarkable reliability with 1960s technology. Their approach of physical isolation combined with multiple independent power/cooling/compute systems is still relevant today.
One detail that caught my attention: they used triple-redundant IBM 7090 mainframes with real-time voting between systems. This is surprisingly similar to modern consensus protocols like Raft, just implemented in hardware. Makes me wonder if we've over-complicated things with software-based consensus when simple majority voting could suffice for many use cases. The key seems to be truly independent failure domains - something NORAD achieved through physical separation that's harder to replicate in cloud environments.
Would love to hear from others who've implemented hardware-based consensus systems. How do they compare to software solutions in terms of reliability and complexity?
> The key seems to be truly independent failure domains - something NORAD achieved through physical separation that's harder to replicate in cloud environments.
It's not hard to achieve separation in cloud environments. It's just expensive, and for most, the need doesn't justify the expense. Every once in a while you do run into things like actually all the clouds in (location) are located in the same building; or you paid for two separate strands, but a backhoe has determined that both strands are in the same bundle. Or your two redundant providers of whatever use each other to provide service.
I'm always in favour of simplifying things, and in our case complexity breeds further complexity. We are making software recursively, building software on software on software on software, and this has more inherent complexity than just hardware. Why do we do this? It allows anyone to sit in their bedroom and build something in no time at all, compared to the large groups of people with specialised, expensive hardware it would require in the 1960s.
With more complexity comes more interesting failure scenarios. Simple redundancy handles problems where components outright fail, but software never really ... fails, in that sense. It always performs exactly to design – but sometimes the design is inadequate. Replication and majority voting does not handle design flaws and errors of e.g. interaction, which are increasingly common the more softwareified a product is.
Thus we need more complicated ways to build fault tolerance into systems where we have to assume that the design is wrong somewhere, and we cannot just replicate ourselves out of the problem.
Northern Europe had a very similar facility, a large "underground city" in The Netherlands near Maastricht, being used as a command center / information hub in NATO for the airspace. Unfortunately in the construction was a lot of asbestos, and it was completely dismantled. Small tours are given by former employees though! https://en.wikipedia.org/wiki/NORTHAG_War_Headquarters_Canne...
There's quite a few extensive underground structures scattered around the UK too that were built to defend against Soviet bombers, some of them started life as part of the original Chain Home system from the Second World War and were reactivated as part of the ROTOR programme in the 1950s. Some had a very short life as radar technology quickly improved, leading to many sites being abandoned while some became regional headquarters for dealing with civil emergencies. They're characterised by a specific design of bungalow which conceals a staircase down to a long, sloping corridor into the bunker system below.
A few are preserved today as Cold War museums, but most are sealed off and occasionally become high-value locations for British urbexers on the odd occasion access becomes available. Historically there was a subculture around exploring such places and documenting them for posterity, but YouTube kind of ruined it by making interesting abandoned sites magnets for vandals and attention-seekers which is the last thing you want in those places.
The ROTOR bunkers would take a braver man than me to explore though, they're often burned out, flooded, and full of asbestos with all the interesting 1950s tech long removed. Not to mention they're often located on farmland where the landowners are fed up of said YouTube crowd traipsing around.
> Near 100 per cent probability of continuing to function against multi-megaton weapons (i.e., underground location).
That of course wasn't possible, only a best effort. There was also the pseudo belief that ICBM missiles could be intercepted, and an entire system was designed and deployed (Project Nike), then decommissioned in 1974 due to it was a waste of money.
Not to be even more persnickety, but "Unauthorized incoming traveler" was also used in earlier seasons, including in the episode with Charlie and the Reetou.
It's sort of a random collection of images. The first image isn't the command center at all.
It's network operations, which is obvious if you look closely.
For a few years after the downfall of the USSR and before 9/11, there were public tours. That was a happy, peaceful time.
Here's a partial tour from the 1970s.[1]
There is no one big control room. There are about a half dozen control rooms
with different functions. There are duplicate control rooms outside the mountain, and for a few years, those were primary and the mountain only had a skeleton staff. Not any more.
(Although Hegseth apparently wants to move some operations to Huntsville, Alabama.)
Modern photos are available. Modest sized rooms with flat screens on the walls, desks, ordinary monitors, and keyboards. About the only unusual thing is that there's video switching, so that monitors can be copied to a big screen, or someone else's screen, when something is happening and many people need to focus on one screen.
There's now a vast flood of crap AI art and mislabeled clickbait for the Cheyenne Mountain Operations Center. Sorting out the real from the fake is becoming harder.
(One of my career achievements from my aerospace days was managing to avoid being transferred to Colorado Springs to work on their networking problems.)
> Well, all the cool kids were apparently doing "Deep space radar telemetry" :)
(Those guys were around. I worked at a location in Silicon Valley which had a 20 meter steerable dish in the parking lot, the prototype for others around the world.
The Colorado Springs guys were trying to migrate off the DEC PDP-11 to some kind of microprocessor. The PDP-11 (16 bit address space) was very popular with the USAF/intel crowd. Making add-ons and plugging them into the bus was a fully documented and supported procedure, not a hack. DEC sold all the parts needed for that. The usual setup was one or more PDP-11 machines connected via a custom interface to something exotic. In the early 1980s, it was time to go to a microprocessor. But what?
They picked the Zilog Z-8000 [1], because it was the closest thing to a PDP-11.
That turned out to be a dead end, but the aerospace company was already building custom interface hardware that talked to it, so they were stuck for the duration of separate projects. Did not end well. At one point I did get TCP/IP onto a pair of Z-8000 based Oynx machines, which were for a brief period the lowest cost UNIX boxes.)
“Deep-space radar telemetry is the cover story used by Captain Samantha Carter, and presumably the other members of Stargate Command, as a cover for their actual duties within Cheyenne Mountain.”
I got to go inside Cheyenne Mountain Complex back in 1988 when I was the project engineer for the DSP satellite ground network upgrade. Unfortunately, I didn't get to see any control room. In fact, I didn't get to see much at all. When we got there we were informed that the problem had already been resolved, so we just turned around and left.
But I did get to go down the tunnel and through the blast door.
I'm guessing some OCR was used to generate this by the many "typos" throughout.
Comparing these images of the COC to what was reimagined for War Games really feels underwhelming. From the few images, it just feels very complex and overloaded with information that is just a lot to take in. Maybe it gets easier to deal with when that's what you do everyday, but it definitely has that feel of "designed by an engineer" instead of "designed by a UI professional". Essentially, it feels like every single UI I've ever made.
My college roommate was stationed there from about 1997 to 2001 and I was lucky enough to get a tour as a civilian. They took us into the VIP room that overlooks the room shown in most of the pictures and then they ran through an exercise. In the modern era, the displays were much more focused. There were a set of large projection displays along the wall. In the center of the room, the personnel each had a workstation with more focused information for their specific task.
I got to tour it in the early 2000s. It was even less impressive to look at then than what these pictures show, and many of the rooms were rebuilt to be even smaller. One of the conference rooms where generals and perhaps the president were supposed to decide the fate of the world in a crisis was so cramped as to resemble that one scene from Attack of the Killer Tomatoes, but with nicer furniture. It was also amusing to me that with all secret operations shut down (to accommodate plebs like me), the skeleton crew left to run the place were almost entirely Canadian.
> One of the conference rooms where generals and perhaps the president
I don't think the President would have gone to Cheyenne wouldn't have been time since Colorado is quite far from Washinton D.C - iirc the plan was always kneecap (NEACP[1]) once it was online (and it still is).
Also, a large number of warheads would have been targeted at the mountain. No need to take the risk of having the commander-in-chief in that location. He'd be better off aboard NEACP to be sure. The risk of being shot down by enemy fighters would not be very high. It's hard to be shot down by an incoming MIRV. Even though Colorado is far from DC, I'd imagine they'd head west away from population centers to ride it out.
I suspect they'd go north (from DC or nearby) via a dog leg to the east, once you clear where the Canadians live (which is mostly bunched up on the border) there isn't much up there the Russians or Chinese would want to vaporise and they can dip south to refuel (assuming anything is left to take off to refuel).
Of all of the Clancy-esque fictional warfare I've consumed, I've never read anything about what these scenarios would be like. I'd imagine an armada of planes (or whatever you call the equivalent for planes) would be available as support to NEACP. Not just fighters for defense but fuel tanker support, possibly AWACs in case ground stations are taken out as well. But where they'd actually fly is an interesting question. They could go out over the Atlantic and turn south and be away from targets faster than flying over land. Of course, that's assuming they know where any hostile fleets would be to avoid. Lot's of options though, which is the point of being airborne instead of stuck in a bunker
Those types of books are where I learnt about kneecap back in the late 80s/early 90s as a kid, which is why I didn’t know it had been renamed post Cold War.
They obviously don’t discuss it for obvious reasons but given the options I’d say east then either north or south, Canada is nearer and air fields with sufficient length to be able to land (and is in NATO) which is why it’d be my guess at least to start with, loiter over an unpopulated area and at least the likelihood is reduced of been near a detonation is reduced, the continental US simply has too many targets, though with modern warheads been lower yield they’d likely be OK orbiting unpopulated areas out west I guess.
Seemed weird to me that they turned it into a secondary site; we were fighting relatively incapable-of-force-projection people in the mountains and deserts at the time, but even if Russia wasn't a clear threat in 2008, it seems like it should have been obvious EMP, conventional infiltration attacks, etc. would be reasonable threats in the future. Unless you're willing to go to fully dispersed command (and thus risk a commander at theater or below level launching on his own authority...), or run 24x7 airborne looking glass (which ended in 1990, and presumably was even more costly than modernizing Cheyenne Mountain Complex), what we had from 2008-2015 was clearly less survivable.
According to Daniel Ellsberg the US did implement a scheme where regional commanders could use nuclear weapons without explicit authority from the President:
"Walking out of the theater, Ellsberg turned to his friend, another nuclear denizen, and said, “That was a documentary.”"
In descriptions of the process it always amuses me that they talk about failure to receive Radio 4 being the test (perhaps of it also being broadcast on long wave)...
House of Dynamite on Netflix is a realistic (at least it feels realistic) look at the modern day equivalent. It’s fundamentally an exciting film but I also enjoyed learning how large scale human/technical systems operate during a nuclear crisis.
I have to wonder what part of me has become jaded to the point that the movie did not feel that intense to me. I won't go into details on why I think this movie isn't intense, but it's too new to spoil anything for those yet to see it.
It's not the same as something like Threads (1984) to me.
not at all realistic, awful movie. somehow even worse than annie jacobsen's book. they would not fire only 2 interceptors, and there is no urgency to retaliate when it is not a decapitation strike
This is easily the best, inadvertent advertisement ever for boosting the nation's missile defense capabilities, and it's ironic that it came out of progressive Netflix studios, given that progressives have insisted since the 80s that missile defense is a worthless, impossible ("like hitting a bullet with a bullet") MIC boondoggle. Thanks, Netflix!
Corporations don't have political leanings, they have market fit. The rest are your prejudices when encountering a product targeted at another consumer category than yours.
On the photos everyone looks busy, but I presume that the guys spent 30 years sitting in front of screens where nothing happen, except perhaps once every 3 months.
They had simulations, although I don’t know how often they were run. There was at least one incident where a simulation was accidentally fed into the system and people came uncomfortably close to retaliating.
If you ever get to Colorado Springs, in the shadow of Cheyenne Mountain is a state park, Cheyenne Mountain State Park. While it’s due to the dry conditions, all the trees and shrubs there look twisted and contorted, like a scene from Roadside Picnic.
reply