Nobody but Us: A History of Cray Research's Software and the Building of the World's Fastest Supercomputer

Requirement for Mastery

We build the world's fastest computers. Period. This was our unofficial Cray Research motto. We were helping win the Cold War in a tangible way. We thus had many masters of our craft working together for that common purpose.

The path from beginner through intermediate or journeyman to master is not linear. Mastery is not having more skills. Mastery means skills become internalized and automatic. The context and technology begin to no longer matter, because your skill now applies to any context or technology.

One example is driving a car or riding a bicycle. Over time, you can handle nearly any bicycle or car. The details for a specific automobile do not matter, because you can handle those details. You know how to drive, or you know how to ride a bicycle. The invariant is "drive" or "ride". Mastery comes when you are inhabiting the invariant rather than using some skill specific to the task at hand.

Another example is debugging and problem analysis. For me, the tooling does not matter, and often gets in the way of seeing through to the answer. The techniques I used 40 years ago, I still use today, because those techniques are the invariants. I can apply those techniques to any modern technology. You might call these timeless skills, generalized skills, or abstract skills.

"It has never been done before" was never a barrier. It was the necessary starting point. It forced us to learn to inhabit the invariants because the specific target technology did not yet exist. These skills became so internalized and automatic that we never wrote them down (until now). We mentored and collaborated with each other. We were each aligned with our common purpose.

Modern frontier-computing AI is just beginning to inhabit the problem space Cray Research held for decades: where you cannot spin up more hardware because you are already sitting on the biggest computing system available. It is somewhat like reaching personal mastery: the rules change. You are on a different plane entirely. With AI reaching this problem space, the next few years will be interesting.

The Problem to Solve

Part I focuses on the reason our direct predecessors invented large-scale computers. High-Performance Computing (HPC) systems were not products brought to market. Systems were cobbled together to solve a specific problem: code breaking. Part I teaches you to see and recognize large cross-cutting patterns in that problem space. Pattern recognition is the foundation for nearly all applied computer science. Your first challenge along the path to mastery is learning to notice patterns.

Prerequisite skills must always be learned to mastery. There is no shortcut. While partial skills are absolutely valuable, when they are prerequisite to something bigger, they must be learned to mastery. As with driving a car, you will know you mastered the skill when the specific context (which car you are driving) no longer matters. A book can only guide you; you must do the work, internalizing the skill.

Applying the Skill

In Part II we will develop algorithms, write software, and learn a style of problem analysis we practiced within Cray Research. We will learn to apply skills and techniques across different contexts and eras. The new invariant is "context does not matter" in the same sense that which car you need to drive does not matter.

Part II can be challenging due to subjects like binary arithmetic no longer being taught to children in school. This is not a failing of students nor of computer science; it is that priorities shifted. The skills you need for Part III are here in Part II.

Top Secret Origins

The U.S. and U.K. created High-Performance Computing systems in the 1940s to solve a specific class of problems (code breaking). Each of these was a special-purpose system, unlike general-purpose programmable computers as we know them today. Part III traces the decisions made along the path to supercomputing and to Cray Research, beginning with a significant Soviet computing system as a baseline for comparison. Examining Soviet development independent of Western development allows us to spot the invariants, the points in common. Since invariants are timeless, we can usefully compare those first-generation computers to modern AI systems, naming the similarities to today.

Once you learn the language of invariants, so to speak, you will likely be surprised to discover that learning about first-generation high-performance computing provides insight into exascale and AI systems. The reason is that similar types of constraints tend to shape similar solutions. Across this 85-year problem space, the similar solutions appear time and again.

Attention Is All You Need

The 2017 paper "Attention Is All You Need", introducing AI Transformers, looks quite familiar to people from Cray Research of the 1980s. My bare metal source code from 1986, isomorphic to parts of Attention Is All You Need, is public on GitHub with a 2019 third-party timestamp.

Cray Research people look past the details to see the invariants guiding the solution: passing the fastest non-classified I/O on the planet through 128 KBytes of local memory that includes operating system, heap, stack, and channel buffers has constraints quite similar to a hot LLM token context under load.

We could keep a 4-CPU Cray X-MP, the fastest system on the planet, with all four CPUs pegged continuously at 100%, fed entirely from a wall of screaming magnetic tapes. Potential customer Texaco went home and returned with their own tapes because they thought we were faking the demo. We screamed their tapes too.

Those magnetic tapes were created in the backs of dusty oilfield pickup trucks. We screamed through the fastest error rates in the business. About 70% of tape code is for error recovery. That later became my piece of the action, pushing simultaneous error-prone megabyte tape blocks through 16 KByte channel buffers.

Part III's final chapter brings you inside a CRAY-1, on the bare metal, keeping track of several hardware resources in your head simultaneously.

Same Patterns, Any Context

Margaret Loftus, Vice President of Software, taught us that if it isn't any fun, it likely isn't worth doing. Part IV demonstrates fun as a first-class engineering principle, while dealing with real issues such as a channel command protocol problem inside the NSA or hardware-supported double buffering. A quiet protest became the Ducky Day annual celebration.

What was it like within the elite environment that was Cray Research? Now you will know, having inhabited our mental models and cognitive environment, and having that bit of fun along the way.

The Wizard's Mirror

Throughout this description I first showed you the invariant to learn, then the details setting you on that path to mastery. The entire book performs what it teaches as it teaches it. Part V brings these skills together into an integrated whole. The first example is my "demotion" from bare metal assembly language with no guard rails to programming in C. But we found C has the same capability of allowing things to go catastrophically wrong. It was still on us to get things right, with no tools to show something is wrong. That danger kept the boredom at bay.

Part V condemns gate keeping, and ends where we started: with pattern recognition, but now as the foundation of an integrated whole.

National Defense Education Act (NDEA)

NDEA funded large school districts sweeping up school children to become future scientists as part of the Cold War space race and arms race. I was identified at age 9, 13 years before joining Cray Research. Part VI is my origin story tracing back to national panic after the Soviet Sputnik satellite launch, and further back to the Nazi scientists brought into both U.S. and Soviet space programs. Many of us found our way into the computing industry thanks to NDEA opportunities.

Part VI wraps up with a complete walkthrough of this book's cognitive design, which is an apprenticeship in written form.