Chapter 3. Apprenticeship Transmission

Within Cray Research Software Training, teaching was easy. The subject matter was complex, but this was long before the "messy details" were being hidden behind abstractions. We reasoned about systems as a whole because we could see whole systems and their constraints.

Code Generation as Example

As an example, the Cray Fortran compiler had a compiler option to produce a copy of its generated code. Analysts could ask to see the code before or after compiler optimizing. Analysts could ask for the generated code to be correctly formatted for the Cray Assembly Language assembler. Do you see why the latter option was crucially important?

Back then, any of us could hand-optimize code better than the language compiler could. I personally taught how to do so. We could take the generated code, edit it for improvements, and feed it through the assembler. At Cray Research, performance was everything. This was one way of improving performance. Of course we then measured results under real workload to confirm the edit was an improvement.

Visibility Lost

Now that we are freed from bare metal details, we are also freed from visibility of hard constraints. The computing system remains bounded by hard constraints, but now it is harder to determine what those constraints might be.

This is why the "HPC Tradecraft" series exists: to teach you what we stopped teaching since about 1995. Few have needed this information, but now we do. Many legacy systems require systems thinking to keep them running, and HPC tradecraft enables new possibilities with AI.

Within Cray Research and its predecessors, we had mentors, collaborations, and informal apprenticeships. We passed tacit knowledge from person to person. Now we are faced with the challenge of material in written form without the mentor (author) present.

Transmission Method

Teaching our HPC tradecraft is pointless unless the teaching can be replicated. If any teacher or professor chose to teach this material, it would be crucial to know the teaching style employed, because it is unusual.

I found the typical IT book format inadequate to this task. I therefore intentionally created an apprenticeship situation, since that is how I learned our tradecraft in the first place. These books require paying close attention. Unknowns will be frustrating. At Cray Research, we treated barriers as opportunities. "Figure it out" was a fun challenge.

I later discovered a Soviet tradition formalizes the learning approach these books use. That Soviet tradition of learning-psychology is the next section.

Book Design

I designed the next two books to transmit HPC tradecraft:

  • Nobody but Us: A History of Cray Research's Software and the Building of the World's Fastest Supercomputer teaches how we designed HPC systems from the software perspective.
  • The Wizard's Lens: Learn to Think Like AI demonstrates how we modeled our HPC systems by observing system behavior under load, using AI as the HPC system under observation.

Both books have a recursive design in the sense that they demonstrate what they are teaching as they teach it. In addition, the book design itself embodies the tradecraft being taught. Both books are essentially executable specifications. That fact is one of my technique's unusual characteristics, and thus important to explain up front.

Conditions for Formation of Thinking

Both books happen to exhibit Dr. Piotr Ya. Galperin's methods. I learned of this after the fact from Dr. Andrei I. Podolskiy's papers expanding and explaining Galperin's methods. Galperin's lectures were at Moscow State University, where Podolskiy continued Galperin's line of work. My independent implementation (reflecting HPC tradecraft) suggests that Galperin knew what he was talking about. Neither of us was aware of the other's tradecraft. Galperin came from a theoretical perspective. I came from an experiential practice perspective. We converged on similar solutions, which tends to confirm both traditions.

Galperin's early training was at Kharkov, in the Ukrainian Soviet Socialist Republic rather than in Moscow. He worked with Aleksei N. Leontiev, who in turn had worked with Lev S. Vygotsky in the 1920s.

  1. Vygotsky described the Zone of Proximal Development (ZPD). Moving away from stimulus-response, his idea was that what a learner can do with assistance today, the learner can do independently tomorrow. The teacher is a guide, creating social/cultural conditions for mental development to occur.
  2. Leontiev shifted the focus to practical activity: actions and operations corresponding to motives, goals, and conditions. I personally remain convinced that experiential education is a necessary condition for developing "operator" level judgment , and I interpret Leontiev as aligned with that belief.
  3. Galperin described the system of necessary conditions, shifting from descriptive theory to prescriptive practice, creating a system of instructional design based on creating conditions where learning happens predictably.
  4. Podolskiy adjusted Galperin's process to become practical, or at least possible, in classroom settings.

Example Usage

The next two chapters are live examples of Galperin's formation of thinking. They are designed for you to experience rather than skim or read for structure. Both chapters perform what they teach.

Both chapters teach how to transmit expert thinking. Learning HPC tradecraft is useless unless it can be replicated. The next two chapters therefore focus on teaching you the replication mechanism.

Then, the final chapter of this book describes the two large books and the commitment required for their apprenticeship. At that point, the choice is yours.