From Attention to Architecture: Cohort 10 Completes Course4 of Google DeepMind AI Research Foundations

Last time in cohort 3 blog, we told you how Cohort 10 learnt about Bias, variance and they tradeoffs between them from Timi Owolabi and Femi Azeez in week 8, built its first true neural network by learning the multilayer perceptron from Chimdi Walter in week 9, the calculus of gradients from Fortune Adekogbe in week 10. They also learned the hard-won lesson that depth and non-linearity, not more data or faster hardware, are what let a network learn anything complex. We closed Course 3 promising that Course 4 would take the cohort somewhere it had been building toward since Week 1, which is the transformer Architecture.

That’s the story of Course 4: Transformer Architecture, covering Weeks 11 to 14. The course finally answered the question the cohort had been circling since Course 2’s tokenizers and Course 3’s MLPs: how do all these pieces actually fit together into the models the world calls “AI” today? It’s also the course that closed out the full 14-week technical curriculum, taking the cohort from four cities’ worth of in-person meetups to a final lecture that tied every single concept like tokens, embeddings, gradients, MLPs, and attention into one whole architecture.

Week 11: Learning to See What a Model Sees

Lab - Josiah Isong, Senior Backend Engineer, Hiyr AI | Lecture - Dr. Olumide Okubadejo, Adjunct Professor (Guest), ESIEE Paris

Course 4 opened on August 12 with a question that had been implicit in every course so far: when a model predicts the next word, what is it actually looking at? Josiah’s lab built the intuition by analogy; the same way a human reader assigns more or less importance to different words when guessing what comes next, a transformer computes that importance on the fly, layer by layer, rather than relying on fixed weights learned once and frozen. Working live in Google Colab with GPT-2 and a 26-layer Gemma 1B model, participants watched attention shift as the prompt “Michael needed clarification so he asked for” context was being varied across different layers of the MLP. They saw different attention patterns being produced and how early layers latch onto specific words while deeper layers capture broader semantic and grammatical context. A GPT-2 demo on the sentence “the cat sat on the mat because it was tired,” spread across 12 layers and 12 heads, made the point vivid: different heads specialize, some tracking the previous token, others punctuation, others resolving what “it” refers to. Josiah was careful to draw a line the cohort would hear again: attention shows what a model considers important, but it doesn’t fully explain the prediction..

Saturday’s lecture, led by Dr. Olumide Okubadejo, zoomed out to the architecture attention makes possible. He traced the field’s pivot point back to 2017’s “Attention Is All You Need” the Google Brain paper that let models mix context and hold long-form memory in a way recurrent networks couldn’t. He then walked the cohort through the now-familiar pipeline, from input embeddings, stacked transformer blocks, and a softmax output layer, making the point that transformer blocks extract and reshape context rather than inventing new information from nowhere. The technical core of the day was the query-key-value mechanism: every token plays three roles at once, asking a question (query), offering an answer (key), and carrying content (value), with relevance measured through scaled dot products and normalized into a weighted average via softmax. When a student asked the question this cohort keeps returning to; what does any of this mean for low-resource African languages? Dr. Olumide was direct: transformers need documented data and context to work well, and closing that gap is a documentation and investment problem as much as a modeling one, the same conclusion Course 2 had reached from a different angle. 

The day closed with the community, not the code: the Abuja chapter was holding its physical meetup at CoLab, spearheaded by some TRI AI Team including Gerald Iwejuo, Favour Barau, and Latifat Idris.  The team welcomed a room full of IT students, data analysts, and professionals, many of whom, tellingly, had first heard of TRI AI at Deep Learning Indaba just weeks earlier.

Week 12: Query, Key, Value, and the Cost of Remembering Everything

Lab & Lecture - Blessed Guda, PhD Researcher (Electrical & Computer Engineering), Carnegie Mellon University PittsBurgh

Week 12 went deeper into the machinery, and Blessed Guda took the cohort through two full sessions of it. Wednesday’s lab built self-attention, cross-attention, and causal attention from the ground up. This helped participants see the intuition behind query, key, and value matrices, and how they combine to produce context-aware embeddings rather than static, one-size-fits-all word vectors. Blessed mapped this onto the three transformer architecture families which includes encoder-only, decoder-only, and encoder-decoder models and what each is good for. He explained that the encoder-only models for tasks like retrieval-augmented generation, where text gets embedded once and compared against a question’s embedding, the decoder-only are for language modeling, and the encoder-decoder are for tasks like speech recognition and translation. A poll landed 74% of the room correctly identifying encoder-decoder use cases which is a solid sign the architecture distinctions were sticking.

Saturday’s lecture pushed into multi-head attention, and the motivating question was a good one: why would one attention head ever be enough? A single head can only really capture one kind of relationship while stacking multiple heads lets the model track several relationships in parallel, like subject-verb agreement alongside pronoun resolution. The elegant part, which Blessed demonstrated in code, is that this doesn’t have to multiply the parameter count. Instead of separate weight matrices per head, a single large matrix gets reshaped, splitting a 512-dimensional embedding into 8 heads of 64 dimensions each (a poll confirmed 73% of the room did that division correctly). From there the lecture moved into causal masking which involves using negative infinity inside the softmax, that stops a decoder from peeking at future tokens while still training on entire sentences in parallel. It closed with a genuinely practical topic for anyone thinking about deployment: KV caching, and the two ways production models shrink its footprint, grouped-query attention and multi-query attention, each trading a bit of expressivity for a lot of efficiency. Off to the side, the project phase was visibly heating up and some teams were already topping the leaderboard of their project. Similarly, as community meetups keep extending, we ended the day with Kaduna that weekend. The meetup was held at Colab Innovation Center and was led by Imran Musa Bello from the TRI AI Comms Team. and had 25 people in attendance.

Week 13: Teaching a Model What Order Means

Lab & Lecture - Blessed Guda, PhD Researcher (Electrical & Computer Engineering), Carnegie Mellon University PittsBurgh

If attention answered “what should the model focus on,” Week 13 answered the question that comes right after: without any notion of order, how would a model ever tell “the dog bites the man” from “the man bites the dog”? Blessed opened by proving the problem was real, not hypothetical with a live demo that showed that standard self-attention is permutation invariant, producing identical token embeddings regardless of word order, unless something explicitly injects position into the computation. The lab worked through the two basic fixes, adding position information to token embeddings versus concatenating it. Afterwards, he introduced the two big families of solution: learned absolute positional embeddings (simple, but expensive and inflexible) and the sinusoidal positional encoding from the original “Attention Is All You Need” paper, which needs no training at all and uses waves of different frequencies to give every position a unique signature. Participants pushed on the edge cases that matter for African-language NLP specifically asking about how sinusoidal encoding holds up with subword tokens, tonal marks, and polysemy. These are the kinds of questions that we love that participants are asking at this time of the cohort, showing their inquisitiveness and willingness  to tackle these problems.

The second half of the week went further, into relative positional encoding and RoPE (Rotary Positional Encoding) which is the method actually used in production models like Gemma and Llama, which encodes position as a rotation angle applied to the query and key vectors rather than as an added vector. It’s a shift in the mental model as much as math: position becomes something the model measures as a relationship between two tokens’ angles, not a fixed label stamped onto each token. Blessed also shared a side project collecting African handwritten datasets, a small but pointed reminder of how much of this course’s technical content circles back to representation. Logistically, the week carried real weight for a cohort entering the home stretch: Cohort 11 was announced to be happening next year, with completion of Courses 1 through 4 as the prerequisite; Week 14 was confirmed as the hard deadline for project submissions and a Port Harcourt watch party and in-person meetup was set for September 5, the same day Course 4, and the whole cohort, would wrap.

Week 14: Everything, All at Once

Lab - Josiah Isong, Senior Backend Engineer, Hiyr AI | Lecture - Dr. Olumide Okubadejo, Adjunct Professor (Guest), ESIEE Paris

The final lab, on September 2, was as hands-on as the cohort had been so far. Josiah walked participants through a complete transformer implementation in Keras, line by line, counting the trainable parameters in every component including layer normalization, token, position embeddings, and multi-head attention. The exercise forced a distinction that had been understood for weeks but barely spoken: parameters belong to the layer, not the data; a dense layer’s weight matrix shape is fixed by its architecture, and activations are simply what happens when data flows through it. By the end, the model the cohort had built up piece by piece landed at roughly 135 million trainable parameters, which is a small model by industry standards, but one where every parameter’s origin was now traceable, layer by layer.

And then, on September 5, it was over! 14 weeks of teaching was brought to close with Dr. Olumide Okubadejo returning to deliver a lecture built to synthesize the whole cohort, not introduce. He walked through the full transformer block one more time, but now as a single assembled machine: multi-head attention feeding into residual connections, layer normalization, and a multilayer perceptron, each piece solving a problem the others create. Residual connections, he explained, are what let you stack transformer blocks dozens deep without gradients vanishing or exploding. This a direct path for information to skip through, allowing each layer to learn only what’s missing rather than reinventing everything from scratch. The MLP block got a concrete walk-through with real numbers: a 512-dimensional token vector expanded to 2,048 dimensions, passed through ReLU to introduce genuine non-linearity, then compressed back down to 512. Layer normalization got its own moment too, walked through as an actual worked example: mean, variance, z-score, and the gamma/beta parameters that let the network learn how much standardization it actually wants. Dr. Olumide closed with decoding, including greedy decoding versus probabilistic sampling with temperature. He also gave a closing argument for why transformers won: they parallelize across GPUs in a way recurrent models never could, and they handle long-distance dependencies without forgetting. 

Where We Stand After Fourteen Weeks

Fourteen weeks in, and the arc is genuinely complete. Participants learned from n-grams and raw multilingual text, through tokenizers built by hand and embeddings that turned words into geometry. From separating signal from noise to mastering full Transformer blocks! The participants can now confidently explain the entire architecture from end to end: input embeddings, attention, positional encoding, residuals, MLPs, and final predictions. Along the way, the community stopped being confined to a screen in any single city: Lagos, Ibadan, Abuja, Kaduna and Port Harcourt all hosted their own rooms of people learning the same material together, with more meetups in the pipeline

The technical sessions are done, but the cohort isn’t. Project submissions closed on the 9th of September, the leaderboard locks ahead of Demo Day on October 3, and the top 9 teams will present what they’ve built and how they built it to the whole community. And just as Course 3 closed by pointing toward Course 4, this closing points further out still: Cohort 11 happening next year will cover the rest of the courses in the Google Deepmind AI Research Foundation Curriculum from course 5 to course 8. This cohort will be open to anyone who has completed Courses 1 through 4. You can register your interest here.

To every learner who debugged an attention matrix, rotated a query vector by hand, traveled to a room in Lagos, Abuja, Kaduna, Ibadan, or Port Harcourt to learn alongside the community behind the screen, and stuck with this for fourteen straight weeks, we say thank you for building this with us. 

See you at Demo Day!

- The TRI AI Saturdays Cohort 10 Organising Team

 Jesuyanmife Egbewale, Programmes Volunteer