MATHIS/index
← Back to the index

N°004 · 21 November 2026

English → French Transformer, from scratch

A full encoder–decoder Transformer in PyTorch for EN→FR translation, built twice: once like the 2017 paper, once with the modern recipe (RMSNorm, SwiGLU, RoPE).

Difficulty
★★★ Deep dive
Status
In progress
Source
english_to_french_transformer_practice ↗
Topics
#pytorch #transformers #nlp
View the code on GitHub ↗

What I built

A complete encoder–decoder Transformer in PyTorch, written from scratch and trained for English → French translation.

Version 1 — the original paper

  • Custom LayerNorm and position-wise feed-forward network
  • Multi-head self-attention, masked self-attention for the decoder, and cross-attention
  • Sinusoidal positional encodings
  • Stacked encoder/decoder blocks assembled into a full Transformer

Version 2 — the modern recipe

  • RMSNorm in place of LayerNorm
  • SwiGLU feed-forward layers
  • Rotary positional embeddings (RoPE) in place of sinusoidal encodings

What I learned

Building both versions side by side made it concrete why modern LLMs moved to RMSNorm, SwiGLU and RoPE.