Research Engineer
Atul Saxena
I build the environments models are trained in, and the tools that show what they learned.
- RL Systems
- Interpretability
- ML Infrastructure
Research Engineer
Atul Saxena
I build the environments models are trained in, and the tools that show what they learned.
Pushp Kharat, Atul Saxena
In plain terms
The intuitive story is that attention heads rewire themselves depending on context. We tested it and it is not what happens. Heads have fixed behaviour; the flexibility comes from which combination of them gets recruited, communicating through the shared residual stream.
A mechanistic-interpretability study of how GPT-2-small routes and represents semantic concepts. It tests two hypotheses: that semantic behaviour depends on the activation trajectory rather than the current state alone, and that meaning emerges from context-sensitive interactions between attention heads. Uses activation patching, pathway perturbation and directional decomposition across six semantic categories.