AI / Augmented Coding
🤖 OpenAI / ChatGPT / Transformers
- 🌐 ChatGPT is everywhere. Here’s where it came from
- traditionally language models have been using RNN (recurrent neural networks)
- in 2017 a group of Google researchers invented transformers - described in 🌐 Att ention Is All You Need
- 🌐 How ChatGPT actually works
- 🌐 How does ChatGPT work? Tracing the evolution of AIGC
- 🌐 The Guardian - Race to AI: the origins of artificial intelligence, from Turing to ChatGPT
🤖 Articles of LLMs and Agent Coding
-> 🌐 My AI Skeptic Friends Are All Nuts - pragmatic article on embracing code agents
-> 🌐 NeoSage Blog - cut through the hype blog, explaining how LLM and AI coding agents works
- On AI Hype - 🌐 The Dangerous Thing About AI Hype? - the four pillars of building with AI
- Expert intuition is not replaceable
AI today can code, write, and generate. But it cannot know.
It has no mental model of your product, your users, your trade-offs, or your non-negotiables.
- AI is not magic — it is a tool
Systems thinking
Don’t forget that every system AI touches needs guardrails, grounding, and fallback modes.
- Security is more important than speed
No customer remembers how fast you shipped. They remember when something failed.
But your real edge isn’t in being fast. It’s about being fast without compromising trust, traceability, or user safety. Cutting corners on plain old security standards in favour of speed isn’t bold.
It’s shortsighted.
- Systems are built on discipline, not hype
💡 The best Software systems in production today? 💡
They aren’t magic. They’re well-architected. They’re layered, observable, retrievable, resilient because someone treated them like systems, not stunts.
- Expert intuition is not replaceable
- Part 1 pre-training - the base model - 🌐 How GPTs Are Born: Internet Feeding, Token by Token
- basically a lossy compression of the Internet
- guesses, does not retrieve current information
- just a “pattern generator”
- stochastic, not deterministic
- Part 2 post-training - 🌐 How GPTs Learn to Be Helpful
- supervised fine-tuning (SFT)
- reinforcement learning (RL)
- reinforcement learning with human feedback (RLHF)
- 🌐 So What Are You Really Talking To?
💡 When you chat with GPT, you’re not talking to a mind. You’re talking to a model that has:
- Compressed much of the internet into its parameters
- Learned assistant behaviour from curated examples
- Discovered reasoning patterns through RL
- Aligned itself with human judgment through preference modelling
It’s not magic. It’s layers of optimisation—stacked, fine-tuned, and trained to predict your next token.
LLMs don’t understand. They don’t reason.
💡 Treat them as engines to design around, not minds to build on. 💡
- LLM models has two kinds of memory
- Parameter memory - baked into the model weights
- Context memory - fed dynamically via the prompt
- only context memory can be changed post-deployment
- for facts, use external tools (RAG - retrieval augmented generation) - like calculators (Python) to do math, or search engines to look up facts ***
- 💡 this is modern LLM architecture in a nutshell
- Know when to predict, and when to execute (i.e. call ‘tools’)
🤖 Working with Augmented Coding
The Pragmatic Engineer - 🌐 Learnings from two years of using AI tools for software engineering - guest post by Birgitta Böckeler, Distinguished Engineer at Thoughtworks
- 🌐 working with generative AI
- Mental Model keep a mental model of the AI teammate
- Eager to help
- Stubborn, and sometimes with a short-term memory
- Very well-read, but inexperienced
- Overconfident!
- Biases beware of different kind of biases
- automation bias - don’t overtrust just because it is automated output
- framing effect - generative AIs are very confident, don’t take it for face value
- anchoring effect - remember to investigate other paths that just the one suggested by AI
- sunk cost fallacy - since less human time is invested into the code, it should actually be easier to throw away generated code
- Workflow the generative agent workflow
- custom instructions - rule sets for the agents about coding style, tech stack, etc.
- plan first - break down the work into smaller tasks
- small tasks - small tasks gives better results - stay in the loop
- be concrete
- memory - keep a memory file, let the agent update it
- example the cline memory bank