Context is Everything
Using Claude Code over the past few months and comparing my experience with Claude to others' experiences with various LLMs, I've noticed that the most important ingredient for LLM success is context management.
Claude Code is to some extent already programmed to facilitate this with CLAUDE.md and /clear, but at this point still requires some attention from the user. In long, sprawling chats, Claude becomes confused and tends to hallucinate more. There is a clear parallel to human reasoning: when we are sleep deprived, the quality of our reasoning drops. It is thought that the main purpose of sleep is to consolidate memories and to fine-tune your connections based on events of that day or before. This is what dreams are for. Daydreaming is not so different: when idle, the mind wanders and reflects. It's no coincidence that shower thoughts are considered to be highly creative and original. The brain is then in its "default mode", commonly associated with reflection, creativity, and future planning, while overactivity of the default mode network is associated with anxiety and depression.
An LLM does not dream. Although Claude Code now has "memory" and the ability to periodically organise its memories (called "dreaming"), this is not yet of much use in working towards gaining a sense of permanence. Claude cannot take a shower or go for a walk. He cannot sleep on it, as we humans are wont to do.
Simply put, humans' context management is superior to that of current AI. That said, context is also everything for humans. It is clear that original thoughts are rare, and when they appear it is often hard to say where they come from. I feel we are more like next-token-predictors than we like to admit.
"In every idea of genius or in every new human idea, or, more simply still, in every serious human idea born in anyone's brain, there is something that cannot possibly be conveyed to others."
— Fyodor Dostoevsky, The Idiot, Part III, Ch. 5
Do you have an entire monologue? I know I do. Our thoughts and capacity to think are directly dependent on our memory, our context.[1] Of course, there is also pre-training and even post-training, but the point remains: if you have never been exposed to clearly elucidated thoughts, it is unlikely you will produce them yourself.
Everything you remember, the totality of your memories, is your context. Let me ask you: How much of your context is occupied by advertisements? How many TikTok/Instagram/LinkedIn influencers have claimed their part of your context?
The feeling that our attention spans are being decimated is exactly due to this context-grab that is happening on social media, on billboards, on the radio (Carglass repairs, carglass does... What?). Countless ads. We are willingly surrendering our context to slop, and as such we, as a generation of humans, are getting dumber.
I don't know what the way out looks like, but for my part, I will try to improve my context management.
| [1] | Harold Bloom makes this point forcefully in his interview with Charlie Rose. |