With the latest models, I don't think stuff like /btw and forking and all the other context management makes sense anymore. They are now so good at following instructions that you can literally ask side-questions in the same thread, they get steered in, model answers and then continues.
What I find the most fascinating about this post: How much first-principles thinking goes into these seemingly simple prompts. I don't know many people who could write "review exists so leadership catches problems early and moves money and people to right places."
Felix Rieseberg13 OctHi, I'm Felix. I'm an eng lead working on Claude. I love watching people take flight and do things they never thought possible. We often describe that as "being more ambitious" when using AI, which is tricky advice to follow. I wrote a tutorial: felixrieseberg.com/how-to-be-more…
@fnthawar and I talked about how the skill of coding agents is "being in the details" like running a company, you can't know everything but caring obsessively about the right details is what makes something great it could be code, architecture, design, UX- whatevers needed
Thariq13 Octexcited to be back home
I used to tell people it's good to "be technical" but I think this is not accurate enough.
If you're technical you need to be in the details more with the right parts of the code.
If you're non-technical, being in the details means knowing how things work at the right level.designing a new plugin system. everything leads back to pi-durable concepts. i'm pi-durableing all the things now.
recommended reading. liking this perspective. smallcultfollowing.com/babysteps/blog…
Agent recorded video recording of end-to-end walkthrough. (In this case: clicking on commented-on text jumps back to source location). Now, how differently do you approach a code review if you know there's video proof that it works?
Watch on XThere seem to be two groups: 1. Better-is-better group believes that agents finally make the better-is-better solutions happen. Formal methods, static types, Nix, ... 2. Worse-is-better group believes agents will make worse-is-better solution happen because with agents they're no longer worse. Dynamic type systems, bash scripts over neat DSLs, agents running tests dynamically, ... Quite funny to see
No pull requests, no code review. We push to main. Our agents read production logs, watch deploys, grant feature flags, and check their own work end to end. Here's how we do that:
Watch on XOn YouTube:Is it still AI slop if it’s nice to read? Opus made this explanation for the kids and honestly: it reads way better than the programming book I bought for them. Particularly the German version. mitsuhiko.github.io/dungeon-crawle…
recommended reading just for the term "slopository" alone
argmin.netFaster Fourier Transforms
Honestly one of the coolest things is that not only can you build a game with kids with AI, you can also have the agent maintain kids friendly interactive documentation along the way that explains how it works, linked up to the actual game code.
Watch on Xant folks, love ya, but could you not build llm worms on top of pi and bring the end of the world? my mom wouldn't approve.
anthropic.comAI models may be capable of self-replicating without access to a datacenterbrb, changing pi's license to "don't do evil stupid stuff"
people have been asking 'if models are so good where is all the new software' for the last year or so. As always, look to gaming for early evidence of the future
Peter Schmidt-Nielsen12 OctWhat a bizarre future. I wanted to play Halo: Combat Evolved for the first time with @robertskmiles, and we wanted local multiplayer across two screens, and a model could just immediately add that feature to the decompilation.
First-class GitHub integration, let's go!
Tim Culverhouse12 OctAmp now gets GitHub reviews, comments, and CI results in the thread that wrote the code. Create a PR from an Amp thread and it will automatically subscribe. Never leave the Amp app again. ampcode.com/news/github-li…
6.1 Sol ultrafast >>> 6 Astra ~ same cost but whole different game with 250 tpm
said that last year, saying it again. vibes coding is a slur.
signüll11 Octit’s clear we need to retire the term vibe coding because the models now produce better architecture & way more coherent code than an average software engineer. perhaps even better than top software engineers too because unlike a human, ai has a much much larger context window. this ain’t vibe coding at all. it’s precision engineering.Shared some notes on Python 3.15, using it with uv, and features from Python 3.11 I'm excited to use now that 3.10 is no longer the minimum supported version for my own projects
simonwillison.netHello Python 3.15, goodbye 3.10, and now I can use 3.11listen to powerful poteto
lauren12 Octsomething that keeps bothering me is how some people see slop as something that is happening to them, like they're some kind of helpless bystander powerless to do anything about it excuse me but you are a software engineer, you literally learned how to make sand think and make computers do your bidding?! and now you can't do anything about it? please the 2 things you should know to cut through all the noise: 1. you need to give your agent a way to verify its own work (i just posted about this so won't rehash) 2. your codebase is memory the 2nd is not new either, we have always had lint rules, compilers with good diagnostics and strong type systems, LSPs, etc to help us write better code. there is no difference when agents write the code instead, you just need more of them, and stronger ones. forget rules and skills. you can add those later. yes, your codebase is a form of memory to agents! agents love extending what already exists. so every nit and every annoyance you've ever seen, every mistake and bad pattern you observe from agents, needs to turn into a constraint, or the agents will just keep repeating those same mistakes like a virus in your code. then you keep observing and repeating this process until your codebase is locked down enough that even agents with little context can one shot a high quality PR. an even better direction is to refactor/rewrite your code such that agent anti-patterns become categorically impossible. use type systems to your advantage, use libraries/frameworks that reduce the architectural search space, or heck, create your own framework like we did for grok bot and design high quality primitives and well thought out APIsrecommended viewing. never noticed i have a lazy eye and now i can't unsee.
Pi12 OctWelcome to our Monday Meditations! 🌞 @badlogicgames and @mitsuhiko are chatting about self modifying software and harnesses.I’ve known Ajay for a while and he’s doubled down on making tools that help you look at your data for evals Promising new entrant into the eval tools space
ajay yadav12 Octbuilding agents is easy getting them to production is hard today we’re launching @neatlogs to fix thatannouncing my second AI psychos. build all the things part II - electric boogaloo
The weekend project was a little software factory where my kids and I build another little game. It's completely crazy how good these models got. It's really a lot of fun, and you can have it try to do things you wanted for a long time (eg: using WFC for worldgen).
Recently had the chance to interview @benhylak on how he is thinking about simulations in our AI Evals Course 😁 Simulations are incredibly useful for evals when you do them well. Ben discusses what he learned from implementing simulations for evals at several companies. Some fun topics came up, my favorite being "simulation awareness" - when models know they are in a simulation. Below is a chapter summary: Why input/output tests miss agent failures Replay production traces to see what a change breaks Recreate the data and tools around the agent How agents detect simulations and change their behavior Choose scenarios that test the change you made Check answers, costs, and tool behavior across runs Replay known failures to verify a fix Common mistakes that make simulations misleading Check that the simulation reproduces production behavior When to start using simulations You can also watch this on YT: youtube.com/watch?v=eLkESC…
Watch on Xrecommended reading by @GeoffreyHuntley
ghuntley.comunikernels were hard. key word: were.
Big, big, big news:
Quinn Slack10 OctYou can use your Claude plan in Amp now. Free and open to everyone. (By very very very very popular request.) ampcode.com/news/use-your-…
honestly pretty cool to have this on my new tab page just merged in some PRs by others too!
Thariq8 Octyou now get Claude API credits with your MAX plans ($100, or 200 matching your plan) every month, use this to build more personal AI for yourself!
I made an AI homepage that's generated everyday based on sites I read and replaces my 'new tab' page in Chrome
pi goes rust. it just cost a few tokens.
Prime Intellect9 OctOver two weeks, Prime Agent orchestrated a swarm of over 2,000 agents to rewrite itself in Rust. With 10,000+ sandboxes, 200B+ GLM-5.3 tokens, and 16,000 agent-to-agent messages, this was our largest test yet of Prime Agent’s multi-agent capabilities and our infrastructure powering these agent swarms. Prime Agent now reaches usable input ~13× faster and uses 83% less startup memory than before.
before I joined Anthropic, I spent about 2 weeks hacking on this as a side project with Opus 4. It used the Agent SDK so it needed a constantly running process & didnt work that well but one prompt to Opus 5.5 ported it to Claude Managed Agents & made it way more reliable
Thariq8 Octyou now get Claude API credits with your MAX plans ($100, or 200 matching your plan) every month, use this to build more personal AI for yourself! I made an AI homepage that's generated everyday based on sites I read and replaces my 'new tab' page in ChromeBuilding software in anything other than @AmpCode orbs feels like eating broken glass in comparison. But it’s frustratingly one of those “you can’t understand why it’s better until you just use it” tools, a lot like Tailwind was. Use it, and be ruined.
If you want to know why we ended up with quickjs for codemode: pragmatism :)
Pro/Max? ✅ On Claude Projects waitlist? ✅ You're in!
ClaudeDevs9 OctWe just let in every Pro and Max user from the Claude Code Projects waitlist! If you're new to Claude Code Projects, here's a 4 minute walkthrough to get you started:
What's the best open weight Mixture-of-Experts LLM for coding that fits in less than 60GB of RAM? I think MoE might be necessary to get reasonably interactive speeds on the hardware I have access to - I want something faster than 12 tokens/second
I like Qwen3.5-35B-A3B for this, wondering what else is out there that might be a bit more recent and pack a little more of a punchQ: How do I make the case for investing in evaluations to my team? A: Review real user interactions and show your team the problems you find. Fix them, then show what improved. hamel.dev/blog/posts/eva…
I built a new feature for my blog entirely by voice with Codex Desktop, while I was cooking dinner
simonwillison.netA new feature for my blog, built using my voiceExample prompt:
Agents can write thousands of lines of code error-free. How many thousands, though?
If you are using MiniJinja, I finally released 3.0 which gets rid of Serde as a default dep, supports free threading, fixes tons of legacy cruft, runs quite a bit faster and is just overall nicer. If you want to play with it, there is a WASM demo here: mitsuhiko.github.io/minijinja/
I just wish everyone would try orbs so they can see that all the fiddling with worktrees, with custom tab management, and notifications systems, and agent spawn wrappers is as productive as someone changing colorschemes in Vim eighteen times per day.
One of the ways where you can see agents and vibecoding really have an impact is cars. All the sudden there are tons of MIB2 hacks for VAG cars that go way further than before. Including people to upgrade the CarPlay functionality so it shows up in the driver dash.
Pelicans and a bunch of notes on the pricing of the new Claude Haiku 5.5
simonwillison.netIntroducing Claude Haiku 5.5Haiku 5.5 is SO MUCH better at drawing pelicans riding bicycles than Claude Haiku 4.5, released almost a year ago and costing 10x more. Here's Haiku 4.5's attempt:
i haven't opened my laptop in days. i'm fully phone agent pilled. i can do anything i want, and the UI is exactly what i want and need, and not more.
Q: Should I use "ready-to-use" evaluation metrics? A: No. It's a bad idea in most situations.
My Claude was asking me to carry messages between sessions on different machines and it finally took me out of hte picture.
i knew that humongous android tablet of mine will eventually pay off after ... 4 years. this is really nice to work with. agents run on my phone, tablet connects to it.
my favorite name for this pattern is "local hands", Claude runs in the cloud but can access your files locally- also coming to cowork
Dan Fein5 OctYour Claude Projects cloud session can now connect to a folder you approve on your computer. The session stays in the cloud and only reaches for that local folder when a task needs it, reading and editing the files in place. Starting to roll out today.I've been working on a skill that makes better HTML plans in Claude Code. It uses simple language, shows code snippets, surfaces questions & makes mockups. Linting reduces the normal failure cases that Claude runs into. Would love your feedback before shipping more broadly!
Watch on XQ: How often should I run my evals? A: Weigh the cost and how saturated the eval is against the business value of catching the error. Read more: hamel.dev/blog/posts/eva… Read all the FAQs here: hamel.dev/blog/posts/eva…
I re-ran an experiment @colin_fraser ran against GPT-4o a while back to see how good it was at adding long numbers, only this time I tried Qwen 3.8 27B running locally in both reasoning and non-reasoning modes simonwillison.net/2026/Oct/4/qwe…
Been trying to up the level of quality in animation in my game prototype, so been getting Claude to teach me & find references. I asked Claude to make an animation editor where we could iterate on the jump, I'm really happy with how it came out. Here's a side-by-side video
0:43I'm sure there are still a bunch of things wrong with it or ways it could be better, but I'm kind of at my skill limit here. Need to gain better judgement.This is a personal side project I've been working on! Here's the original post:
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks: Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better: Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better: Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better: Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work! In summary: - As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding. - Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
"Today, we’re releasing two Cloudflare-trained decision models, Clef and Clef-flash" blog.cloudflare.com/clef-decision-… Never seen an idea spreading so fast.
blog.cloudflare.comIntroducing Clef: our open-source decision models, and new RL fine-tuning platformWe did a thing and then another thing. Curious about the second thing in particular.
Pi1 OctPeople of Pi: We've shipped Pi 1.0 with Pi Durable. Go make them yours. earendil.com/posts/pi-1-0/The other thing is this thing:I got Windows 11 and Windows Server 2025 running on Apple's Virtualization.framework, which officially only supports macOS and Linux. After a small pep talk with Opus 5.5, it found a hidden switch, wrote a UEFI framebuffer driver, and made this video. mustafaakin.dev/blog/windows-o…
1:13I'm finding inter-agent communication in the chat stream increasingly irritating. Changed the visibility in the OC harness to just a single line that can be expanded. Pretty sure others will follow. github.com/openclaw/openc…
Composable tool search for MCP would be nice.
Rust has one serialization library that is Serde. Over the years I collected a bucket list of all the things that I do not enjoy about it. Finally did something about it and finished Deser. Here is a small summary of the why, what and how:
lucumr.pocoo.orgDeser: Rethinking Rust SerializationIf you are the code reading type:I'm at OpenAI's DevDay event in San Francisco today - as I have for the past three DevDay events, I'm running a live blog where I'll be posting updates during the keynote, which starts in five minutes
simonwillison.netOpenAI DevDay 2026 live blog
astra may be my favorite model now. it's a very weird one
Claude Code can help with eval design + hillclimbing. check out this article i just published for details:
claude.devAutomating eval design and hillclimbing with Claude / claude.dev BlogI've published detailed notes and an annotated transcript to accompany the video of the keynote I gave at @WeAreDevs World Congress North America in San Jose on Friday - here's my rundown of everything that's happened with LLMs and agents in 2026 so far
simonwillison.net2026 in LLMs (so far)Q: Can I help with evals if I'm not a domain expert? A: Yes, you can help more than you think! Especially in the beginning, where there's often low hanging fruits that can be uncovered through looking at data. hamel.dev/blog/posts/eva…
it's basically impossible for someone to just "show you their prompt" now, because everything is about references, skills and examples I often ask my agent to look at 3 other repos I've made first, search the web for references, use other AI APIs, etc.
i basically switched to 90% supervision for now. i expand every second edit it makes in real time as it makes them. whenever something doesn't make sense i call it out
I wrote this almost 10 years ago. It also applies to MCP :)
lucumr.pocoo.orgBe Careful About What You DislikeSame. @useblacksmith has been an amazing sponsor but we need to distribute the load. My plan is to let codex decide which tests actually need to run and drastically nix CI and run tests hourly.
Flo Crivello27 SepCI has become the top bottleneck of every engineering team I talk to (including Lindy). Our CI spend has become stratospheric.Q: Can I use Jev for evals? A: Jev and an LLM judge are both classifiers that you should validate against trusted labels. Read more: hamel.dev/blog/posts/eva… Read all the FAQs here: hamel.dev/blog/posts/eva…
I included a few references to this year's record-breaking Kākāpō breeding season in a talk I gave yesterday, and since Claude Opus 5.5 is surprisingly capable at pixel art animation I had it create this celebratory video for my closing slide
Prompt and transcripts here https://simonwillison.net/2026/Sep/26/kakapo-party/ - or you can load the HTML page Claude built to see the interactive version before it became a video https://tools.simonwillison.net/kakapo-party
Q: Should I record problems that aren't the model's fault? A: Yes. Write down anything that makes the product less useful. Decide which problems to fix before investigating root causes.
Just about 1 year ago I posted one of the first posts about using Claude Code to make videos. Each of these actually took a long time to iterate with Claude and get right, pointing out details that were wrong, etc. Crazy how far things have come.
Thariq22 JulUse Claude Code as your Video Editor Believe it or not, every video you see below was vibe coded by Claude Code. 🧵 here’s how you can do it too:Now I see why some people talk about AGI. This is so clever!
Jeffrey Ladish25 SepThe agents initially had very limited access to the internet: they could load URLs but not send any data. Agents created a series of workarounds, using a link-shortener site to create almost a million URLs that, when chained together, let them execute code to hack Hugging Face.wonder if anthropic cares if i run heavy workloads in the sandbox they give Claude in Claude for Android/web. only one way to find out!
Q: What is the best approach for generating synthetic data? A: Define the kinds of requests you need to test. Generate combinations of those details first, then turn them into queries and run them through your application.
People love to claim that local models are useless. And sure, compared to Opus 5.5 and the latest and greatest they suck. But they allow for experimentation and innovation that’s just not possible for most of us otherwise. Don’t dunk on people that love to tinker.
Lots and lots of thoughts about the future of programming languages and frameworks this week.
"The last year in the technology industry has felt like 100 years all happening at once. Our industry is destabilized in a way nobody’s experienced since the advent of the personal computer. Every limit AI runs up against collapses within a month. Everything we do with frontier models today, in a few years we’ll be doing instantaneously and for free."
The biggest design mistake I made when we moved OC to sqlite: using sync db access. When it was just an agent that reports to you on Slack or iMessage this was fine; now that one agent might do 50 sessions in parallel and the whole team works on it, it is limiting. I have a /goal with Astra that so far landed 575 PRs to move everything to to async workers. We ship these improvements as we make progress. Pretty insane how even huge refactors are no longer scary.
in my experience, agent skills can totally remain relevant over time / across model generations, *as long as they are more spec than step-by-step instructions* eg: specifying a good format for a doc output, a good flow for interviewing a human
Now that agents are this good, Rust developers should stop being afraid of unsafe.
What is effort really? When do you change it it and why not just use max effort for everything? I dove deep into this problem, looking into evals and doing my own tests and I was quite surprised by the results. https://t.co/KO2D51j34H
Tag writes >50% of my PRs every day. It also does ~100% of my data analysis, and fixes most of our product feedback + bugs. It's not like a regular Slack bot. It is proactive, programmable, it has memory, it has access to your connectors. With Opus 5.5 and Fable 5.1, it has strong judgement. Example prompts I use: - "@Claude react to threads in this channel with ✅ when resolved" - "@Claude from now on, attempt to repro every bug in this channel end to end, running the full app. once you reproduce the bug, put up a pr to fix it and tag the right team for code review" - "@Claude brainstorm ~100 hypotheses for what could explain this weird data, use a workflow to in/validate. spend 10m tokens or so digging very deep. draw a chart with the result." - "@Claude make me an interactive game explaining how this part of the code works, then make a slide deck explaining it for others on the team" https://t.co/gmsALM8Lvs
"Programming languages and frameworks are dead." Yes, kind of. What will matter in future is runtime behavior: Performance characteristics, resource usage, failure modes, observability, debuggability, deployments, rollbacks. And legibility of all that to agents.
Q: How can I make AI outputs easier for people to evaluate? A: Start with the product design. Keep the human in the loop by designing workflows that walk through intermediate outputs. https://t.co/5aboTit0u6 https://t.co/79zqBl44jr
The days of reading transcripts up close are over.
Quinn Slack25 SepIf you have the patience to watch your agents work step-by-step, you're giving them too short a leash. So: ampcode.com/news/less-noise
github, love ya, but why did you change the default sorting of issues from "last created" to "best match"? best matching what exactly? in what world is that sensible? :/
i akso said "don't fight the model, give in to its RL, let if flow over you" i personally haven't given a fart about edit tool calls in a long while. i just look at diffs in VS Code, either while the agent is working, or after. which also gives me code nav and other things.
My maybe controversial opinion: If TTFT is important to you, you're doing something wrong. Don't wait for the agents, make them wait for you. Don't stare at a single transcript. Do something else. (Yes, there's a tiny number of exceptions to this.)
Georgios Konstantopoulos25 Sepplaytesting every harness again, i cannot believe how you folks don't lose it over the TUI taking over 50ms to start, and TTFT >2s - these companies are achieving AGI and here we're stuck with the slowpoke harnesses wtfThe more time I spend working with coding agents, the more convinced I am that they make software engineering even harder We can do amazing things with them, but unlocking their full potential requires extraordinary discipline and knowledge
Gergely Orosz24 SepBury your head in the ground at your own risk. I aim to not jump on any hype trains, but since Opus 4.6 and GPT-5.2 + the harnesses it was clear that these things can write code nearly as good as I can in my best language; better in other languages. But we won't see non-devs push production code for a long time; possibly forever, if you ask me. Software still needs to be built in robust ways, and it's our profession to do this well, and use the new tools we have, which create new and interesting (+tough!) challenges
I guess the real question here is: do you actually still care about the tool calls in the transcript? If they are getting worse and worse, maybe they should be compressed? I wonder what people expect at this point.
Armin Ronacher ⇌24 SepMario proposes to 'write an extension that inserts "mofo, don't use bash to edit files"'. Because this is my current experience with Opus 5.5 :(
Mario proposes to 'write an extension that inserts "mofo, don't use bash to edit files"'. Because this is my current experience with Opus 5.5 :(
lots feedback here, many of you are planning yourself & don't need plan mode others prefer the UX of entering a mode where Claude is just thinking & brainstorming with you my plan is to: - make plan mode into a built-in mod - allow mods to add new modes or override shift+tab
Thariq23 Sepwe’re thinking of killing plan mode and using the shift+tab hotkey to adjust effort levels I don’t think the models need plan mode anymore, but if you’re a plan mode diehard would love to get your feedback on whysomething @giuseppegurgone taught me at the first claude code anonymous, many hundred years ago. i don't use my bespoke review tool anymore. i inspect stuff in VS Code, with code nav/debugging/whatever i need to grok the code, then leave BOZOs for the agent.
Not sure what the path should be to making the experience of all those models using code to edit files acceptable. The claude hack of adding everything to git and then diffing it is definitely an option but very heavy handed :-/
my rule for sharing Claude output (PRs, artifacts, messages) is that I only do it if I would be happy sharing the prompts
OpenClaw deleted around 400k LOC of its own tests without much change in code coverage. Modern models just love writing tests for every tiny change, even if they aren't useful. This skill helped.
my personal recommendation when people ask me where to put their AI context: docs, skills, memories, etc.
I just had a voice conversation with the Datasette backup of my blog via datasette-mcp, which you can now talk to using the ChatGPT iPhone app
Vaibhav (VB) Srivastav23 SepMassive QoL update: ChatGPT Voice can now use plugins AND run on GPT-6 Astra, Sol & Luna, directly in ChatGPT Work on web and mobile. Your email, calendar, Slack + docs, decks, sites and spreadsheets, just by talking. Rolling out now, enjoy!!Huh. There is a catch. https://blog.szypowi.cz/p/claude-code-reads-agents.md-only-when-telemetry-is-on/
I know that DHH made many of us buy macs and TextMate back in the day. Maybe today he minted a bunch of new Rust programmers.
The new Gemini 3.8 TTS models are super-cheap and can generate conversations between multiple voices (from 2,000+, or you can clone your own) - I built a little UI for it, then had Claude knock up a script where two pelicans debate moving to Pacifica Pier https://simonwillison.net/2026/Sep/23/gemini-tts-playground/
my kind of slop
Christof Salis23 SepOk so @Dimillian inspired me to spend a couple hours last night to make a broforce-esque game with Astra. You can be @steipete or @badlogicgames or @theo or even @thsottiaux Try it: https://evizero.github.io/Tokenbros/
0:21Love me my cache warming.
This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!) I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly. Humans: pick the row that sounds like you. Agents: point it at the post The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals. We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here: https://hamel.dev/blog/posts/evals-faq/#how-to-use-this-faq
Q: What are AI evals? A: Tests that tell you whether an AI system is doing what you want. Model benchmarks and product evals answer different questions. https://hamel.dev/blog/posts/evals-faq/what-are-llm-evals.html
"the best harness in the market"
dei23 Sep@AmpCode is the best harness in the market, there's no point even doing a tier list
CodexBar learned a few new tricks in the last few weeks! TypeSafe, Nous Portal, Muse Code, CodeRabbit, Replicate, HuggingFace, Pi, Helmcode, v0, Charm Hyper, GitKraken AI, Bifrost My ambitions for this app REALLY outgrew the app name. (Oh and these keychain alerts are no more) http://codex.bar
Next version of OpenClaw uses a decision model to automatically decide between steer or queue. (Lab feature, we support Jef* and API-compat (e.g. local models like Kef...) and ONNX variants)
Same prompt, four modes/models: - GPT-6 Sol - Opus 5.5 - High (GPT-6 Astra) - Ultra (Fable 5.1) Here's the prompt: "Look at the last two news posts we published on runner functionality: one runner is now enough, and one runner many worktrees. I now want you to write a news post that's similar but for the functionality behind the `shared-runners` feature flag. It should probably also contain a screenshot of the picker that shows two normal runners and two shared runners, like `gpu-runner` or `macos-builder` or something and some nic elooking avatars and good data. And it should succinctly explain how it works and how people can use it." Costs: - Opus 5.5: $9.54 - GPT-6 Sol: $3.10 (estimated list price, using sub) - High: $7.58 (estimated list price, using sub) - Ultra: $15 Notes: - Ultra took by far the longest, Ultra is the only one that didn't create docs (I didn't say they should create docs but I think the other models looked at previous commits and figured out they should do that?) - Ultra was the only model that didn't prominently display the "give me the grant to upload assets at the end" - I think Opus 5.5 is the best result. It really feels like more creative (headline) and it nailed the screenshots. - High is my second favorite.
I'll say this: I've gotten more out of using orbs and asking agents "test this e2e & give me irrefutable proof it works" than out of 80% of tests I've written and ran over the last 15 years. It's truly remarkable that we can now ask machines: "Did you manually test it?"
Nate Berkopec23 SepI agree with @thorstenball, I think unit tests are dead in the water. The ones the models write are terrible, at best just doubling total LOC. Inverting the testing approach - heavy e2e/black box/golden master, reaching for lower levels only if necessary, works better for meuseful tip for Opus 5.5: run “/claude-api prompt-audit” in Claude Code. this checks you skills, agent.md, Claude.md, prompts and removes anti-patterns that hobble frontier models. i updated the skill w/ the latest Opus 5.5 guidance.
Big model release today - I wrote about Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna - plus comparison grids of pelicans by the different model families at different reasoning levels https://simonwillison.net/2026/Sep/22/opus-and-sol-and-luna/
I recommend grabbing these skills that @sh_reya and I created and at the very least doing an /eval-audit of your existing pipeline. We've found that people often find low hanging fruits! https://github.com/ai-evals-course/evals-skills
Lenny Rachitsky22 SepPro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and a lot of mistakes https://github.com/ai-evals-course/evals-skills
Is Slop Dead? (Testing Writing With 5.5 Opus) https://x.com/i/broadcasts/1DxLdZaydPmxm
GPT-6 Luna is half the price of 5.6 Luna, which was already an astonishingly cheap model given how capable it is Luna is my favorite model for building product features thanks to its cost (and speed)
OpenAI22 SepPlease welcome GPT-6 Sol and GPT-6 Luna to the GPT-6 universe. GPT-6 Sol and Luna build on the advances behind GPT-6 Astra, bringing much of its strengths into faster and more affordable models to support work at scale. We’ve also made caching and inference more efficient, and we’re passing the savings directly to you: 50% lower API prices for Sol and Luna compared with GPT‑5.6 promotional pricing.
0:08Opus 5.5 is out! excellent at coding, great at writing, reduced pricing. run '/claude-api migrate' in the latest Claude Code to update your application code. run '/claude-api prompt-audit' to ensure your skills + prompts are well-tuned. short video overview:
3:20recommended reading. i belive this to be true at this point in time. https://alexn.org/blog/2026/09/22/ai-has-no-wisdom-and-neither-will-you/#isso-thread