While this is very impressive benchmark-wise, GPT-6 Astra showed us that benchmarks don't always correlate 1:1 to intelligence of a model.
When Astra launched, I think Artifical Analysis showed that it was on par with GPT-5.6 Sol and lower than Opus or something like that? Then, they updated the scoring.
I hope that more open source models, including this model, to be "as good to use" as Astra.
Apparently the scoring on a lot of difficult benchmarks can also be extremely influenced by something as simple as waiting for the model to exhaust its reasoning, realize it hasn't come to a conclusion yet, and give it a simple prompt like "you can do this, I know you're capable, please keep going".
I don't know why, but the benchmarks still fails to cover the difference between large models and small ones. The small ones are great for many things, including general coding, but the larger ones, like fable and astra, have some kind of intelligence that is not present in the small ones.
I think a curriculum like this is neat and might help with interviews since you go wide and have a checklist of things that you need to learn.
I'm on a totally different path for learning inference engineering. I self-host a voice AI app that has ~2000 monthly active users on my own GPU box.
This forces me to learn about production serving, KV cache, quantization, inference engine, observability and economics, prefill optimization since I'm optimizing for TTFT instead of decode speed, and many more.
It's fun since every optimization you do directly translate to a better user experience or allow you to serve more users using the same hardware.
This 100%. I was Anthropic-pilled. I had a $200/mo subscription and I only used Anthropic models. I was frustrated by the verbose output and the writing style. I tried ASD-STE-100, it helped a bit, but it's still too verbose for my taste.
Then I tried GPT 5.6 Sol. It's night and day.
I think Anthropic just RL too hard on coding capabilities and never calibrated or benchmarked the writing styles.
Yeah I don't know that any of the benchmarks index on "understandability". I'm amazed at how Claude can produce a page of text describing what it did and it can take me a full five minutes to decipher it, often just to find it's something I could have expressed in a simple sentence.
The simpler angle is that more text lets them bill you more. I don't think that was necessarily their intent, but it does mean they have a negative incentive to fix it.
Have you tried asking it for a lay explanation of what it did? That’s usually all it takes for me. Sends garbage -> request -> sends something readable
No, I’m not interested in fighting my model all day long. Plus is fucking annoying to talk to and collaborate with, so I’m not using it when Sol 5.6 is about 1000 times better in that regard. I have colleagues who spent a lot of time trying to improve their harness with user rules and whatnot and Opus really does not want to follow them.
Yeah but Sol shows it is possible to just send the readable explanation in the first instance. And I don't want to spend tokens and time on asking for a better version of each response.
When I ask it to make a CL description, it's worthless unless I tell it to dumb it down as much as possible, assume the reader has zero knowledge of the codebase. And then it makes a perfectly cromulent description that just needs a touch of trimming-down. If I don't do this, the description is just a wall of gibberish and paraphrasing of every little thing it encountered.
I've tried telling it to "fix" comments with varying degrees of specificity and in my experience it just... fundamentally doesn't get it. Presumably using a different model for it would help.
My theory is that Claude's learned approach to comments is to treat them as a sort of persistent in-band thinking trace, or a "memory" tied to an in-code location, which is a little at odds with the way humans use comments (human comments are intended to be read and understood by other humans, whereas Claude comments are their own dialect).
I bet this is a result of iteratively training Claude on output from other successful Claude sessions. Presumably it's good for making benchmark scores go up.
I also suspect comments are very much tied to how Claude reasons because not only are they bad comments, I can't get rid of them. Commenting is the one area in which I've been unable to get Claude to respect any rules. It can follow code conventions I prefer, it can do other things, but it can't keep the comment volume down.
My CLAUDE.md has rules about not including any redundant comments in the code that are obvious from the code itself. I reiterate that occasionally while working. It's absolutely disregarded and any Claude-written code is full of comments. Some of them are simply redundant, like "Collect Foos and pass them to the requested sink" on a function that's void CollectFoos(IFooSink sink). But worse, many comments include in the moment reasoning like "added parameter bar because we can no longer use the frob to automatically derive bar". That's stuff for a commit message, or just a mental note, and absolutely not for comments.
I haven't found any way to stop Claude from doing these, so I have to tell Claude afterwards to clean the comments up. Which it does, making a note in memory to comment less, and it still does the exact same thing next time.
> Commenting is the one area in which I've been unable to get Claude to respect any rules.
Exactly my experience! Since the release of Opus 5, no amount of instructions helps. In CLAUDE.md, in a separate file, in memory, as brief bullets, as long detailed guides, with reasoning from medium to max — nothing.
Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.
I have a relatively low bar for prose, could live with some junk. But Claude's comments are _poisonous_. They always require maintenance, instantly become out of sync with the actual code, and are a token black hole — for all agents, but especially for Claude itself.
Gave up and canceled Anthropic subscription yesterday. To my taste, it has become unusable for coding.
> Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.
For me, Claude knows how I want the comments due to all the memories and CLAUDE.md, so funnily it's now enough with even a brief groan from me like "Come on, the comments" and then Claude goes through its recent additions and fixes comments quite well per my long-term instructions. But only ever during an extra pass that I initiate, never during the initial writing of the code.
> But worse, many comments include in the moment reasoning like "added parameter bar because we can no longer use the frob to automatically derive bar". That's stuff for a commit message, or just a mental note, and absolutely not for comments.
I've noticed this a lot, and before your remark I couldn't put my finger on what was wrong. Now I know: Claude is writing its thought processes and maybe parts of the conversation it had with you as comments in the code!
I always end up manually trimming those comments, which is cumbersome.
It also loves to reference internal notes and scratch docs that never go into source control, so a reader will have no idea what it’s talking about. For example:
// load_tree() loads the binary tree with data, but only the recently updated data, not all data (INTERNAL_NOTES.md section 4)
Ok but nobody reading the source code knows what this doc is. You don’t have to cite it.
It also seeps into all documents and artefacts it creates.
Claude will include actual comments ("// ...") into Excel sheets, and include the thinking that led to the output, instead of just focusing on the final result.
So if Claude questioned whether a vendor should be replaced, and you said "oh no, they are critical and we're already negotiating a great price") you'll now need to be careful to not send your vendor a document that contain text like ("Cost: X. // Management confirmed to not fire this vendor as they are critical to infrastructure and a better price will be negotiated later")
I'm not sure why you all have issues with CC commenting too much. My rules in the CLAUDE.md specify that comments are evil, never comment unless there is an actual need to explain a WHY and since I do read what CC writes, if I spot it still adding such WHY comments and they make no sense, I'll have it adjust, in many cases by removing them.
Given the code base has a minimal amount of such comments, it's also less likely to go "copy what the rest of the codebase does".
Of course I've now jinxed it and some update will cause it to ignore the instructions coz I didn't write them in the new model's style or something.
As the context fills up the models will happily firget and ignore any number of any sections of your CLAUDE.md/AGENTS.md.
Edit:
I've had explicit instructions for communication style in CLAUDE.md, in Claude's project "memory", in global "memory", in "skills": it couldn't care less where it was. It would just ignore it.
When I would point this out it would just say "Yes, I violated communication guidelines, I won't do that again". Only to do that again in the next session.
This applies to everything: code guidelines, communication guidelines, preferences, decisions etc.
They did release an Opus 5 prompting guide saying you need to explicitly prompt it to be concise or it will be very verbose. YMMV but it got better for me to some extent.
Funny, I'm the same. And if find Sol way more pleasant to work with, not to mention way faster. And Sol's compacting is superior, I haven't yet run into it forgetting something crucial from the pre-compact conversation, meanwhile Fable does that notoriously.
When they eventually make Fable available to cheapest plan, I'll downgrade. It's worth keeping for reviewing the code and the UI tasks, but nothing else.
I canceled my personal Max 20x subscription because since the 5 series models I simply cannot understand what the LLM is saying without a lot of reading and re-reading, and no amount of CLAUDE.md exhortations to speak plainly seemed to fix it.
I don’t have the energy to spend twice as long to understand its plans, and pay Anthropic prices for the privilege.
GPT seems not to have been infected by this yet, whatever it is, and Grok is quite refreshing for how normally it speaks.
I wonder if everyone at Anthropic talks like this.
If it’s watermarking, lol, good luck with that, it’s enough negative value to make me switch providers and I’m in a position to make this decision at a company level as well (we spend millions a month on Anthropic).
N=2 anecdata but just this week we were discussing setting up a couple of seats with OpenAI as a trial for switching. There are other advantages too, such as being able to bring your own harness including Ai-integrated editors / ACP clients such as Jetbrains, VS Code, and Zed. I think OpenAI and Altman are a clear step more evil than Anthropic and Amodei so I really hate to say it, but with the degradation in model output interpretability, all of the cleverness and power of the Claude Code harness hasn't been enough to offset a genuine falloff in productivity for anything other than total hands-off automation.
That said, the duo of Opus 5 and Sonnet 5 do a fantastic job at fully automated work, and Claude Code still stands head and shoulders above the rest.
Yeah OAI really nailed the communication style with GPT. It also seems just way more token efficient and faster compared to cc. Myself and all my friends have cancelled our $200 Anthropic subs. I'm using a $20 personal plan and even that is enough for my usage so far.
Also using Codex or Pi makes you realise how slow and clunky the cc harness is. Even the desktop app is more responsive and has better UX.
This is something that annoys me working in companies over the years. It’s that you can't just suggest "calm down, chasing the latest thing will not make you faster and is a huge distraction to actual work". Whether it's dot-com tech 20 years ago, latest JS framework 10 years ago, now it's the AI thing of the day. Being calm is interpreted as anti-whatever.
I didn't like to use GPT for agentic coding, review yes, but with Opus 5, well I really can't stand anything of that model. I feel that sol xhigh is even better than fable.
I think it's a deliberate steganography choice. You can spot Claude vocabulary a mile away, which maybe means you can spot distillations a mile away.
But I agree, the GPT models are so much simpler to work with, they have so much less personality and fewer quirks. They also are a little less aggressive about triple checking every little assumption immediately in a stack of 30 tool calls (but I haven't used 5.6 Sol yet so maybe that's not true anymore).
> which maybe means you can spot distillations a mile away.
I doubt this is the reason. The fact that Chinese labs are all distilling Claude/GPT/etc isn't exactly a well kept secret, they don't even bother removing the name "Claude" from the training data, so the models randomly refer to themselves as "Claude" all the time.
I think it's far more likely to be a side effect of how much synthetic data is being fed back into the models to make them better at coding. The degradation of Claude's prose has been gradual but steady ever since they shifted towards focusing only on code with Opus 4.5.
Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.
ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
At this point the options are llama-server or vLLM if you're serious about running things at your desk in the under 256GB RAM size class (70B, 120B size models). In addition to, of course, 27B to 35B size things. With of course a ton of compile time build customization options for whatever specific hardware platform you want to run either llama or vllm on.
> At this point the options are llama-server or vLLM
Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.
I think it is due to the new website? it now looks like every other vibe coded site,the only upside is that is looks more saleable for people unfamiliar with it, e.g., explaining OSPO,IT the stack you are using. they should also add a pricing page for eenterprise where they promise 99.9% uptime for local models*.
Wow it’s aggressively vibe coded. Nothing inherently wrong with that, but it looks a bit amateurish which is funny.
I’m still waiting on 98.css to become the standard for vibe coded sites. You don’t have to read docs anyway if you’re just using LLMs! All you have to do is say “use 98.css” and you have a 10/10 site
possibly hitting front page because this website is fairly new? For me, it's certainly the first time I've seen a one-liner curl|bash installer for llama.cpp, which was basically the only reason to use ollama.
The gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.
I thought the main advantage of oMLX is it's less likely to invalidate the KV cache when working with coding agents, which is key when working on a Mac because of the slower prompt processing.
Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry.
Fable recommended n-gram speculation so I'm working on that now.
Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins.
ps. ngram didn't work for me very well, but dedicated speculative model works very well
ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like:
I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.
true, switched from ollama to llama.cpp these days and it's good. wonder if this is also the best option for edge ai deployment(currently use it on desktop)
I am working on a project where we have to classify customer calls into more than 10 categories. As the client wants everything locally I tried a few local LLMs. Gemma turned out to be the best model for this task. The classification accuracy is impressive, and the client is happy that I am using an American model.
Not necessarily. Except for SpaceXAI's (haha) models, there's less risk of reality distortion in Western models. Probably doesn't affect his use case but you wouldn't really want to ask Kimi K3 for a history of China for example.
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
Most people are going under identity crisis right now because of recent LLM advancements. This post is a good example that shows that it's not only happening at the individual level, but also on the company/organization level.
Is it still worth building products or companies that can be one-shotted by AI? Probably not.
One interesting consequence is that this force everyone to be more ambitious and do something bigger that's impossible before.
I hope more people are working on something that can always bring net positive to humanity even if there are hundreds of people working on the same thing, like clean energy.
When Astra launched, I think Artifical Analysis showed that it was on par with GPT-5.6 Sol and lower than Opus or something like that? Then, they updated the scoring.
I hope that more open source models, including this model, to be "as good to use" as Astra.
reply