The engram stuff is great because RAM is often still cheaper (or at least expandable).
My company does currently look into buying some hardware as we handle confidential data and code.
Qwen 3.8 Flash is viable on two Nvidia 6000 96GB with a wood quant because you can put the 50GB Engram into RAM and the hit should be below 10% performance.
At least that is what I have seen so far.
Correct me if I'm wrong.
I am running that on a single 6000 96GB with 4-bit quants for both weights and PLE table. Needs just 32GB RAM and fits snugly into the 96GB VRAM with KV cache equalling ~300k context tokens. Not sure if I quantized the KV
I need to support integrators for a mobile app SDK.
Bank stuff.
We have one integrator which consistently uses a truly bad AI.
My brain skips after two sentences already.
There are always +9 questions what should just be 2 max.
Redundant info is requested (e.g. please provide a change history of this function and if it will be deprecated) and everything is just unbelievably verbose.
90% of questions we get from them are already answered by the web documentation (which even has a search function) or are just non sense.
I'm truly considering to build an MCP just for them...
Scaling a model on a chip is quite hard.
ChatJimmy is based on Llama 3.1 8 billion.
Kimi K3 has 2.8 trillion parameters.
That are 350x more parameters.
I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
Scaling is definetly hard - but there is no absolute requirement to put huge flagship models into this technology (although it might be possible over time).
A fairly dumb but FAST model has it's own totally distinct use-cases even if it can't be scaled in size. Think about a LLM-infused-Alexa where the response time is instant. Where you can request it looks at hotel options in Montreal, and it starts answering in half a second rather than a few minutes.
Plus some sort of slow smart + fast dumb combo architectures might also work really well for different classes of problems.
You can change the prompt, and give the agent tools, and that keeps the agent up to date.
The agent doesn't know the date, or know what hotels there are in Montreal, it sees:
> System: You are an AI agent. The date is 11th August 2026. Your knowledge cut-off is March 2023. User is based in <date>. If you need to search for something to support the user say {search:<term>} and a list of options will be provided along with instructions on how to access. Or say {help} for a full list of commands.
> User: Can you help me find hotels in Montreal for next weekend?
The AI then interacts with the tools given in the base prompt, which can obviously be updated. So it then goes:
> AI: Of course, let me search for that. {search: hotels in montreal for 16th August}
> System: [Provides list of websites]. Say {read[n]} to read option or say {start subagent:<goal>) to register subagent.
> AI: {start subagent: List hotels on booking.com available on 16th August}.
[etc etc, then eventually]
> AI: Yes I have searched for you and I found a few options!
While you can't change embedded knowledge, a good model knowing that the date is 5th January 2040 can infer certain things (e.g. while it might not have been trained on certain deaths, it can probably guess that it should search before answering if it means a person would be 102 and their last information is from 2024)
Yes, to update the blueprint for new models two layers will be updated. That is the NN.
To instead update the data on which to operate you could use a RAG to query.
(As in "the Pathfinder 2.0 NN is on the chip; the geodata is in the OpenGeoMaps dump-DB-nightly" - not really overlapping with LLM+RAG but may give an idea in a different scenario.)
I mean, it's pretty damn reliable now. Has been for simple high contrast Q&A for a while now.
I've just been doing research and experiments for work related stuff.
Typically we've used plain embeddings for a lot of high contrast documents aka discrete facts.
However I've been working with a >1000 page document of complex procedures with incredibly low contrast where embedding falls flat.
There's top down/graph searching, bottom up/embedded; alts like colbert, reranking, reasoning, search agents and now (though seemingly quite new) specific search agent models.
Ultimately I found that a reasoning enabled search agent doing a hybrid of bottom up (with reranking) followed by top down, gave the absolute best results. Paired with Luna for cheaper and faster tokens it benchmarks pretty well even for vague references to procedures.
I would imagine that search specific models just coming out are even better and I'll have to evaluate using these but for now the above works well for us.
Having an agent get vector search results to use as anchors and then being able to explore the sections and subsections above that, then eventually digesting as much as is relevant (big context, cheap tokens) is amazing.
Very interesting and very good (though, let us say, much more recent that ancient): only, it paradoxically does suggest that the computational cost (when compared either to a simple "search by vectors distance" or to the task failures that we can have witnessed by chief implementations*) strongly suggest the benefit of 1000x speed boosting, energy conservative Taalas board.
*(Of course it has "always" worked well for «simple high contrast Q&A», ever since the base embeddings technology worked properly: that is almost by definition; it is on real world use cases, where the nuances of reality are present, that it failed miserably.)
And outside of idiotic demos, who exactly is going to ask an LLM to look at hotels in Montreal for them? This usecase has never made sense to me in the slightest
In the slightest? The user goes to their computer and types in "Montreal hotels" and Google comes up with no shortage of results, including a bit from their LLM. So that's already happening, but how do you narrow down the results from that initial search? Click around on Expedia for an hour? You probably know what you care about, just tell the LLM that you have dogs or are a vegan or whatever instead of wasting a bunch of time doing it by hand yourself.
> who exactly is going to ask an LLM to look at hotels in Montreal for them
Anybody who has a specific informal query ("SELECT ... FROM ... WHERE has_carpark AND ... ORDER BY score(has_jacuzzi , walk_distance(...) ...) DESC") but does not want to research and cross the different scattered info himself (does not want to build the virtual DB himself).
I mean it was just an example, but I get AI to do things like this all the time... ChatGPT planned my latest work trip by looking through flights, hotels and pulled together 3 options of itineraries for me, and wrote an email I could send to my client with estimated costs etc.
It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about half the area on ROM. There's a reason AI accelerators just use a ton of silicon area (each HBM3 die is >1000mm² of silicon). I imagine this is not terribly viable unless they make it a lot more space efficient e.g. using MLC ROM if they don't already, or use stacked dies with a ROM-optimized process. And then we're back to not cheap, though reticle chips were never in the cheap area to begin with.
Taalas exploits the low cardinality to store one 4-bit weight with one transistor. (They are using metal layer traces for the ROM, and connecting an access transistor to light up one of 16 options.)
Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent advancements that made attention more efficient are probably going to be very useful for them.
A full wafer like Cerebras is about 60x that, and N2P has about 3x the transistor density. So right now it's technically feasible to etch a 1.4 trillion parameter model. So roughly DeepSeek-V4-Pro class. Imagine that running a factory, for example.
Cerebras have special techniques to work around etching errors / bad cores on their wafers. This is possible since their wafers are effectively hundreds of identical copies of redundant cores. Can't do that for a globally unique model.
Etching failure in that situation would be like brain-damage in a human, all sorts of weird effects would start appearing.
There's several ways to engineer around that as the errors are detectable. There's a big literature on how to trade off speed or transistors for error correction. [1]
(Is Cerebras doing something novel? CPUs and memory blocks have been doing those things for a long time too, since the error rate is otherwise too high for normal size chips as well)
A few hundred bad bits/transistors in a trillion+ parameter model would compromise its abilities not one iota...the models are inherently lossy and resistant to "brain damage"...
Did you mean that each HBM3 stack is that large? Because it only takes one glance to see that the memory chips are much smaller than reticle-sized GPUs they sit next to.
I have found that for some personal prose-related projects, QWEN 3.6 35B A3B is an amazing model even quantized down to 4 bits. I actually find it's "writing" style as a GM for an LLM-powered solo text adventure game, better than even some of the faster/dummer frontier models like GPT-5.6-Luna or Haiku 4.5, and it runs (slowly) on a 3090 with a 80k context.
So I have faith in these embedded LLM chips when it comes to fun projects like that. I have not personally found my quantized QWEN good at agentic tasks, though, and it LOVES to make shit up when asking questions about documents in the prompt.
Now imagine you have a chip which is just that model, but can do it at absolutely insane speed. Like tens of thousands of documents a second.
Same for things like text-to-speech or speech-to-text. Think of the accessibility wins if subtitling becomes insanely accurate and fast and omnipresent.
There are all sorts of domains like that, and the trend has been such that smaller models are getting smarter and smarter. If you can stick them in parking meters, traffic lights / street crossings, mobility aids, etc etc I just see so much potential win.
Channels like project farm https://youtube.com/@projectfarm or other reviewers that are not sponsored are truly my main source of information in this age.
Some direct reviews between 2 and 4 stars are also sometimes useful.
Always discard the 5 star ones...
https://www.youtube.com/@arduinoversusevil2025 also great for teardowns of specific tools. Seeing a few really put into perspective how many companies were transitioning to trading on their brand goodwill instead of making good stuff.
right, but the cache retention time is very short for Anthropics LLMs.
5 minutes or 1 hour (with additional costs).
So you have to prompt basically non stop to not get a cache eviction.
A const prompt across all of Anthropic's subscribers could draw from a global cache rather than per-user?
Although saying that out loud makes me question it - each per-user chat and growing cache would need eventually to own its own ~contiguous memory block.
If there are no changes to the system message, yes this is possible and also likely done by Anthropic.
When there are additional local MCPs, reusability will be lower.
I think that Anthropic will bill you in any case :D
The OpenCode CLI does not work as well for me as the PI CLI.
I'm a subscriber of OpenCode Go (the sub, good value for me really) but I had not great experiences with OpenCode CLI.
It multiple times with different models deadlocked itself into listing endlessly to non ending processes (Android Debugging Bridge, COM serial log, ...).
There was also a problem where the OpenCode CLI would crash after sometime with a Bun error.
I switched to the PI CLI and have no problems with hanging processes anymore.
OpenCode Go allows for API access so I'm keeping this sub.
It is quite interesting how this is handled world wide.
For me PII is very sensitive and I advice people to be very cautious.
Every business in the EU (were I live) also has to be very careful with such data by law. Fines are now at a level were they can hurt the business significantly.
During vacation in an Asian country on the other side all of this was basically a no brainer for smaller to medium businesses.
I once rented a scooter there and the business owner had all her documents organised in WhatsApp chats.
Including now my passport plus drivers licence...
The people in general in that country were also very relaxed when it came to giving out their contact details to random businesses.
I don't want to throw shade on them, thus no country name.
Incredible friendly and welcoming people there.
The article in the OP is about a company in the EU (the clubs) not taking this seriously by outsourcing their stuff to an Irish company, who also is not taking it seriously. Hell, in the article, the CEO pretty much says “yea we ignored EU law, we will get fined, whatever”.
With parallelism of 16 you can still get around 25 to 30 tokens per user when all 16 channels are running.
Not everyone will use the model at the same time but it certainly will be tight, especially for agentic coding.
For pure chat applications this should be quite fine.
The problem with wide parallelism with most models is that it blows up your KV cache. There's open models with KV caches lean enough to parallelize inference or even to offload the KV cache itself to disk without immediately running into wearout concerns, but they're quite exceptional.
Qwen 3.8 Flash is viable on two Nvidia 6000 96GB with a wood quant because you can put the 50GB Engram into RAM and the hit should be below 10% performance. At least that is what I have seen so far. Correct me if I'm wrong.
reply