> My two favourite hypothetical questions regarding this used to be:
> If I'm running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the "regurgitation" problem and assured me that they take great pains to prevent that... but wouldn't describe how.)
> If I brainstorm with ChatGPT about potential new directions for my company, what's the chance that information might be exposed to a competitor in six months' time who asks "what might company X plan to do next"?
> My new preferred hypothetical for this is:
> If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?
They wouldn't appear in weights but could be added to the context. My conversations regularly go "regarding your Java problem"... which was a separate item in the history from earlier. As long as I only see these (and nobody else sees mine), it can be helpful.
fwiw I found Astra to be faster than Sol w/ both on medium reasoning for simple agentic coding
difficult to compare though because for more open ended, complex tasks Sol might miss something that Astra notices and then Sol might yield a cheaper but worse outcome
Same experience here, but I have some strange feeling.
Sol I'm used to working a month ago doesn't feel the same I'm using today, slower and less accurate. My gut feeling is that they quantize previous models to prioritize new ones and, who nows, make the new one look better.
Up to July I was using mostly anthropic models and the feeling was the same, so much so that I was able to predict every model release 1 or 2 days before public announcements.
It seems to use less than half the tokens for the same task compared to sol, and in some benchmarks closer to 2/3 less tokens. So the actual cost may be roughly the same or cheaper overall.
Sol is already brutal (even after their recent fixes, it's just a token-hungry model: I go through a full 20x account per day, on Sol Med/High standard speed, with ~2 threads). I hope the efficiency gains are true, since their token efficiency claims for Sol were bullshit.
How do you manage to run out of tokens so quickly? I probably run more threads every working day, usually on medium, and I'm still below the 5x limits.
Do you use the official harness? OpenAI's models are generally best in class for token efficiency. It seems to me like they push for that much more than their competitors.
I've long speculated this when I see these types of comments, because it's actually really difficult to hit usage caps with an efficient dev flow, even when running multiple threads for hours every day.
I think some combination of:
1) Using 1 thread for everything
2) Reviving old threads which are no longer in cache
3) Really broad prompts on badly vibecoded codebases, so model spends huge amount of time tracking down whatever you're trying to do.
4) Non-coding workflow which is more output than input heavy
5) (Less likely IMO) Intelligent use of many passive CI/cron-like scans. E.g. regular security, quality etc scans. Automated issue resolution/PR
Just a guess. I think 3 is likely the primary reason.
You can literally go all day every day with multiple threads with Sol on the Codex 100/month plan IME
That's my experience too. I've found OpenAI really quite generous with tokens. I sometimes wonder how some people manage to run out of them really. Do they just type prompts that much faster than me or use the highest reasoning mode for everything just because they can? Idk.
I generally agree with those reasons, although using a single thread may be less of an issue than it seems because of context compacting which should happen automatically when you're near the limit.
My use cases are iterative and sometimes require reading a lot of code or reevaluating work.
Token efficiency is near meaningless when the workload is input-heavy. It can't always just choose to read less, depending on the task.
I can have cheaper agents do the reading but it's not appropriate for all use cases because they'll misjudge and choose the wrong things to emphasize, summarize, extract for the bigger model.
The problem is also that at a lot of code bases are not designed around LLMs and their token usage.
If you have a monolete codebase, you need to clearly define in the prompt what modules are involved. And even then, your wasting tokens with the first 1 or 2 steps where it needs to located the modules.
if you have a github codebase, with a lot of your code into nice little repos, its even worse because then the model needs to pull data, and a ton of more steps.
Most people do not open their coding agents in the module directories because "it may need something out of it".
I mean, we used to program by creating utils directories to deal with repetitive code but models (a) find it and use it (but it cost steps and tokens to read), (b) do not realize it there and make their own version of whatever or (c) combination of both.
And ironically, i feel like we need to give up on this idea of reusable code, and literally keep things into single modules, with as minimum external dependencies. That in return reduces searches and thus steps/tokens burn. But very few agents / harnesses have proper implementation of groups/projects and sub-module structures. Aka they only open a single dir, so your then forced to create dozens of tabs > per dir > cli ...
2. That is also a issue if you open multiple agents. Maybe now your working in A, B but C, D, E are not doing anything. And their cache expires... Now you go back to D because A, B needed to be done, and now your paying Cache Write + Input cost.
Its hard to have a good flow to keep things cached, when to really /new and when to not have it expire (and that assumes there are no issue with the provider moving your session around and forcing new cache hits. MiMo did that a lot in the past).
Something that i also advice more and more to people. Get a microphone, download openwisper and talk (text to prompt). You tend to give more information vocally, then writing as its in our habit of programming to not be verbose. Its like people are afraid of long prompts. While just talking to the LLM tend to give it much more information to work with, often resulting it being able to skip steps.
The guy said medium/high regular speed so that's why I'm very puzzled! Ultra + Fast will absolutely slurp up your whole usage quickly but I've never found it gives substantially better results so I stick to extra high.
Sub-agents. I have 7 20x accounts and I burn them within 1-2 days if I go fully parallel. In some scenarios I use 50 sub-agents for a session which is literally hours of usage for a single 20x account. I'm at the point where I need to parallelize over multiple machines because I just don't have enough CPU and RAM.
Decompilation of a game and another larger decompile project. I'm working on it solo. I use 50 sub-agent, one per target function or translation unit. Often there is some progress in a unit but it's not done. So it requires a lot of cycles per function. Notably a single ~80kb function took about a week of constant sol-ultra attention before reaching exactness. The game I'm targeting has ~5000 total functions. The other decompile project has ~10k+ functions.
I'm sure I could be more token efficient, but this was/is also a learning process for me since I never did such an extremely large project before that would take multiple man years before AI.
Fascinating! I think that’s the main difference is my usage is probably tool-bound, meaning it writes some code but then there’s a long period of verification where it compiles things and then waits for the compilation and CI to complete before it can continue. That probably doesn’t consume as many tokens as constantly churning on a problem despite the same wall time.
Yes, this is why I mentioned having so many parallel agents and being compute bound. I run on my own laptop and 2 high-end desktop machines all with 64gb RAM. And it still occasionally happens that one OOM kills codex. They also mostly run unattended until I need to switch their accounts because a usage limit has been hit. Each instance usually can keep going when I sleep or do other things.
I only save the last 30% of usage on a single account for most of my other work, and that is almost always enough.
Are you not better off with cheaper models like GLM 5.3 or GLM 5.3 Flash? As a lot of that work is repetitive and only needs a stronger model at later stages of cleanup, no?
Of course I don't run everything on Sol. But weaker models can only do very basic things. I have a round robin style workflow where each unit first gets 10 cycles of each luna medium -> high -> terra medium -> terra high -> Sol Medium -> Sol high -> Sol Ultra. And you can clearly see that luna and terra only get a small amount of real progress.
Ugh. certifi the most annoying thing ever. It snuck into numerous python dependencies, never made anything better, and confuses all the junior devs when their venvs/containers etc can't access internal CA signed resources. Probably have explained what to do over 9,000 times. And for what? "To provide updated root CAs". Meh. I don't need daily CA trust store updates. If your CA roots are that new, I want nothing to do with it. Cert issuers (cough sectigo and their amateur hour CA root rotation recently, not even Mozilla had it).
> If I'm running Codex and one of my API keys accidentally gets consumed in the context, what are the chances that someone else might ask for an API key in the future and get mine back? (I asked someone at OpenAI once and they called this the "regurgitation" problem and assured me that they take great pains to prevent that... but wouldn't describe how.)
> If I brainstorm with ChatGPT about potential new directions for my company, what's the chance that information might be exposed to a competitor in six months' time who asks "what might company X plan to do next"?
> My new preferred hypothetical for this is:
> If I use ChatGPT to help me partially solve a Millennium Prize problem, what are the chances that my work will influence training such that a later model helps someone else solve it first?
reply