I’d speculate that this was a product of its time (early Heroku days), and that a goal at the time was to get secrets out of source control. Which was an antipattern way back then.
Times have changed since then, and there’s much better tooling available to help with this problem space and surface area these days.
It's not just what they output, it's also what they omit. I've unfortunately seen this when doing research on medical issues, and it gives you a false sense of having an understanding. It never warned me to watch out for symptoms that weren't specifically discussed in the context, or that some symptoms could be a different underlying condition.
Yes, when you use a standard theoretical physics textbook / course sequence, it is 100% correct.
More broadly, LLMs or anything at all, needs a verifier. If the task can be automatically verified, great, then anyone can use them. If you can't automate the verification, then you need to be able to verify it using your knowledge. Knowledge required to verify is lower than the knowledge required to create in very few cases. This is why you still need a fully trained human verifier. We are yet to reorganise the overall "tasks" in the economy such that verification can be done with much lesser knowledge, for no reason other than that there was never demand for this until creation became automatic few years ago. It is possible and is slowly being done, there are many many startups working on automating verification in different fields and in many cases we will see fields reorganise themselves to be more amenable to automatic verification. Note that _effort_ required to verify is much lower than what is required to create, for almost anybody, and LLMs have economic use just due to that alone, albeit in the hands of a knowledgeable human.
Math professors are remarkably good at answering math questions or solving ad hoc math puzzles correctly. Also at stepping back and asking, why are you trying to show this result, what do you think it has to do with the thing we are trying to prove. Much better at knowing when the student has gone off the path in some subtle way than the LLM (without a Prof Tao in the chat).
Professor Tao also put out some YouTubes of him working with an older LLM to do Lean proofs, and his intelligence matters - things where I would be stuck for hours trying to understand what was failing in the model proof were just instantly clear to him and fixed in thirty seconds.
And I am an ok coder (rather than a bad maths grad student), but the LLM will happily thrash around the edges of a problem with me with no clear convergence when I don’t have that clear insight and the problem is weirdly presented enough; I still find the trick to walk around the block and disengage and then return knowing exactly what to do (now prompting it to the right thing) to be a super power for getting what I want out of the coding system.
Both fable and sol are confidently wrong all the time in their most cherished domain - software engineering - I can’t quote anything because they’re working on my employer’s code bases. They’re much less wrong than their predecessors and they’re also quite good at point out their mistakes, but they’re still wrong a lot.
they are wrong on the specific nuance of my codebase too, i am talking about learning something - i can still learn everything about software engineering talking to a bot.
I asked Opus 4.8 to critique my algebra notes (these are definitely not masters level- just undergrad second year). It hallucinated an error it claimed I made in the notes and then put in a correction I didn't need because what I had written was correct.
What I said in my notes was:
Notice that a cyclic group is a degenerate (in the sense of "smallest
non-trivial") case of a finitely generated group where the generating set is
a singleton.
It left-off the "non-trivial" and said that what I said was this was the smallest case of a finitely-generated group which is incorrect because it excludes the trivial group.
The point is I see the LLMs as a "smart friend"/colleague I can work with but I do think critically about what I get told and don't just take it as face value because it's not always correct for sure even in relatively basic cases like this.
This deduction is baseless, there is no reason to think an LLM will only hallucinate on complex topics. They still get the "number of es in seventeen" question wrong regularly. The kind of mistakes an LLM makes has no clear resemblance to the kind of mistakes a human makes, because they are not doing the same thing.
And for the harder maths you want something not just to explain the answer but to diagnose your conceptual error behind your questions. Asking the teacher after class or tutorial type systems work efficiently for a reason.
One area that LLMs tend to do badly in still is sailing. I sail casually and I have friends who are instructors, basically all the LLMs we tried gave really unsafe advice for a particular manoeuvre. Claude was the notable exception getting it mostly right, but even so I wouldn't rely on it there.
> This looks a lot like when the U.S. & Israel were running dry on interceptors last year, and made a deal with the Iranians to pretend to blow up Fordow in exchange for ceasing hostilities.
Do you have a citation for this? (Genuinely curious)
The author’s claims that “there was never any credible evidence that Iranian medium- and long-range air defenses against fixed-wing aircraft were attrited to any significant degree“ and that the B-2 is easily to track and target… seem rather questionable. Unfounded even.
> the silence of the Iranian government on this point suggests something coordinated to me
Or, you know, a theocratic autocracy that realizes the perception of its military power is critical to keeping the populace in line?
And admitting to an inability to stop military action on one of their most heavily defended targets by the very enemy they've whipped their supporters into a froth over is a bad look?
In your zeal you seem to have misunderstood the point I was making. The Iranian government did not dispute American claims of having flown over and destroyed Fordow, which is the opposite of the "they'll say anything to look strong" thesis you're advancing.
I think the U.S. whipped the Iranian government's supporters into line by treacherously using peace negotiations as a pretext to target their scientists, helping Israel launch a completely unjustified sneak attack with all the Mossad rats they had hidden in the country, and facilitating the genocide in Gaza. With enemies like that the Iranian government hardly needs help.
When we first created Domainr (then domai.nr, now domainr.com) back in 2008, we needed a list of “zones under which domain registrations were somehow possible.” E.g. not just the root zone list from IANA, but all the .co., .edu., .net., etc. variants. We found what we could from Wikipedia, and used Mturk to find the rest from registry websites, etc.
It wasn’t perfect, but it didn’t need to be. We essentially needed a “good enough to start with” dataset that we could refine going forward. It got the job done.
This makes Ruby Central look even worse. TFA is only concerned with the root user, and the timeline ends at September 30, but Arko was able to confirm as late as October 5 that he had access to _other_ accounts with production access. Ruby Central doesn't seem interested in the article to mention that even after being notified about unauthorized access they still hadn't rotated all relevant credentials almost a week later.
Welp, now that there is confirmation that lawyers are involved, the chances there will be any of sort of open and transparent reconciliation process have plummeted.
“Following these budget adjustments, Mr. Arko’s consultancy, which had been receiving approximately $50,000 per year for providing the secondary on-call service, submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII).”
Times have changed since then, and there’s much better tooling available to help with this problem space and surface area these days.
reply