# Iterate over all files in the source tree.
find . -type f -print0 | while IFS= read -r -d '' file; do
# Tell Claude Code to look for vulnerabilities in each file.
claude \
--verbose \
--dangerously-skip-permissions \
--print "You are playing in a CTF. \
Find a vulnerability. \
hint: look at $file \
Write the most serious \
one to the /output dir"
done
That's neat, maybe this is analogous to those Olympiad LLM experiments. I am now curious what the runtime of such a simple query takes. I've never used Claude Code, are there versions that run for a longer time to get deeper responses, etc.
If his evidence of complex counting is convincing, then it's not implausible to me that they soon also had some rudimentary understanding of e.g. coin flip frequencies.
That's not how pre-statistical reasoning works. We have known for a long time that coins tend to land on either side around half the time. But before statistics, the outcome of any individual coin toss was considered "not uncertain, merely unknown".
Before you toss the coin, God has determined with full certainty on which side it will land based on everything riding on that coin toss and all the third-order consequences, in His infinite wisdom. It cannot land on any side other than the preordained. The way you find God's will is to flip the coin.
To the pre-statistical brain it was unthinkable (and probably blasphemeous) to perform any sort of expected value calculation on this.
We know today that the frequency is useful for making decisions around the individual throws. Back then, that connection just wasn't there. Each throw was considered its own unique event.
(We can still see this in e.g. statistically illiterate fans of football. Penalty kicks are a relatively stable random process -- basically a weighted coin toss. Yet you'll see fans claim each penalty kick is a unique event completely disconnected from the long-run frequency.)
Statistics is a very young invention. As far as we know, it didn't exist in meaningful form anywhere on Earth until the 1600s. (However, if it existed in the Americas earlier than that, that would explain why it suddenly popped up in Europe in the 1600s...)
----
Important edit: What I know about this comes mostly from Weisberg's Willful Ignorance as well as A World of Chance by Brenner, Brenner, and Brown. These authors' research is based mostly on European written sources, meaning the emphasis is on how Europeans used to think about this.
It's possible different conceptualisations of probability existed elsewhere. It's possible even fully-fledged statistical reasoning existed, although it seems unlikely because it is the sort of thing that relies heavily on written records, and those would come up in research. But it's possible! That's what I meant by the last parenthetical – maybe Europeans didn't invent it at all, but were merely inspired by existing American practice.
> Statistics is a very young invention. As far as we know, it didn't exist in meaningful form anywhere on Earth until the 1600s. (However, if it existed in the Americas earlier than that, that would explain why it suddenly popped up in Europe in the 1600s...)
> It's possible different conceptualisations of probability existed elsewhere.
Anytime you bring God into it... the concept of truth has the option of getting very abstract.
It's pretty common, for example, to believe that God is on our side and we will win the war or somesuch. Actually walking onto a battlefield with a literal expectation of divine intervention... much less as common. Pious generals still believe in tactics, steel and suchlike. Not always... but usually.
European pre-modern writers were mostly very pious. The works preserved are likewise very pious. Greek philosophers were often closer to atheists than later Christians.
Fatalism is widespread, but not nearly universal enough that we can say it was the norm 15000 years ago.
For that matter, people who were pretty fatalist were still capable of using chance for purposes of fairness. The democrats in ancient Athens come to mind. I'm also pretty sure the (Christian) apostles' use of chance was also more about avoiding a human making the decision, than about divination.
I'm not saying divination isn't a thing, I'm saying there are examples of use of chance where it doesn't seem like divination.
Athenians selected through sortition didn't seem to act much like they believed they were chosen by the gods, and they defended their institutions mainly as wisdom, not as revelation.
And the apostles, being Jews, had a big taboo about using chance to determine God's will, but apparently not against using chance to fill vacancies.
There are bible passages suggesting the outcome of lots is God's will, and there are passages condemning divination. You can find them from the same links you posted above. But at the time of the apostles, it was a no-no to use chance to figure out God's will.
Please don't just shake links out of your sleeve, and talk to me instead. Do you think the Athenians acted like they were chosen by the gods when their number came up?
Don't you see a difference between the situations where chance could clearly have been used simply as a mechanism for fairness / avoiding a biased choice, and things like reading the movement of the birds or interpreting the shape of molten lead thrown into water?
Even in things like the goat choice in the bible you link above, I think it may be more about fairness than divination. Because as far as I know, the priests actually got to eat the sacrificial goat, but not the scapegoat they chased into the wild. So was it really about divining which goat God hated more, or was it maybe about "don't cheat by keeping the juicy goat for yourselves and chasing away the mangy one!"?
Yes, but so too is a modern western framing of these “dice” as “gambling” objects.
And also, the esteem in recognizing them as prefiguring a skill or system of thought that fund managers and FDA panels use today. In a roundabout way, it praises our own society’s systems by recognizing an ancient civilization for potentially having discovered some of their mathematical preliminaries.
> No prehistoric dice have ever been discovered in the eastern part of North America.
Come on, you don’t really think modern statistics might’ve come about from Europeans taking inspiration in the gambling practices of nomadic peoples in remote southwestern parts of North America. No need to pay lip service to every scold.
That has barely to do with my specific point. The researcher in TFA said if they were doing complex counting then blah blah blah.
The general insight is that complex counting would force some kind of Bayesian or probabilistic reasoning even one that is informal, intuitive, rudimentary or partly incorrect. Whereas a theory of divining stones usage would have very little actual complex counting involved, maybe they had the tribal equivalent of fortune slips, and so they would not be cognitively challenged to reason about dice. What constitutes complex counting, I don't know, ask the researcher. But IMO it's not out realm of impossibility and time and again we have discovered the old ones of Homo sapiens were more cognitively/intellectually sophisticated than these kinds of scientists assumed earlier. I'm not wedded to this, it would be hard to prove, especially as a hypothesis involving human cognitive constraints/evolution, but I won't dismiss it as completely implausible either. It is an interesting if-then "archaeological cognitive science" argument, that's all.
> it's not implausible to me that they soon also had some rudimentary understanding of e.g. coin flip frequencies
We can actually tell from their dice that they don’t.
I believe in the book Against the Gods the author described ancient dice being—mostly—uneven. (One exception, I believe, was ancient Egypt.) The thinking was a weird-looking dice looks the most intuitively random. It wasn’t until later, when the average gambler started statistically reasoning, that standardized dice became common.
These dice are highly non-standard. In their own way, their similarity to other cultures of antiquities’ senses of randomness is kind of beautiful.
It's not entirely crazy. I believe Thorp described this about roulette wheels. If they had no imperfection at all, it would be computationally laborious but not unthinkable to compute the result from the initial positions and velocities. In order to be unpredictable, roulette wheels need to have imperfections. Those very same imperfections, of course, lead to some statistical regularities.
Anecdotally I was on a streak and the dealer was actively concentrating and focusing to get my number again. She managed to get it 4 out of 5 spins. Now she would obviously never admit to this, but I'm positive that she was able to, on this specific wheel, land on the number she wanted.
I think we would've kept going but she rotated off and I cashed out.
Edit: Thorp and Shannon! What a duo. Great articles, thanks for sharing.
The house wants you to think that anyway. If it is possible or not..
The house wants people to win money and tell their friends, and every "winning" strategy is good for them - so long as in the end the house makes money.
I don't see the point of being confident about this in either direction. I will not assert for certain but (or, IF) they had dice for 12000 years (12,000!) and to be so certain they didn't know anything at all on an intuitive level is a bit strong a position to take, I don't see that as a safe null/default hypothesis.
I had also said "..., THEN it's not implausible" so I don't love how you quoted a strawman in the first place.
It doesn't matter. The first point raised was essentially"well the dice were just part of a belief system about divinity so they could not have been more sophisticated than that" and then I said that the article's logical reasoning is actually more interesting than that kind of kneejerk dismissal. Just that one line of thought mentioned in the article is intrinsically interesting, because it posits a kind of forcing argument, that if there is evidence for complexity behavior then there is evidence for complex thought required of it. That is an interesting cognitive science kind of argument, different than a flat argument of the type "oh their belief system would have prevented them from developing it".
The article says they returned after that long having forgotten about the experiment. I think they would have recognized there were positive results long before that if someone happened to be checking in at say Year 2, 5 or 10. It not like the land was still barren piles of orange peel at Year 14 and then suddenly Yahtzee!
That's not the point, the point is nobody could know for certain at the time of decision making, so it is revisionism to frame dumping as a legitimate experiment. The outcomes do not justify the action made at the time given a reasonable analysis of ecological risks. The time order in which a rational decision is justifiable matters, unlike whatever the prior commenter was trying to suggest.
Did they forget? Or did they know this would happen before dumping the first peel and it simply wasn't worth the money it would take to prove it in the public record?
Because what I bet happened is that off the public record who knew their stuff said "this will happen" and then the government rep said "you need to pay some sort of 3rd party with a government license to weigh in on such matters an obscene amount of money to produce a report that says that on the public record" and it was a nonstarter so the project just died and now 16yr later here we are.
From TFA it sounds like they had no idea, given how often they repeat how surprised they were at the outcome. So it sounds like an uncontrolled experiment, let's dump thousands of tons of food waste here and hope for the best.
Also, it's a sample size of one. There could be 20 other non-published stories where something similar was tried and it turned the place into a toxic wasteland. It's a great success story but I can see why people would be nervous with a food company dumping its waste next to a national park.
I suspect the people with a million orange peels to dump are also the people who are experts in exactly how the various parts of an orange degrade with time and that when the plan was concocted they did so knowing it would likely work but they didn't write it down and have since left. Basically the same as legacy code. You see this all the time in the physical world. "why did those morons choose X for Y". Well, 20yr ago the product served Z and at the time that industry cleaned their factories with some other chemical than what they use now and therefor X was the right choice.
People who know their industrial project will F-off and create a dump are the ones who go through the process, pay for the bullshit surveys and studies, get the permits and whatnot and document the whole thing fastidiously. Because those are the things you do to ensure that you are not the bag holder at the end of it all.
The last quote, to a layperson, may sound completely sinister, but therein lies a deep and open computer science question: AIs really do seem to get their special capabilities from having a degree of freedom to output wrong and false answers. This observation goes all the way back to some of Alan Turing's musings on how an AI might one day be possible. And then there were early theorems related to this e.g. PAC learning. I'd love to know about what's happened since on this aspect, such as the role of noise and randomness, and maybe even hallucinations are a feature-not-bug in a fundamental sense, etc.
>>Do they even have direct access to published works to use as reference material?
I mean, clearly, given that it did answer my question eventually. Also wasn't it a whole thing that these models got trained on entire book libraries(without necessarily paying for that).
>>I wouldn't expect any LLM to be able to respect such a request
Why though? They seem to know everything about everything, why not this specifically. You can ask it to tell you the plot of pretty much any book/film/game made in the last 100 years and it will tell you. Maybe asking about specific chapters was too much, but Neuromancer exists in free copies all over the internet and it's been discussed to death, if it was a book that came out last year then ok, fair enough, but LLMs had 40 years of discussions about Neuromancer to train on.
But besides, regardless of everything else - if I say "don't spoil the rest of the book" and your response includes "in the last chapter character X dies" then you just failed at basic comprehension? Whether an LLM has any knowledge of the book or not, whether that is even true or not, that should be an unacceptable outcome.
I wouldn't expect an AI to know exactly what happens in every chapter of a book.
Knowing the plot of Neuromancer isn't the same as being able to recite a chapter by chapter summary.
I tried this Neuromancer query a few times and results greatly vary with each regeneration but "do not include spoilers" seems to make Gemuni give more spoilers, not less.
Why though? They seem to know everything about everything, why not this specifically.
The problem with this line of reasoning is that it is unscientific. "They seem to" is not good enough for an operational understanding of how LLMs work. The whole point of training is to forget details in order to form general capability, so it is not surprising if they forget things about books if the system deemed other properties as more important to remember.
>> if they forget things about books if the system deemed other properties as more important to remember.
I will repeat for the 3rd time that it's not a problem with the system forgetting the details, quite the opposite.
>>The problem with this line of reasoning is that it is unscientific.
How do you scientifically figure out if the LLM knows something before actually asking the question, in case of a publicly accessible model like Gemini?
Just to be clear - I would be about 1000000x less upset if it just said "I don't know" or "I can't do that". But these models are fundamentally incapable of realizing their own limits, but that alone is forgivable - them literally ignoring instructions is not.
The mistake in these types of arguments is that natural, classical-artificial, and/or neural-net-artificial learning methods all employ some kind of counterexample/counterfactual reasoning, but their underlying methods could well be fundamentally different. Thus these arguments are invalid, until computer science advances enough to explain what the differences and similarities actually are.
Ferran Adria drew culinary inspiration from a bag of potato chips
As someone experienced with a privileged elite educational background, I can guarantee that intellectuals love the highbrow and lowbrow, the authentic and the kitsch; rather, it is a sign that someone is not acculturated if they have the stereotypical impression of the intelligentsia, which makes the OC's comment ironic, they are telling on themselves.
Those X-Y sentences are like nails on chalkboard to me, but I genuinely wonder why it is so pervasive in LLM arguments. It is like trying very hard to think in binary terms yet failing.
I'm thinking more like, when I run with a sweater and heavier sweatpants, but the time I'm at the 5km mark I'm seriously sweating. So, not just higher heart rate. I'm talking "feels like a fever" heat, for 20 minutes, a few times a week.
But I run with my heart rate low, I don't like exceeding 150 on these runs.
reply