Recently I've used Claude Code to perform some entry to mid level web-based CTF hunting in a fully autonomous mode (--allow-dangerously-skip-permissions in an isolated environment). It excels at low hanging fruit - XSS, other injections, IDOR, hidden form fields, session fixation, careful enumeration, etc.
Though I'm still pissed at Kagi about their collaboration with Yandex, this particular kind of fight against AI slop has always striked me as a bit of Don Quixote vs windmill.
AI slop eventually will get as good as your average blogger. Even now if you put an effort into prompting and context building, you can achieve 100% human like results.
I am terrified of AI generated content taking over and consuming search engines. But this tagging is more a fight against bad writing [by/with AI]. This is not solving the problem.
Yes, now it's possible somehow to distinguish AI slop from normal writing often times by just looking at it, but I am sure that there is a lot of content which is generated by AI but indistinguishable from one written by mere human.
Aso - are we 100% sure that we're not indirectly helping AI and people using it to slopify internet by helping them understand what is actually good slop and what is bad? :)
> AI slop eventually will get as good as your average blogger. Even now if you put an effort into prompting and context building, you can achieve 100% human like results.
Hey, Kagi ML lead here.
For images/videos/sound, not at the current moment, diffusion and GANs leave visible artifacts. There's a bit of issues with edge cases like high resolution images that have been JPEG compressed to hell, but even with those the framing of AI images tends to be pretty consistent.
For human slop there's a bunch of detection methods that bypass human checks:
1. Within the category of "slop" the vast mass of it is low effort. The majority of text slop is default-settings chatGPT, which has a particular and recognizable wording and style.
2.Checking the source of the content instead of the content itself is generally a better signal.
For instance, is the author posting inhumanly often all of a sudden?
Are they using particular wordpress page setups and plugins that are common with SEO spammers?
What about inboud/outbound links to that page -- are they linked to by humans at all?
Are they a random, new page doing a bunch of product reviews all of a sudden with amazon affiliate links?
Aggregating a bunch of partial signals like this is much better than just scoring the text itself on the LLM perplexity score, which is obviously not a robust strategy.
> Are they using particular wordpress page setups and plugins that are common with SEO spammers?
Why doesn't Kagi go after these signals instead? Then you could easily catch a double digit percentage of slop and maybe over half of slop (AI generated or not), without having to do crowd sourcing and other complicated setups. It's right there in the code. The same with emojis in YouTube video titles.
The current search engine doesn't go after WordPress plugins we consider correlated to bad pages.
By far the most efficient method in the search engine for spam is downranking by trackers/javascript weight/etc.
Slopstop is going after page formats but we didn't plan to scale that back to rankings for everyone quite yet, only use it as features to detect AI slop. Otherwise the collateral damage on good actors with bad websites would be risky early on.
What I was meaning with "are you certain" is regarding how Kagi treats the spam signals from WordPress plugins and themes. And now you gave the answer, thanks for that! I believe you will have good returns in using those signals.
If you're concerned about money ending up at companies that are taxed by countries that mass murder people, you should be as pissed about Google, Microsoft, DuckDuckGo, Boeing, Airbus, Walmart, Nvidia, etc... there is almost no company you should not be pissed off by.
I would be happy that Google is getting some competition. It seems Yandex created a search engine that actually works, at least in some scenarios. It's known to be significantly less censored than Google, unless the Russian government cares about the topic you're searching for (which is why Kagi will never use it exclusively).
> Are we personally comfortable with such an approach?
I am not, because it's anti-human. I am a human and therefore I care about the human perspective on things. I don't care if a robot is 100x better than a human at any task; I don't want to read its output.
Same reason I'd rather watch a human grandmaster play chess than Stockfish.
There are umpteenth such analogies. Watching the world's strongest man lift a heavy thing is interesting. Watching an average crane lift something 100x heavier is not.
Objectively we should care because the content is not the whole value proposition of a blog post. The authenticity and trust of validity of the content comes from your connection to the human that made it.
I don't need to fact check a ride review from an author I trust, if they actually ride mountain bikes. An AI article about mountain bikes lacks that implicit trust and authenticity. The AI has never ridden a bike before.
Though that reminds me if an interaction with Claude AI, I was at the edge of its knowledge with a problem and I could tell because I had found the exact forum post it quoted. I asked if this command could brick my motherboard, and it said "It's worked on all the MSI boards I have tried it on." So I didn't run the command, mate you've never left your GPU world you definitely don't actually have that experience to back that claim.
“It's worked on all the MSI boards I have tried it on.”
I love when they do that. It’s like a glitch in the matrix. It snaps you out of the illusion that these things are more than just a highly compressed form of internet text.
We have many expectations in society which often aren't formalized into a stated commitment. Is it really unreasonable to have some commitment towards society to these less formally stated expectations? And is expecting communication presented as being human to human to actually be from a human unreasonable for such an expectation? I think not.
If you were to find out that the people replying to you were actually bots designed to keep you busy and engaged, feeling a bit betrayed by that seems entirely expected. Even though at no point did those people commit to you that they weren't bots.
Letting someone know they are engaging with a bot seems like basic respect, and I think society benefits from having such a level of basic respect for each other.
It is a bit like the spouse who says "well I never made a specific commitment that I would be the one picking the gift". I wouldn't like a society where the only commitments are those we formally agree to.
I do appreciate this side of the argument but.. do you think that the level/strength of a marriage commitment is worthy of comparison to walking by someone in public / riding the same subway as them randomly / visiting their blog?
That is how illustrations work. If someone doesn't see something, you amplify it until it clubs them over the head and even an idiot can see it.
And sometimes of course even that doesn't work but there has always been and always will be the clued, clue-resistant, and the clue-proof. Can't do anything about the clue-proof but at least presenting the arguments allows everyone else to consider them.
This fixation on the reverence due a spouse is completely stupid and beside the point of the concept being expressed. As though you think there is some arbitrary rule about spouses that is the essense of the problem? The gift-for-spouse is an intentionally hyberbolic example of a concept that also exists and applies the same at non-hyperbolic levels.
The point of a clearer example is you recognize "oh yeah, that would be wrong" and so then the next step is to ask what makes it wrong? And why doesn't that apply the same back in the original context?
You apparently would say "because it's not my wife", but there is nothing magically different about needing to respect your spouses time vs anyone else's. It's not like there is some arbitrary rule that says you can't lie to a spouse simply because they are a spouse and those are the rules about spouses. You don't lie to a spouse because it's intrinsically wrong to lie at all to anyone. It's merely extra wrong to to do anything wrong to someone you supposedly claim to extra-care about. Lying was already wrong all by itself for reasons that don't have anything special to do with spouses.
This idea that it's fine to lie to and waste the time of everyone else, commandeer and harness their attention of an interaction with you, while you just let a robot do your part and you are off doing something more interesting with your own time and attention, to everyone else who isn't your spouse simply because you don't know them personally and have no reason to care about them is really pretty damning. The more you try to make this argument that you seem to think is so rational, the more empty inside you declare yourself to be.
I really can not understand how anyone can try to float the argument "What's so bad about being tricked if you can't tell you were tricked?" There are several words for the different facets of what's so wrong, such as "manipulation". All I can say is, I guess you'll just have to take it on faith that humans overwhemingly consider manipulation to be a bad thing. Read up on it. It's not just some strange idea I have.
I think we are having a fundamental disagreement about "being tricked" happening at all. I'm intelligent enough to follow the argument.
I see that, in the hyperbolic case, you are actively tricking your wife. I just don't agree that you are actively tricking randomly public visitors of a blog in any real way? there is no agreement in place such that you can "trick" them. Presumably you made commitments in your marriage. No commitments were made to the public when a blog got posted.
It's equally baffling to me that you would use one case to make the point of the other. It doesn't make any fucking sense.
Why was it wrong in the wife case? What specifically was wrong about it? Assume she never finds out and totally loves the gift. Is purely happy. (I guess part of this also depends on the answer to another question: What is she so happy about exactly?)
I find them comparable, but not equal, for that reason.
Especially if we consider the summation of these commitments. One is obviously much larger, but it defines just one of our relationships within society. The other defines the majority of our interactions within society at large, so a change to it, while much less impactful to any one single interaction or relationship (I use them interchangeably here as often the relationship is just that one single interaction) is magnified by how much more often it occurs. This does move towards making the costs of losing some trust in such a small interaction as having a much larger cost than it first appears, which I think further increases how one can compare them.
(More generally, I also like comparing things even when the scale doesn't match, as long as the comparison really applies. Like apples and oranges, both are fruits you can make juice or jam with.)
There are many discussions of what sets apart a high trust society from a low trust society, and how a high trust society enables greater cooperation and positive risk taking collectively. Also about how the United States is currently descending into a low trust society.
"Random blog can do whatever they want and it's wrong of you to criticize them for anything because you didn't make a mutual commitment" is low-trust society behavior. I, and others, want there to be a social contract that it is frowned upon to violate. This social contract involves not being dishonest.
We should care if it is lower in quality than something made by humans (e.g. less accurate, less insightful, less creative, etc.) but looks like human content. In that scenario, AI slop could easily flood out meaningful content.
I don't care one bit if the content is interesting, useful, and accurate.
The issue with AI slop isn't with how it's written. It's the fact that it's wrong, and that the author hasn't bothered to check it. If I read a post and find that it's nonsense I can guarantee that I won't be trusting that blog again. At some point there'll become a point where my belief in the accuracy of blogs in general is undermined to the point where I shift to only bothering with bloggers I already trust. That is when blogging dies, because new bloggers will find it impossible to find an audience (assuming people think as I do, which is a big assumption to be fair.)
AI has the power to completely undo all trust people have in content that's published online, and do even more damage than advertising, reviews, and spam have already done. Guarding against that is probably worthwhile.
Even if it's right there's also the factor of: why did you use a machine to make your writing longer just to waste my time? If the output is just as good as the input, but the input is shorter, why not show me the input.
> AI slop eventually will get as good as your average blogger. Even now if you put an effort into prompting and context building, you can achieve 100% human like results.
In that case, I don't think I consider it "AI slop"—it's "AI something else". If you think everything generated by AI is slop (I won't argue that point), you don't really need the "slop" descriptor.
Explicitly in the article, one of the headings is "AI slop is deceptive or low-value AI-generated content, created to manipulate ranking or attention rather than help the reader."
So yes, they are proposing marking bad AI content (from the user's perspective), not all AI-generated content.
How is this any different from a search engine choosing how to rank any other content, including penalizing SEO spam? I may not agree with all of their priorities, but I would welcome the search engine filtering out low quality, low effort spam for me.
Yes, that's why we'll publish a blog post on this subject in the coming weeks.
We've been working on this topic since the beginning of summer, and right now our focus is on exploring report patterns.
> AI slop eventually will get as good as your average blogger
At that point, the context changes. We're not there yet.
Once we reach that point––if we reach it––it's valuable to know who is repeating thoughts I can get for pennies from a language model and who is originally thinking.
T-shirt folding is nice et al, but its final size is ok only for normal-sized shirts. With this method XL and larger t-shirts don't fold into neat small things.
I suppose that Simon, being all in with LLMs for quite a while now, has developed a good intuition/feeling for framing questions so that they produce less hallucinations.
Yeah I think that's exactly right. I don't ask questions that are likely to product hallucinations (like citations from papers about a topic to an LLM without search access), so I rarely see them.
But how would you verify? Are you constantly asking questions you already know the answers to? In depth answers?
Often the hallucinations I see are subtle, though usually critical. I see it when generating code, doing my testing, or even just writing. There are hallucinations in today's announcements, such as the airfoil example[0]. An example of more obvious hallucinations is I was asking for help improving writing an abstract for a paper. I gave it my draft and it inserted new numbers and metrics that weren't there. I tried again providing my whole paper. I tried again making explicit to not add new numbers. I tried the whole process again in new sessions and in private sessions. Claude did better than GPT 4 and o3 but none would do it without follow-ups and a few iterations.
Honestly I'm curious what you use them for where you don't see hallucinations
[0] which is a subtle but famous misconception. One that you'll even see in textbooks. Hallucination probably caused by Bernoulli being in the prompt
When I'm using them for code these days it is usually in a tool that can execute code in a loop - so I don't tend to even spot the hallucinations because the model self corrects itself.
For factual information I only ever use search-enabled models like o3 or GPT-4.
Most of my other use cases involve pasting large volumes of text into the model and having it extract information or manipulates that text in some way.
I don't think this means no hallucinations (in output). I think it'd be naive to assume that compiling and passing tests means hallucination free.
> For factual information
I've used both quite a bit too. While o3 tends to be better, I see hallucinations frequently with both.
> Most of my other use cases
I guess my question is how you validate the hallucination free claim.
Maybe I'm misinterpreting your claim? You said "I rarely see them" but I'm assuming you mean more, and I think it would be reasonable for anyone to interpret this as more. Are you just making the claim that you don't see them or making a claim that they are uncommon? The latter is what I interpreted.
I don't understand why code passing tests wouldn't be protection against most forms of hallucinations. In code, a hallucination means an invented function or method that doesn't exist. A test that uses that function or method genuinely does prove that it exists.
It might be using it wrong but I'd qualify that as a bug or mistake, not a hallucination.
Is it likely we have different ideas of what "hallucination" means?
> tests wouldn't be protection against most forms of hallucinations.
Sorry, that's a stronger condition that I intended to communicate. I agree, tests are a good mitigation strategy. We use them for similar reasons. But I'm saying that passing tests is insufficient to conclude hallucination free.
My claim is more along the lines of "passing tests doesn't mean your code is bug free" which I think we can all agree on is a pretty mundane claim?
> Is it likely we have different ideas of what "hallucination" means?
I agree, I think that's where our divergence is. Which in that case let's continue over here[0] (linking if others are following). I'll add that I think we're going to run into the problem of what we consider to be in distribution, in which I'll state that I think coding is in distribution.
Haven't you effectively built a system to detect and remove those specific kind of hallucinations and repeat the process once detected before presenting it to you?
So you're not seeing hallucinations in the same way that Van Halen isn't seeing the brown M&Ms, because they've been removed, it's not that they never existed.
I think systems integrated with LLMs that help spot and eliminate hallucinations - like code execution loops and search tools - are effective tools for reducing the impact of hallucinations in how I use models.
That's part of what I was getting at when I very clumsily said that I rarely experience hallucinations from modern models.
On multiple occasions, Claude Code claims it completed a task when it actually just wrote mock code. It will also answer questions with certainity (for e.g. where is this value being passed), but in reality it is making it up. So if you haven't been seeing hallucinations on Opus/Sonnet, you probably aren't looking deep enough.
This is because you haven't given it a tool to verify the task is done.
TDD works pretty well, have it write even the most basic test (or go full artisanal and write it yourself) first and then ask it to implement the code.
I have a standing order in my main CLAUDE.md to "always run `task build` before claiming a task is done". All my projects use Task[0] with pretty standard structure where build always runs lint + test before building the project.
With a semi-robust test suite I can be pretty sure nothing major broke if `task build` completes without errors.
What do you think it is 'mocking'? It is exactly the behavior that would make the tests work. And unless I give it access to production, it has no way to verify tasks like how values (in this case secrets/envs) are being passed.
Plus, this is all besides the point. Simon argued that the model hallucinates less, not a specific product.
Yes. Though an easier to solve hallucination. That is, if you know what to look for, but that's kinda the problem. Truth is complex, lies are simple. More accurately, truth has infinite complexity and the big question is what's "good enough". The answer is a moving target.
I think if you ask o3 any math question which is beyond its ability it will say something incorrect with almost 100% probability somewhere in output. Similar to if you ask it to use literature to resolve some question which is not obvious it often hallucinates results not in paper.
This thread's responses are a classic whataboutism. Instead of addressing this particular issue, people address others. I support motion for Kagi to drop Yandex integration, because of the reasons cited.
When they first got mentioned, I started aggregating their links. Just now did a quick analysis to look if they've changed something. At the beginning Kite contained a lot of rt.com (Russian propaganda) and a few other discutable sources. rt.com has not surfaced since May 22, which is good. Hope they'll reconsider Kagi's Yandex integration as well.
reply