Yes, any non E2EE cloud storage system has strict scanning for CSAM. And it's based on perceptual hashes, not AI (because AI systems can be tricked with normal-looking adversarial images pretty easily)
I built a similar photo ID system, not for this purpose or content, and the idea of platforms using perceptual hashes to potentially ruin people's lives is horrifying.
Depending on the algorithm and parameters, you can easily get a scary amount of false positives, especially using algorithms that shrink images during hashing, which is a lot of them.
Yeah, it’s not a great system due to the fact that perceptual hashes can and have been tricked in the past. It is better than machine learning though because you can make any image trigger an ML model without necessarily looking like a bad image. That is, perceptual hashes are much harder to adversarially fool.
I agree, and maybe I'm wrong, but I see a similarity between phash quantization and DCT and ML kernels. I think you could craft "invisible" adversarial images similarly for phash systems like you can ML ones and the results could be just as bad. They'd probably replicate better than adversarial ML images, too.
I think the premise for either system is flawed and both are too error prone for critical applications.
I imagine you'd add more heuristics and various types of hashes? If the file is just sitting there, rarely accessed and unshared, or if the file only triggers on 2/10 hashes, it's probably a false alarm. If the file is on a public share, you can probably run an actual image comparison...
A lot of classic perceptual hash algorithms do "squinty" comparisons, where if an image kind of looks like one you've hashed against, you can get false positives.
I'd imagine outside of egregious abuse and truly unique images, you could squint at a legal image and say it looks very much like another illegal image, and get a false positive.
From what I'm reading about PhotoDNA, it's your standard phashing system from 15 years ago, which is terrifying.
But yes, you can add heuristics, but you will still get false positives.
I thought Apple’s approach was very promising. Unfortunately, instead of reading about how it actually worked, huge amounts of people just guessed incorrectly about how it worked and the conversation was dominated by uninformed outrage about things that weren’t happening.
> Unfortunately, instead of reading about how it actually worked, huge amounts of people just guessed incorrectly about how it worked
Folks did read. They guessed that known hashes would be stored on devices and images would be scanned against that. Was this a wrong guess?
> the conversation was dominated by uninformed outrage about things that weren’t happening.
The thing that wasn't happening yet was mission creep beyond the original targets. Because expanding-beyond-originally-stated-parameters is thing that happens with far reaching monitoring systems. Because it happens with the type of regularity that is typically limited to physics.
There were 2ndary concerns about how false positives would be handled. There were concerns about what the procedures were for any positive. Given Gov propensities to ruin lives now and ignore that harm (or craft a justification) later, the concerns seem valid.
That's what I recall the concerned voices were on about. To me, they didn't seem outraged.
> Folks did read. They guessed that known hashes would be stored on devices and images would be scanned against that. Was this a wrong guess?
Yes. Completely wrong. Not even close.
Why don’t you just go and read about it instead of guessing? Seriously, the point of my comment was that discussion with people who are just guessing is worthless.
> Why don't you just explain what you want people to know instead of making everyone else guess what you are thinking?
I’m not making people guess. I explained directly what I wanted people to know very, very plainly.
You are replying now as if the discussion we are having is whether it’s a good system or not. That is not the discussion we are having.
This is the point I was making:
> instead of reading about how it actually worked, huge amounts of people just guessed incorrectly about how it worked and the conversation was dominated by uninformed outrage about things that weren’t happening.
The discussion is about the ignorance, not about the system itself. If you knew how it worked and disagreed with it, then I would completely support that. I’m not 100% convinced myself! But you don’t know how it works, you just assumed – and you got it very wrong. So did a lot of other people. And collectively, that drowned out any discussion of how it actually worked, because you were all mad about something imaginary.
You are perfectly capable of reading how it worked. You do not need me to waste a lot of time re-writing Apple’s materials on a complex system in this small text box on Hacker News so you can then post a one sentence shallow dismissal. There is no value in doing that at all, it just places an asymmetric burden on me to continue the conversation.
The actual system is that they used a relatively complex zero-knowledge set-matching algorithm to calculate whether an image was a match without downloading or storing the set of hashes locally.
That said, I think this is mostly immaterial to the problem? As the comment you’re responding to says, the main problem they have with the system is mission creep, that governments will expand the system to cover more types of photos, etc. since the software is already present to scan through people’s photos on device. Which could happen regardless of how fancy the matching algorithm was.
Among many many issues: Apple used neural networks to compare images, which made the system very exploitable. You could send someone an image where you invisibly altered the image to trip the filter, but the image itself looked unchanged.
Also, once the system is created it’s easy to envision governments putting whatever images they want to know people have into the phone or changing the specificity of the filter so it starts sending many more images to the cloud. Especially since the filter ran on locally stored images and not things that were already in the cloud.
Their nudity filter on iMessages was fine though (I don’t think it ever sends anything to the internet? Just contacts your parents if you’re a minor with Family Sharing enabled?)
> once the system is created it’s easy to envision governments putting whatever images they want to know people have into the phone
A key point is that the system was designed to make sure the database was strongly cryptographically private against review. -- that's actually where 95% of the technical complexity in the proposal came from: to make absolutely sure the public could never discover exactly what government organizations were or weren't scanning for.
Sorry, but you're relaying a false memory. Conversation on the subject on HN and Reddit (for example) was extremely well informed and grounded in the specifics of the proposal.
Just as an example, part of my responses here were to develop and publish a second-preimage attack on their hash function-- simply to make the point concrete that varrious bad scenarios would be facilitated by the existence of one.
> instead of reading about how it actually worked, huge amounts of people just guessed incorrectly about how it worked and the conversation was dominated by uninformed outrage
I would not care if it worked 100% accurately. My outrage is informed by people like you who think it is OK in any form whatever.
No amount of my device spying on me is acceptable, no matter how cleverly implemented. The fact that your comment said anything positive about it at all without acknowledging that it is an insane idea and should never be put into practice is what I was referring to.
I read the whitepaper they published and worked at Apple at the time this idea was rightly pulled. I understand it perfectly fine and stand by my words.
