The repo could lead to 404 to everyone, but if someone had starred it they would get a 403. They would be able to see all their stars but not access certain repos because they were made private. Probably less surprising then some of their stars disappearing. They would at least know what happened and it’s not like wouldn’t know the repo was public at some point anyway. Additionally, there could be another button to clear stars and watchers, if the owner of the repo really wants to.