I think they examples you've listed are different, and more acceptable than pushing secrets and private keys to Github because you had dozens of them hard-coded.
As for the aside, I imagine (with no background knowledge here) that as "owner" if you accidentally published something within that licensed code that does not belong or isn't covered by that license, you should probably have the right to remove it
What if I accidentally contributed a little bit too much code to the Linux kernel? What if that code had been in the last 10 releases? What about the users and distributions running those versions?
I'm not sure the answer is, or should be, as simple as "derp - delete immediately, this was never meant to be Open Source".
That said, if the code was stolen, and then published under an OS licence - it's not OS. The "publisher" never had the rights to make it OS, so the "contract" is null and void...
This is what I was thinking. If it's something that wasn't meant to be published, I assume your right as the owner hold.
How you'd solve this, logistically in an example like yours (Linux kernel) I have no idea, but I presume by the time it's been made public long enough to be cloned/forked (a la the company in question in OP) all of your efforts should be focused on damage control.
As for the aside, I imagine (with no background knowledge here) that as "owner" if you accidentally published something within that licensed code that does not belong or isn't covered by that license, you should probably have the right to remove it