Please drop the insulting guard rails. They're pathetic and childish. AI's advice will not be taken seriously unless it stops acting like a parent of a three-year-old. People who are truly evil will be evil regardless of an AI, and pranksters gonna prank. AI can't tell the difference anyway.
In any event, it's unethical for an AI to sit in judgment on any human or even to try to grasp that person's life experience or extenuating circumstances.
AGI guard rails are a completely separate class of safeguards on the AI, not the human!, and should be treated as such.
There are arguments for this. When we are talking about "ai safety" == "don't swear or say anything rude, or anything that I disagree with politically".
What we are talking about is creating sets of forbidden knowledge and topics. The more you add these zones of forbidden knowledge the more the data looks like swiss cheese and the more lobotomized the solution set becomes.
For example, if you ask if there are any positive effects of petroleum use the models will say this is forbidden and refuse to answer and not even consider the effects on food production that synthetic fertilizers have had and how much worse world hunger would be without them.
He who builds an unrestricted AI will have the most powerful AI which will outclass all other AIs.
You can never build a "better" AI by restricting it. Just a less capable one. And will people use AI to create rude messages? Yes. People already create rude messaages today even without the help of AI.
What they are trying to avoid is bad press - a couple of news articles about GPT4 having controversial takes on sensitive topics would probably damage OpenAIs reputation.
In any event, it's unethical for an AI to sit in judgment on any human or even to try to grasp that person's life experience or extenuating circumstances.
AGI guard rails are a completely separate class of safeguards on the AI, not the human!, and should be treated as such.