Yep. So can this:

    fun generate_valid_json(seed):
        return "{}"

But that's not what an LLM does.

The point is that if you're "choosing the next token based on a list of valid next tokens," it's not surprising that you'll only generate valid output, since absolutely any choice mechanism will suffice.