And those messages are the ones that reveal your intention to hide messages. That metadata is some of the most valuable.
Also, people don't really want to use secret chats because they aren't cross platform. Sure, some people only own phones, but those that switch to laptop/desktop computer don't want to whip out phone hundreds of times a day, but opt-in for the insecure cloud chats that are accessible with simple alt+tab.
So yeah. Sure, secret chats can be mostly secure, the problem is the E2EE isn't practical to any reasonable extent and again, using it leaks metadata.
(not the E2E ones)