As a user of websites, I definitely notice when a website uses web fonts and relies on font-display. Preloading them is the only way to avoid jarring changes to the layout. Even if the changes are relatively small, they're very jarring. I won't do that to users of my sites. It's not hard to take care of properly.
That's a very subjective take on what properly means in this context. Many would argue that serving the content is the main thing that matters to your users, and if your font takes so long to load that you need to deal with it separately, it would be better to not do that to begin with. To put another way, as a fellow user of websites, I would much rather read what's written than look at a blank page while some marginally different version of Helvetica is fetched in the background.
That's the point of preloads. You don't need to choose. As soon as the HTML is loaded, the font is ready. You see the HTML immediately (about 10-20ms) with the web font. Win win.
