Not really, most of the bundles are pretty small, in the 30/40Kb range gzipped, even smaller with brotli, the problem is when the user has to wait 5s to start using the site, waiting half a second to load the first page is ok, waiting a quarter second to open other page is also ok.
It isn't purely about speed, it is about perception, a very slow first page load is way more annoying that a couple half or quarter second loads distributed over a long interaction, after the js is cached all is nearly instantaneous, but you don't have to wait a while to start using the site in your first visit.
If you want you can also wait for the first page to download completely and them import the remaining js while the user is on the first page, but I didn't tried to see how well it works.
It depends on the use case. For something like a web storefront, this is probably true. For a web application where I spend a long time doing work, I'd rather have a bit of a wait upfront followed by snappy responses on all the other pages. Even a half-second load time can be frustrating when I'm in the middle of a task.
It isn't purely about speed, it is about perception, a very slow first page load is way more annoying that a couple half or quarter second loads distributed over a long interaction, after the js is cached all is nearly instantaneous, but you don't have to wait a while to start using the site in your first visit.
If you want you can also wait for the first page to download completely and them import the remaining js while the user is on the first page, but I didn't tried to see how well it works.