I haven't had any issues just using the stock image on an SD card - it seems to be fast enough. YMMV especially if you plan to run other things on the Pi, though in this case I'd recommend going for a mini-PC instead and just installing a standard Linux distribution on it.
I see. Really it was just going to be something like pihole which is supposedly light. Maybe better not to use the supervised if this is focused on running HA?
In the future I'll consider the mini-PC, thanks. This will have to do for now.
