Long before the UK joined the EU it was a member of the Commonwealth of Nations and had close relationships to countries that once were part of the Empire. Before the EU, most UK immigration came from Commonwealth countries. Freedom of movement allowed easy immigration from Europe and greater restrictions on Commonwealth immigration. People from those countries believe that the UK has prior obligations to members of the Commonwealth, and that it should prioritize those obligations (and often that privileging EU immigration is racist.) You can see the post-Brexit shift to previous obligations with the easing of immigration rules for Hong Kong citizens.
Voters felt levels of immigration in general were too high and leading to strained public services. EU law forbids restrictions on intra-EU immigration, so the only legal path to making voters happier was to heavily restrict immigration from everywhere else. It didn't work: the numbers from the EU were far too high to be offset via such a tactic, but it was at least something.
The numbers are still viewed as being too high though post-lockdowns who knows what will happen. But the limits can at least now be spread around all countries equally, and people can be prioritised based on e.g. demand for their skills. This was previously forbidden.
