Memory B and T cells would disagree with you. This is the basic knowledge for how vaccines work.

They would not disagree with you after, say, a measles infections, which thoroughly destroys the immune system at many levels.

It’s not a good idea to confuse the impacts of vaccines and infections. Vaccines are carefully developed specifically to help the immune system. Viruses evolve to hurt it.