Utilization has the lowest impact on your score. By your reasoning, homeowners with a mortgage should have a terrible credit score, when in fact it’s usually the opposite.
Utilization is 30% of your credit score making the 2nd largest single factor, behind only payment history (35%). That said utilization only counts revolving credit sources. That means installment loans like home and auto loans are not part of that particular factor.