If non-gov investors had to provide the mortgages instead of the fed, that access to money would dry up.