It's referring to the much-talked-about effect of emergent discrimination, where the model fitting process has the effect of amplifying the status quo, despite the fact that the status quo is informed in large part by structural injustices. For (oversimplified) instance: poor black people represent a cohort of loan applicants likely to default, and the model fitting process may go a step worse and attribute "default risk" to all black people.

The key thing to understand is that we're talking about discrimination that is usually unintended and unexpected by the designers of these systems.

> despite the fact that the status quo is informed in large part by structural injustices.

This is presented as if it's an unambiguous fact, when it's largely a political stance.

It's an unambiguous fact that the status quo is shaped heavily be many generations of de jure discrimination including chattel slavery and continting structural inequalities in political power that still exist that were designed to protect those other unequal institutions.

It's a subjective political view that any or all of those things are injustices, of course, since justice is a subjective thing.

This happens to be something I believe strongly, but the comment you're responding to doesn't make sense even if you strongly disagree with that, as it supposes we can snapshot American society as it is in 2016 and synthesize from it reliably just and sensible decisions. Nobody believes this, no matter what their politics (unless there's a "status-quo-ism" I'm unaware of).

The problem (or at least, one of the more important problems) being addressed in this work is the unintended amplification of the status quo --- the implicit notion that if something is a certain way now, it is best that it always be that way.