I could not disagree more with these comments. Psychologists are just now starting to study the phenomenon of "algorithm aversion", where people irrationally trust human judgement far more than algorithms. Even after watching an algorithm do far better in many examples.
The reality is humans are far worse. We are biased by all sorts of things. Unattractive people were found to get twice as long sentences as attractive ones. Judges were found to give much harsher sentences right before lunch time, when they were hungry. Doing interviews was found to decrease the performance of human judges, in domains like hiring and determining parolle. As opposed to just looking at the facts.
Even very simple statistical algorithms far outperform humans in almost every domain. As early as 1928, a simple statistical rule predicted recidivism better than prison psychologists. They predict the success of college students, job applicants, outcomes of medical treatment, etc, far better than human experts. Human experts never even beat the most basic statistical baseline.
You should never ever trust human judges. They are neither fair nor accurate. In such an important domain as this, where better predictions reduce the time people spend in prison and crime, there is no excuse not to use them. Anything that gets low risk people out of prison is good.
I believe that any rules that apply to algorithms should apply to humans too. We are algorithms too after all. If algorithms have to be blind to race and gender, so should human judges. If economic information is bad to use, humans should be blind to it also. If we have a right to see why an algorithm made a decision the way it did, we should be able to inspect human brains to. Perhaps put judges and parolle officers in an MRI.
Stating that method A is problematic does not automatically mean method B is better.
> The reality is humans are far worse
Citation needed - especially when comparing against a specific instantiation of a machine learning model. Papers published by the statistician in the article used only 516 data points. Most data scientists running an A/B test wouldn't change their homepage with only 516 data points. There's no guarantee the methods he is using for the parole model involve better datasets or models without deep flaws.
An algorithm or machine learning model is not magically less biased than the process it is replacing. Indeed, if it's trained on biased data, as you believe by stating "never ever trust human judges", then the models are inherently biased in the exact same way.
If you give a machine learning model a dataset where one feature appears entirely indicative (remember: correlation is not causation), it can overfit to that, even if that does not reflect reality.
I highly recommend reading "How big data is unfair: understanding unintended sources of unfairness in data driven decision making"[1], by Moritz Hardt, a Google machine learning researcher who has published on the topic (see: Fairness, Accountability, Transparency). It is a non-technical and general introduction to some of the many issues that can result in bias and prejudice in machine learning models. To summarize, "machine learning is not, by default, fair or just in any meaningful way".
Algorithms and machine learning models _can_ be biased, for many reasons.
Without proper analysis, we don't know whether it's a good or bad model, full stop.
>Citation needed - especially when comparing against a specific instantiation of a machine learning model. Papers published by the statistician in the article used only 516 data points. Most data scientists running an A/B test wouldn't change their homepage with only 516 data points. There's no guarantee the methods he is using for the parole model involve better datasets or models without deep flaws.
516 is more than enough to fit a simple model. As long as you use cross validation and hold out tests to make sure you aren't fitting. 516 data points is more than a person needs to see to be called an "expert". Many of the algorithms I referenced used fewer data points, or even totally unoptimized weights, and still beat human experts.
>An algorithm or machine learning model is not magically less biased than the process it is replacing. Indeed, if it's trained on biased data, as you believe by stating "never ever trust human judges", then the models are inherently biased in the exact same way.
We have ground truth though. Whether someone will be convicted is a fairly objective measure. Even if it's slightly biased, it's still the best possible indicator we have of whether or not someone should be released. If you had a time machine that could go into the future and see who would be convicted, would you still argue against using that information, because it might be biased? Leaving people to rot in prison, even if all the statistics point to them being very low risk, is just wrong.
>"machine learning is not, by default, fair or just in any meaningful way"
Humans are not, by default, fair or just in any meaningful way. Nor are they accurate at prediction. Any argument you can possibly use against algorithms applies even more to humans. That's my entire point. You should trust humans far, far less than you do.
> You should trust humans far, far less than you do.
Which is why I don't trust the people picking the algorithm. You still have human bias, but now they are easier to hide behind complicated algorithms and unreliable data.
edit: removed original editing error
edit2: You say I should trust the algorithm, but y9u seem to be going out of your way to ignore that the algorithm itself has to be created by someone. You haven't reduced the amount of bias; trusting an algorithm simply codifies the author's bias.
You should trust the "complicated" algorithm far more than you trust the complicated human brain of the judge, who was also trained on unreliable data.
Look it's easy to verify whether parole officers are better at predicting recidivism than an algorithm. If the algorithm is objectively better than it should be used.
Given an unbiased algorithm that does better at predicting recidivism, it would be easy to deliberately construct an algorithm that does almost as much better, but is egregiously biased. For example, if you had been wronged by somebody named Thiel, you could persuade it to never recommend parole for anybody named Thiel. There aren't enough people named Thiel for this to substantially worsen its measured performance.
Given that it's easy to construct an example of how you could deliberately do this, and it's so easy to accidentally overfit machine-learning algorithms, we should be very concerned about people accidentally doing this. An easy way would be to try a few thousand different algorithm variants and have a biased group of people eyeball the results to see which ones look good. If those people are racist, for example, they could subconsciously give undue weight to type 1 errors for white prisoners and type 2 errors for black prisoners, or vice versa.
The outcome of the process would be an algorithm that is "objectively better" by whatever measure you optimize it for, but still unjustly worsens the situation for some group of people.
A potential advantage of algorithms and other rules is that, unlike the brain of the judge, they can be publicly analyzed, and the analyses debated. This is the basis for the idea of "rule of law". Aside from historical precedents, though, the exploitation of the publicly-analyzed DAO algorithms should give us pause on this count.
Deeper rule of law may help, but it could easily make the situation worse. We must be skeptical, and we must do better.
Re: most data scientists wouldn't change their home page based on 516 data points:
Out of curiosity, if you find that in 240 out of 261 small round blue cell tumors, your algorithm can make a better treatment decision than the pathologist, do you wait for more, or get the algorithm into trials ASAP?
Careful now. I didn't mention how long you might be waiting. This is a domain specific trick question.
Furthermore, we have an appeals process to address sentencing mistakes. It isn't easy, but if the risk scoring algorithm turns out to be materially wrong and contributes to a harsher sentence than warranted, that can be considered a factual error and the basis for resentencing.
It's almost as if our justice system is geared towards a reinforcement learning approach, due to the consequences of both action and inaction...
Additionally, before we can approach AI justice we'd need artificial intuition. If you know what the local minima are, you could game the model to convict or defend anyone. The AI would need a way to determine that it is being gamed. For example: taking pictures of bullet casings framed in a specific way might guarantee convictions.
The problem with putting machines in control, is that it centralizes power. In these kinds of complex situations, I would rather have thousands of humans each making suboptimal decisions, than decisions by half a dozen algorithms designed by a small group of people.
It's the same reason many people oppose electronic voting. Sure, letting humans count votes seems inefficient, and might enable some small scale fraud, but it's much more difficult to commit large scale fraud if there are many people involved. When you use voting machines, you put a tremendous amount of power into the hands of the people who design and build those machines.
It's easy enough to make the machines open source. And anyway, I can't imagine there is any incentive for there to be some kind of mass conspiracy involved here.
>I could not disagree more with these comments. Psychologists are just now starting to study the phenomenon of "algorithm aversion", where people irrationally trust human judgement far more than algorithms. Even after watching an algorithm do far better in many examples.
Those algorithms are also designed by humans and codify biases and prejudices and injustices that are considered OK by society into them.
Besides, it's like going for a run. Other might might be far better at it, but we want to do it ourselves, even badly, than just be spectators in our own life. Same for humans becoming spectators while algorithms run business "better".
One quality that's not accounted for in accessing better is "more participative".
Human judges also "codify biases and prejudices", and they do it far worse. That's my entire point. And algorithms shouldn't codify prejudices, if they are looking at objective data, and also blind to those features. Humans are not objective nor blind.
And I don't think preferring to do stuff yourself is an excuse. These are people's lives on the line! More accurate predictions mean fewer people in prison, fewer murders and robberies, etc. Or in a medical domain where more accurate predictions mean fewer people die of terrible diseases. You should always prefer algorithms if there are consequences for worse predictions!
But the humans would have to explicitly build the biases into the algorithm. Many human biases are unconscious, so we aren't aware that our conclusions are biased by race, a subject's attractiveness, etc. Going with an algorithm would minimize unconscious biases, and prevent us from reaching a conclusion based on emotion and then just back-fitting a rational-sounding argument.
>But the humans would have to explicitly build the biases into the algorithm.
Explicitly as in coding them, but not necessarily as doing it consciously. After all biases are just how people see reality, so it would be natural to code based on one's biases.
Also it's less the biases (that are inevitable), and more the blind following of biases (algorithmically) that I'm concerned about.
>Going with an algorithm would minimize unconscious biases, and prevent us from reaching a conclusion based on emotion
The latter is not necessarily (or even commonly) bad, it's more of a citation needed kind of thing. I'd rather we DO reach conclusions based on emotions. Naked logic sees everything as black and white, it's emotions that make civilisation nuanced (including empathy, pity, gut instinct, abhorrence, etc).
It's a lot harder to hide biases in code than you assume. Even if that takes place, code can be updated and those biases can be removed, easily. If they're not removed, at least we know what they are.
Can you do the same with the human subconscious? Maybe, given lots of time and effort.
How do you evaluate success?
Can you do it at scale? I don't think so.
>Racists have emotions, too, and often make decisions based on them
And knives can be used both for murdering people and spreading jam on your bread.
The answer to "racist emotions" are emotions of empathy and love towards the other, and not some objective logic that one can follow to fix racism.
After all, one could always use objective logic to justify racism if one wants to -- e.g. "blacks are involved in a higher percentage of violent crimes compared to their population". That's a totally objective statement, it just skips trying to find the reasons why that might be so, and uses the numbers to justify thinking of them as violent inherently.
You then have the same issues as with electronic voting.
How can you be sure your 'fair' algorithm is running for each individual court case? That it hasn't been tampered with, swapped out, hacked or otherwise changed? Do you give the entire court room the ability to look at the machine and what it's running?
How do you then stop any of them messing around with it when you're not looking?
You've then got the unfortunate implications that an automated system might create. Okay, the court system very much isn't neutral in regards to race, wealth, gender, nationality, religion or anything else. How do you stop an automated system discriminating based on those factors? Because there are probably different statistics and likelyhoods of different groups committing crimes.
And there are various other issues too. Do you trust a small group of people, a company, some open source contributors or an individual to program a system to make 'fairer' decisions than a country's worth of judges? Because the amount of factors that go into a court case means that a piece of software has to deal with a thousands or even millions of possibilities. Which provide a place for an unscrupulous, lazy or shortsighted programmer to screw things up.
And there's also the old trust issue. People tend to trust other people more than machines, especially those seemingly associated with a faceless corporation, large government or unknown force. A system that replaces people, no matter how good the algorithm, would get a lot of bad press in this sort of situation. People won't be happy about being sentenced to prison by a computer program. Heck, even a judge using a smartphone app would cause an uproar about how 'soulless technology is replacing human intuition' or something similar.
As for trying to make human judges blind to race and gender. That's its own challenge. Facial expressions and courtroom behaviour can indicate whether someone is guilty or not about as much as what they actually say.
Try and anonymise that, and you end up in a situation where those who can act regretful over the phone/without being soon have an advantage over those that can't.
There are also situations where being blind to race and gender would be a bad thing.
Economic information can cause bias, but it can also be used to treat a defendant more fairly as well. If someone was really poor/starving on the streets and stole to stay alive, shouldn't that be indicated somehow? There's a difference between someone like that and a rich guy who stole because he was bored or wanted more.
How is that any different from other machines used in court? From transcripts/recordings, to forensic tests like DNA machines and algorithms.
But in the worst case, you can print the algorithm out on paper. They are generally simple linear models or decision trees. You could easily verify the output by hand, albeit a bit tediously.
In the case of more complicated models like random forests or ensembles, you can generate lots of artificial data, and then train a simpler model to approximate its output as best as possible. Then you can print out that model.
In the old days, they printed out actuarial tables. Big tables that you would look up your case, and find the recommended output.
>Try and anonymise that, and you end up in a situation where those who can act regretful over the phone/without being soon have an advantage over those that can't.
That's exactly the problem. People vastly overestimate how well they can read facial expressions. I recall studies that officers couldn't tell liars from nonliars better than chance, but they believed they could.
Nothing on the planet can do the job of a judge perfectly. There will always be edge cases and vague situations. Human judges are sometimes unfair and inaccurate, but we can listen to their reasoning, criticize it, continue the discussion by appealing their decision, and so on. How do you criticize the reasoning of a statistical algorithm?
You can't listen to their reasoning. You can listen to their justifications, which aren't the same thing at all.
The attraction of an algorithm is that you can control strictly what information goes in, and what objective is targeted. A judge may - consciously or not - use the information of the accused's skin color. A judge may - again consciously or not - target objectives besides lowering recidivism.
An algorithm isn't always better, but when it produces unwanted results, we can criticize it, think about why it went wrong, improve it. When a judge makes mistakes, there's not any easy way to improve him.
I think artificial objectivity is "bigger" than artificial intelligence. It's not that ML algorithms are so much better than human brains at what they do - it's that we can have more confidence about how it does it. Oddly enough, considering how they're often criticized for opacity.
> An algorithm isn't always better, but when it produces unwanted results, we can criticize it, think about why it went wrong, improve it. When a judge makes mistakes, there's not any easy way to improve him.
I believe that now that there is much fuzz about AI you can improve lots of things. I would like to see in 200 years if you can - no, you can't, why? For the same reason you hardly can change things now: "the algorithm has been studied, improved, tested over and over" -> this is the law. But who says it's correct or fair? A judge, exactly because he is a human, can improve. How? For example, with a feedback system or by requiring explanations for his judgement (based on facts, not moods) - if he doesn't do so, he shall be accountable for his actions.
I am not too much into law or politics, but things are that way since always and hardly have changed. But we know how things will go. Algorithms/machines will slowly replace everything, because they lower the costs with the promise that things will be more dynamic and that life will improve - however, the usual gang of people will manage to be "more equal" than others, also when the algorithm works perfectly.
> The attraction of an algorithm is that you can control strictly what information goes in [...]
How is this attractive? Information can be easily manipulated, delayed, withheld. Essentially, you are just shifting the accountability - from the judge to the guy who feeds the algorithm.
And by the way, we are talking about judges who don't apply the law appropriately. Let's talk about those judges who applying the law put their lives at risk - or simply get killed. Shall we? Can we at least trust those judges?
The question is predicting recidivism. Whether someone will commit a crime again after being released. If they are a danger to society. And grouping people into low risk or high risk categories, so people spend less time in prison, but really dangerous criminals are watched.
If human judges do better at this task, it's very easy to test it. We ask them to predict what criminals are low risk or high risk. And they barely do better than chance, and are easily beaten by the simplest statistical baseline. Yes algorithms have edge cases, but humans have far more.
Lots of replies here that the algorithms can also be biased since humans design them or the data they are built from is biased. And this is certainly correct.
But I believe an even bigger advantage of algorithms is repeatability. Yes initially the algorithms might be biased, but they will always given the same input, deliver the same verdict. And over time we will be able to perfect the algorithms.
It's like automated testing. Initially at first automated testing is far less effective than manual testing, but as you add and improve your tests they outperform manual testing almost always.
And just like with testing you reduce the need for manual testing, algorithms like these do not have to completely eliminate human judges, but can reduce the amount of cases they are needed for, with improvements in sentencing and processing times.
The problem I have with this, is the assumption that the "we" perfecting the algorithm is going to be doing so with the best moral and ethical intentions. In this case, we are dealing with an area (law enforcement) which has in the past been misused for political / power purposes.
In my opinion, such areas definitely needs some degree of checks and balances to prevent power abuse. Reducing risk assessment to a ML style single algorithmic number would give the people in charge of the algorithm an awful lot of power.
I do agree that if humans were entirely moral and ethical, ML can be a huge net gain. I'm not sure I trust humanity that much, personally.
You point of reference, your past experience, your his-tory. We are seriously in bad shape, not only do we have no right to privacy, and can we be prosecuted for mere words alone.. Now they will try to prove we are guilty prior to committing an offense, and we are doomed.
I count at least seven statements that should be cited. I've heard the "harsher sentences before lunch" thing too but have never seen it cited. I've never heard ugly people get 2x the punishment of attractive people.
I agree. Sorry I was on mobile and didn't have them on hand, but I will have sources soon. I'm currently writing an article on this so it was fresh in my memory.
I agree that human judgement is often flawed and biased. This particular algorithm appears to be always racially biased[1], and wasn't even tesed for it. Substituting flawed human judgement for systematic bias is a bad, bad idea.
It's not racially biased. It doesn't even use race as a feature. It just so happens that blacks statistically have worse criminal records, and people with worse criminal records tend to commit more crimes in the future. A white person with the same record is not treated any differently. And the decision is based on objective data and statistics, not flawed human judgement. Human judges are incredibly biased and their predictions are barely better than chance. It's insanity that humans were ever allowed to make important decisions like this.
It just so happens that blacks statistically have worse criminal records, and people with worse criminal records tend to commit more crimes in the future.
Right, but there is plenty of evidence that this (the statistically worse criminal records) is an artefact of sentencing[1].
Using that biased feature as an input is going to make it worse: the algorithm will recommend higher sentences because previously blacks had previously had more severe sentences.
This is almost the exact definition of implicit bias[2]
Well first of all that article is exactly why we should use algorithms. Judges' sentences are arbitrary and biased, and not very correlated with the actual degree of risk. We should set sentences algorithmically, based on how likely the person is to commit a crime again if they go free. Not how the judge feels that day.
Second, a criminal record is the crimes you committed in the past, not the sentence a judge gave. The algorithm would see your record and determine your level of risk, along with other objective factors.
The algorithm has no bias. It sees the data and uses it to make the most accurate predictions possible. It doesn't care about race, and it is blind to it. A white criminal with the same record, should be given the same prediction as a similar black criminal. And the predictions it makes should be pretty accurate and objective.
I think the whole idea here is frightening and unjust. We are supposed to give all people equal rights. What people might do is irrelevant. A person whose demographic/conditional expectation is highly criminal should be given an equal opportunity to rise above it, else they might see the system is rigged against them and turn it into a self-fulfilling prophecy.
It's frightening depending on how you use the data.
A good example perhaps is that I like to horse around when I'm at the beach. I'm more like to get hurt than others who are more cautious. I'm also more likely to hit people accidentally.
I had some parents of younger children approach me and ask me to stay on the far side of the beach. On one hand it felt rude, but on the other it allowed me to be rambunctious and it allowed the parents to prioritize their children's safety.
The world isn't flat enough for this to be a reality yet, but if you cluster people by their morals, you don't have to throw them in jail. Put all the drug users together. Keep the drugs away from people who don't want anything to do with them.
Usually if people are more likely to commit crimes, it's either because they are desperate (which means successful intervention is likely provided you can solve their core problems), or it's because they find that activity/action/crime to be morally or culturally acceptable. To the extent that you can exclude that culture from your own daily life, you don't have to punish/kill that culture.
Pollution is a good counter example. You can't really isolate a culture of pollution because it's going to affect everyone else anyway. So there are limits.
As long as our methods for dealing with criminals evolve appropriately against our ability to detect them, I am okay.
Human history is full of genocide though. I don't think that bodes well for our ability to respect cultures that allow or celebrate things we consider to be crimes.
"between 29 percent and 38 percent of predictions about whether someone is low-risk end up being wrong"
Wouldn't win a Kaggle contest with that error rate. What's not disclosed is the percent of predictions about whether someone is high-risk ending up being wrong. These are the ones society should be worried about.
And these are the ones that are, if such a system is put into practice, impossible to track. Because all the high-risk people are locked up. The socio-political fallout of randomly letting some high-risk people free to validate the algorithm makes this inevitable.
This leaves us in a situation where political pressure is always towards reducing the number of people classified as low-risk who then re-offend. Statistical competence is not prevalent enough in the general population to prevent this.
TL;DR our society is either not well-educated enough or is improperly structured to correctly apply algorithms for criminal justice.
The question is whether human judges do better, and they don't. We have no better method of determining whether someone is low risk or high risk. But keeping everyone locked up forever is just stupid. If these predictions let some people get out of prison sooner, I think that is a net good.
The entire concept of using statistical algorithms to 'predict crime' is wrong. It's just a kind of stereotyping.
What needs to happen is a consideration of the social-justice outcomes if 'profiling algorithms' become widely used. Just as in any complicated system, you cannot simply assume reasonable looking rules will translate to desirable emergent properties.
It is ethically imperative to aim to eliminate disparities and social inequalities between races, even if, and this is what is usually left unsaid, judgments become less accurate in the process.
Facts becoming common knowledge can harm people, even if they are true. Increasingly accurate profiling will have bad effects at the macro scale, and keep marginalized higher-crime groups permanently marginalized. If it were legal to use all the information to hand, it would be totally rational for employers to discriminate against certain groups on the basis of a higher group risk of crime, and that would result in those groups being marginalized even further. We should avoid this kind of societal positive feedback loop.
If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass, you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed.
"The social norm against stereotyping, including the opposition to profiling, has been highly beneficial in creating a more civilized and more equal society. It is useful to remember, however, that neglecting valid stereotypes inevitably results in suboptimal judgments. Resistance to stereotyping is a laudable moral position, but the simplistic idea that the resistance is costless is wrong. The costs are worth paying to achieve a better society, but denying that the costs exist, while satisfying to the soul and politically correct, is not scientifically defensible. Reliance on the affect heuristic is common in politically charged arguments. The positions we favor have no cost and those we oppose have no benefits. We should be able to do better."
–Daniel Kahneman, Nobel laureate, in Thinking, Fast and Slow, chapter 16
> It is ethically imperative to aim to eliminate disparities and social inequalities between races, even if, and this is what is usually left unsaid, judgments become less accurate in the process.
Why? Why is it 'imperative' to be wrong?
> Facts becoming common knowledge can harm people, even if they are true.
Well, they can harm people who, statistically speaking, are more likely to be bad.
If anything, I see accurate statistical profiling being helpful to black folks. Right now, based on FBI arrest data, a random black man is 6.2 times as likely to be a murderer as a random white man; a good statistical profiling algorithm would be able to look at an individual black man and see that he's actually a married, college-educated middle-class recent immigrant from Africa, who lives in a low-crime area — and say that he's less likely than a random white man to be a murderer.
Perhaps it could even look at an individual black man, the son of a single mother from the projects, and see that he's actually not like others whom those phrases would describe, because of other factors the algorithm takes into account.
> If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass, you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed.
That statement implies that we should avoid the algorithm 'arrest anyone who has committed a crime, and no-one else,' because that algorithm will necessarily result in increased differential arrest rates. On the contrary, I think that algorithm is obviously ideal, and thus any heuristic which leads to rejecting it should itself be rejected.
> a random black man is 6.2 times as likely to be a murderer as a random white man
But I bet the likelihood of a random man or random person to be a murder is so low that "6.2 times" doesn't really tell you much about the underlying data.
> If anything, I see accurate statistical profiling being helpful to black folks.
But that's the thing... These statistical models aren't accurate, and we know that they aren't. The algorithms used for sentencing or predictive policing are just reflections of the creators, or more likely, the current legal system.
> It is ethically imperative to aim to eliminate disparities and social inequalities between races
Agreed 100%
> even if...judgments become less accurate in the process.
Absolutely not. You are actually advocating that because someone may come from a "marginalized higher-crime group" that they not be punished accurately should they commit a crime?
> If you accept that government should want to avoid a segregated society, where some groups of people form a permanent underclass
I agree that society should want that, but I don't agree that the government should act as a force to impose social justice upon society.
> you should avoid any algorithm that results in an increased differential arrest rate for those groups, even if that arrest rate is warranted by actual crimes committed.
Literally saying that someone from a "marginalized higher-crime group" should get away with criminal activity.
How is it "accurate" to punish someone not based on their own actions, but because they share traits with a particular group of people?
Prejudice doesn't become acceptable just because it's based on a complicated algorithm and a lot of data. Judge people for what they did instead of pre-judging them because they seem similar to a group you don't like.
The problem with stereotyping isn't stereotyping per se, it's that it makes people lazy and not bother to take new information into account when they become available.
I like the proposal from the EU that automated decisions with a material impact must firstly come with a justification -- so the system must be able to tell you why it came out with the answer it gave -- and must have the right of appeal to a human.
The implementation is the difficult bit, of course, but as a principle, I appreciate the ability to sanity-check outputs that currently lack transparency.
Interpretability is going to be a huge area of research in machine learning, what with the advent of deep learning techniques. It's hard enough explaining the output of a random forest, what about a deep net with 100 layers? In some cases it doesn't matter, e.g. you generally don't care why Amazon thinks you should buy book A over book B, but in instances where someone's prison sentence is the output, it will be vital.
As someone who does machine learning, this absolutely terrifies me. The "capstone project" of determining someone's probability of committing a crime by their 18th birthday is beyond ridiculous. Either the author of the article hyped it to the extreme (for the love of everything that's holy, stop freaking hyping machine learning) or the statistician is stark raving mad.
The fact that he does this for free is also concerning, primarily as I doubt this has any level of auditing behind it. The only thing I agree with him on is that black box models are even worse as they have even worse audit issues. Given the complexities in making these predictions and the potentially life long impact they might have, there is such a desperately strong need for these systems to have audit guarantees. It's noted that he supposedly shares the code for his systems - if so, I'd love to see it? Is it just shared with the relevant governmental departments who likely have no ability to audit such models? Has it been audited?
Would you trust mission critical code that didn't have some level of unit testing? Some level of code review? No? Then why would you potentially destructively change someone's life based on that same level of quality?
> "[How risk scores are impacted by race] has not been analyzed yet," she said. "However, it needs to be noted that parole is very different than sentencing. The board is not determining guilt or innocence. We are looking at risk."
What? Seriously? Not analyzed? The other worrying assumption is that it isn't used in sentencing. People have a tendency to seek out and misuse information even if they're told not to. This was specifically noted in another article on the misuse of Compas, the black box system. Deciding on parole also doesn't mean you can avoid analyzing bias. If you're denying parole for specific people algorithmically, that can still be insanely destructive.
> Berk readily acknowledges this as a concern, then quickly dismisses it. Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race.
There are so many proxies for race within the feature set. It's touched on lightly in the article - location, number of arrests, etc - but it gets even more complex when you allow a sufficiently complex machine learning model access to "innocuous" features. Specific ML systems ("deep") can infer hidden variables such as race. Even location is a brilliant proxy for race as seen in redlining[1]. It does appear from his publications that they're shallow models - namely random forests, logistic regression, and boosting[2][3][4].
FOR THE LOVE OF EVERYTHING THAT'S HOLY STOP THROWING MACHINE LEARNING AT EVERYTHING. Think it through. Please. Please please please. I am a big believer that machine learning can enable wonderful things - but it could also enable a destructive feedback loop in so many systems.
Resume screening, credit card applications, parole risk classification, ... This is just the tip of the iceberg of potential misuses for machine learning.
Edit: I am literally physically feeling ill. He uses logistic regression, random forests, boosting ... standard machine learning algorithms. Fine. Okay ... but you now think the algorithms that might get you okay results on Kaggle competitions can be used to predict a child's future crimes?!?! WTF. What. The actual. ^^^^.
Anyone who even knows the hello world of machine learning would laugh at this if the person saying it wasn't literally supplying information to governmental agencies right now.
I wrote an article last week on "It's ML, not magic"[5] but I didn't think I'd need to cover this level of stupidity.
From "Developing a Practical Forecasting Screener for Domestic Violence Incidents":
"We had data from the screening instrument for 671 households.
...
For the 516 households with complete data, there was at least one return
call for 109. Thus, about 21% had a return call within three months after
the screener information was collected. Although there is no guarantee that
all of these calls were for domestic violence incidents, no doubt most were."
Admittedly this is one paper, and not necessarily related to the system he created for parole prediction, but this would be a worryingly small dataset (and likely bad set of assumptions) for any data scientist, machine learning researcher, or machine learning practitioner.
As Francois Chollet noted on Twitter, it's very reminiscent of the random forest approach to launching drone strikes[2] that had insanely small amounts of data:
"The NSA evaluates the SKYNET program using a subset of 100,000 randomly selected people (identified by their MSIDN/MSI pairs of their mobile phones), and a a known group of seven terrorists. The NSA then trained the learning algorithm by feeding it six of the terrorists and tasking SKYNET to find the seventh."
> There are so many proxies for race within the feature set.
Yeah but, so what? Surely you don't believe race is a strong predictor after controlling for all the hundred other things? Algorithms are not prejudiced and it has no reason to use racial information when so much other data is available.
Even if somehow race was a strong predictor of crime in and of itself, so what? Lets say economic status correlates with race, and it uses that as a proxy. It still isn't treating a poor white person different than a poor black person.
And if it makes a prediction like "poor people are twice as likely to commit a crime", well it's objectively true based on the data. Its not treating the group of poor people unfairly. They really are more likely to commit crime.
> Surely you don't believe race is a strong predictor after controlling for all the hundred other things?
It can be if you select the right data, algorithms, and analysis method.
> Algorithms are not prejudiced
That's correct. However, the selection of algorithm and input data is heavily biased. You're acting like there is some sort of formula that is automagically available for any particular social question, with unbiased and error free input data. In reality, data is often biased and a proxy for prejudice.
> It still isn't treating a poor white person different than a poor black person.
I suggest spending a lot more time exploring how people actually use available tools. You seem aware of how humans bring biased judgment, but you are assuming that the creation of an algorithmic tool and use of that tool in practice will somehow be free of that same human bias? Adding a complex algorithm makes it easy to hide prejudice; it doesn't do much to eliminate prejudice.
> Its not treating the group of poor people unfairly.
Yes, it is. The entire point of this type of tool is to create a new way we can pre-judge someone based not on their individual behavior, but on a separate group of people that happens to share an arbitrary set of attributes and behaviors.
The problems of racism, sexism, and other types of prejudice don't go away when you target a more complicated set of people. You're still pre-judging people based on group association instead of treating them as an individual.
Algorithms _can_ be prejudiced. See [1]. Not being defined with bias or prejudice does not make them impervious to it.
If a dataset is biased, as you have stated by "not trusting judges" elsewhere, then the algorithm will follow that. If the dataset contains a feature that is highly indicative (be it racial information or otherwise), your model may come to depend entirely on it. The presence or absence of other data may be irrelevant, especially if you don't perform regularization or another similar tactic.
Racial (or specific feature) profiling is a very complex topic. If there's open legal questions as to whether policemen are allowed to use it, I don't feel algorithms should be allowed to.
[1]: "How big data is unfair: understanding unintended sources of unfairness in data driven decision making"[1], by Moritz Hardt, a Google machine learning researcher who has published on the topic - https://medium.com/@mrtz/how-big-data-is-unfair-9aa544d739de...
I share your concerns. The post-9/11 scramble for homeland security funding in academia really killed my enthusiasm for the AI/ML stuff I was doing at the time.
The message that is heard when tech people sell AI/ML is 'magic cure-all' and it basically comes down to not understanding Bayes' theorem: If your algorithm has any capacity to make type one or type two errors (even if the success rates seem impressive) bad things will happen when you look for infrequent events (terrorism/crime) with low priors: you get more bogus hits than true ones. Once the priors get to 50% great, but then you have bigger issues (civil war and chaos). Law enforcement is then obliged to follow up on those bogus hits which given the size of the population can be overwhelming.
And that is even before you get to ethical minefields mentioned above. Making decisions about people based on models that the implementer might not even fully understand is unnerving. It makes it easy to slip in social/socio-economic assumptions about people for those who would like to try. Even if the implementer acts in good-faith, it can easily happen inadvertently. Given how little is known about how to deal with social problems in general, trying to paper over that ignorance with AI/ML makes everything worse. It is operationalizing ignorance with high-efficiency.
> but you now think the algorithms that might get you okay results on Kaggle competitions can be used to predict a child's future crimes
Do you think anything can be used to predict crimes? Can you imagine any fair application of justice that doesn't boil down to an algorithm?
Sentencing guidelines are an algorithm. It's literally a table on page 3[0]. Do you think sentencing guidelines, which were invented to address unfairness in the justice system like this guy's work was, are unfair?
What alternative do you propose? Case-by-case judgements? Even if you could find me a model judge, be honest: the judge probably uses something that boils down to an algorithm to be just.
> brilliant proxy for race
Suppose we only used variables with a low correlation to race. Or we decorrelate them from race somehow. Suppose error on the validation set increased. Would this be sufficient to make this system more just to you? Would a reasonable person think it is more just?
> Specific ML systems ("deep") can infer hidden variables such as race
Considering how scientific you are trying to be about your understanding of ML, you're being dishonest about how much ML systems can "infer hidden variables." There's strictly anecdotal evidence of "deep"ness inferring something hidden. We think GoogLeNet is seeing higher-order features, but then we show it a picture of a cat and it thinks it's looking at a paper towel.[1] You're overstating the power of these systems to serve your point, ironically exactly what you accuse this guy of doing.
I can see how to an individual these things are unfair. But how about in aggregate? If I can show you a way to process credit card applications that makes more profit, reduces bankruptcies and increases credit usage, would you say, "don't use it"? So what if it's made with ML, or pixie dust? Would you reject these things?
The issue is not whether justice can be codified. Let us assume that you can and it's outlined on page 3 of your document. Providing examples to an ML model don't guarantee it will codify anything similar to that. This is a fundamental issue. Most machine learning models aren't highly interpretable and can also act in unexpected ways.
Re: (a) variables with a low correlation to race / (b) dishonest about how much ML systems can "infer hidden variables":
(b) The example you point to is specifically adversarial modification to the input data. I'm not sure how that's relevant to your point at hand except to say that ML models can be misguided when given slightly modified input. It doesn't show that inferring hidden variables is anecdotal. If you look at the work of machine translation with attention, it explicitly learns how to do alignments even though it is never given training data for it. We can visualize that[1].
(a) There is interesting work on trying to produce variables with a low correlation to race. One recent example[2] has two networks use the same intermediate representation created from the raw input. One of the networks tries to identify a particular variable (race / gender / ...) while the other tries to perform the overall objective task. We aim to modify the way we generate the intermediate representation such that the former network has a harder time identifying the sub groups whilst allowing the latter group to continue to improve on the original task. This is recent however. There are many other interesting explorations.
Regarding when to use features that may leak information, this is referred to as the fairness / utility tradeoff. If you want to see an example of both that and trying to prevent information leakage about gender in a dataset on adult income, refer to "Certifying and removing disparate impact"[3].
Note: the models used by the statistician in the article do not appear to feature anywhere near this level of analysis. Ensuring to avoid features that result in strong bias or prejudice against specific groups in a dataset is still an open problem.
Given that ML is likely going to be used more and more -- even if not for criminal justice applications, at least in the private sector (credit scores/loan applications, insurance, etc) -- I wonder if there's a way to develop standardized "anti-discrimination standards" and then apply some regulation to any ML algorithm that makes a "life-altering decision" by some definition?
E.g. -- a "differential discrimination" test of sorts: altering any one variable of a "protected class" such as race, gender, religion, etc. does not change the answer. You would maybe want to pick a set of canonical test profiles among real people who differ only on one axis (as closely as possible), rather than just take a test point and alter one axis directly, because you'd want all the relevant correlations (ZIP code vs wealth, etc) to remain authentic. The end result would be a set of "equivalence classes": sets of human profiles who must be considered equivalent on all relevant life-altering judgments.
Or perhaps a "unit test"-like approach: similar to how one creates a unit test for each bug one fixes, create a "criminal justice ML algorithm test suite" with canonical profiles and their results: you must judge this person to likely not re-offend, you must judge that person as a high risk, you must judge this person worthy of a home loan for $X, etc. Sort of like a body of case law. I guess the risk is overfitting -- so maybe this data set is held in trust by some regulatory agency and not revealed.
People have probably thought about this and I haven't read your links -- is building a test data set and building regulations around it something that's considered?
The European Union are likely to accelerate much of this type of research.
They recently introduced regulations specifically targeting algorithmic decision-making and a "right to explanation" for automated decisions.
There's a Wired article[1] on the general scene and a paper that investigates the impact it may have on the industry[2].
Regarding differential discrimination, check out these two papers[3][4] - they're very similar to your idea :)
>Risk scores, generated by algorithms, are an increasingly common factor in sentencing. Computers crunch data—arrests, type of crime committed, and demographic information—and a risk rating is generated. The idea is to create a guide that’s less likely to be subject to unconscious biases, the mood of a judge, or other human shortcomings. Similar tools are used to decide which blocks police officers should patrol, where to put inmates in prison, and who to let out on parole.
So, eventually a robot police officer will arrest someone for having the wrong profile.
>Berk wants to predict at the moment of birth whether people will commit a crime by their 18th birthday, based on factors such as environment and the history of a new child’s parents. This would be almost impossible in the U.S., given that much of a person’s biographical information is spread out across many agencies and subject to many restrictions. He’s not sure if it’s possible in Norway, either, and he acknowledges he also hasn’t completely thought through how best to use such information.
So, we're not sure how dangerous this will be, or how Minority Report thoughtcrime will work, but we're damned sure we want it, because it's the future and careers will be made?
This is a very scary trend in the U.S. Eventually, if you're born poor/bad childhood, you will have even less of a chance of making it.
We can already pinpoint at risk children with high accuracy, just check out any inner city ghetto. We just lack the caring needed to do anything about the root cause (e.g. and mostly poverty) that causes kids to go bad later. It isn't a mystery.
Predictive policing is quite the buzz word these days. IBM (via SPSS) is one of the big players in the field. The most common use case is burglary, I suspect because that's somewhat easy (and also directly actionable). You rarely find other use cases in academic papers (well I only browsed the literature a couple of times preparing for related projects).
The basic idea is sending more police patrols to areas that are identified as high thread and thus using your available resources more efficiently. The focus in that area is more on objects/areas than on individuals so you don't try to predict who's a criminal but rather where they'll strike.
It sounds like a good enough idea in theory but at least in Germany I know that research projects for predictive policing will be scaled down due to privacy concerns even if the prediction is only area and not person based (noteworthy that that's usually mentioned by the police as a reason why they won't participate in the research). I'm not completely sure and only talked to a couple of state police research people but quite often the data also involves social media in some way and that's the major problem from what I can tell.
Here's something I really dislike about all the coverage I've seen about these "risk assessment algorithms": There is absolutely no discussion of the magnitude of the distinctions between classifications. Is "low risk" supposed to be (say) 0.01% likelihood of committing another crime and "high risk" (say) 90%? Or is "low risk" (say) 1% vs. "high risk" of (say) 3%?
Having worked on human some predictive modeling of "bad" human events (loan defaults) my gut says it's more like the latter than the former, because prediction of low-frequency human events is really hard, and, well, they're by definition infrequent. If that suspicion is right, then the signal-noise ratio is probably too poor to even consider using them in sentencing, and that's without considering the issues of bias in the training data, etc.
But there is never enough detail provided (on either side of the debate) for me to make an informed assessment. It's just a lot of optimism on one side and pessimism on the other. I'd really love to see some concrete, testable claims without having to dive down a rabbit hole to find them.
What is Berk's model? How well does it do across different risk bands? What variables are fed into it in the states where it is used? How does prediction success vary across types of crimes, versus demographics within crime?
This article treats ML like a magic wand, which it isn't. There's not enough information to make a judgement on whether the tools are performing well or not, or whether that performance, or lack of it, is based on discrimination.
Where we do have information it is worrying:
"Race isn’t an input in any of his systems, and he says his own research has shown his algorithms produce similar risk scores regardless of race."
What?!? The appropriate approach would be to include race as a variable, fit the model, and then marginalise out race when providing risk predictions. Confounding is mentioned but the explanation of how it is dealt with, without doing the above isn't given - just a (most likely false) reassurance.
This is like machine introduced bias/racisim/castism... we need a new term for that.. and its based on statistically induced pseudo-sciences many times similar to astrology. This is the kind of AI everyone should be afraid of.
I fail to see what is unscientific about stating conditional probabilities. Astrology is unscientific because the orientation of the heavens is for the most part independent of the observables of a person's life. But the inputs to Berk's system clearly do effect the probability of committing crime. A frequentist would say that a large group of people with this set of characteristics would yield more or fewer criminals; a Bayesian would say we have more knowledge about whether such an individual will commit future crime. These are scientific conclusions. The question "how should we use this data" is a question of ethics, not science.
> I fail to see what is unscientific about stating conditional probabilities.
1. Create conditions which disadvantage and impoverish a segment of society.
2. Refine those conditions for centuries, continually criminalizing the margins that segment of society is forced to live in.
3. Identify that many of the people in that segment of society are likely to be identified as criminals.
4. Pretend that you're doing math rather than reinforcing generations of deleterious conditioning, completely ignoring the creation of those conditions that led to the probabilities you're identifying.
And science can't be divorced from ethics. These are human pursuits.
This looks like sloppy ML. But human judges do all those things already (susbtitute "just applying the law" for "doing math" in 4) and they can't be inspected - their brains are "closed source".
Sure, these humans can come up with wordy justifications for their decisions. But there are plenty of intelligent people who employ their intelligence not to arrive at a conclusion, but to justify the conclusion they already arrived at. Legal professionals aren't merely capable of this, they're explicitly trained to be experts at post-hoc justification.
And legal professionals basically ignore all criticism not coming from their own professional class. They are rarely taught any kind of statistics in law school. Nobody wants to discuss math in debate club - answers with hard right and wrong answers are no fun to them.
Your pessimism wrt. modeling may be justified, but you're not nearly pessimistic enough about people or the legal system.
I frankly don't understand your response. I described a list of despicable things humans have done, and you're suggesting that I'm not pessimistic about people.
I see statistics sometimes edges near astrological conclusions. Because sometimes there are many assumptions being made to simplify those equations. Machine learning just amplifies that assumptive behavior unless you have all the possible test-train-data. Predictions from astrology are just probabilistic - lets say the heavenly object positions are metadata. But then people started treating the results as absolute. Now you know why people dislike astrology.
Its not okay to assign risk-score because humans tend to show bias given a quantifiable number. And thats just plain bad.
Would the following be common risk factors for a child becoming future criminal? Would it not be cheaper for society to invest in this risk children early on rather than dealing with their actions as an adult? Minority report. What are your observations for risk factors? Has there been Social science any interviews of prisoners and their background feed into classification engines?
Classification ideas:
* Bad parents not raising their child
* Living in a poor neighbourhood with lots of crime
* Going to a bad school
* Parents who are workaholics.
* Single parent
* Parent who is in jail
So why do we hold computers to higher standards than humans? Either it's OK to not being able to define criminality objectively and in that case algorithms shouldn't be disqualified for this very reason, or it's not OK, but in that case humans should not be allowed to do the job either.
Convicting people for criminal behaviour based on a subjective definition of it, is what we already do wrong. Way too many innocent people end up being punished or killed, which I expect to be (objectively) a criminal act in itself. So, I thought we better first have a tool that solves that, instead of a tool that amplifies it.
I find this really interesting. I think what most people seem to be missing is the wider social context. Think about this. If you exclude white collar financial crime, pre-meditated murder, and organised crime - most other crimes are committed by the socially disadvantaged. So, if the algorithm identifies an area where crime is more likely to be committed, instead of being narrow minded and just putting more police there to arrest people, why not instead try to institute programs to raise the socioeconomic status of the area?
People are just concentrating on the crime aspect, but most crime is just a symptom of social inequality.
The American thing to do is to cry "personal responsibility" and treat the symptoms with jail time, fines and a lengthened rap sheet.
Suggesting that we treat the cause suggests we all have responsibilities as members of communities to ensure no one is in a place where crime might make sense despite the consequences.
The main question should be, like with autonomous vehicles, is does this system perform better than people (however you want to qualify that)? If so, it's better than what we have.
Second, even if it's proven better (fewer false positives, less unduly biased results) it can be improved continuously.
There is a danger in that people may not like the results because if we take this and diffuse it, has the potential to shape people's behavior in unintended ways (gaming), on the other hand this system has the potential for objectivity when identifying white collar crime, that is surfacing it better.
I'm curious how good an algorithm would be at identifying future white collar criminals. What would the risk factors be for things like insider trading, political corruption, or other common crimes?
consider the (fictional) possibility that an AI will be
"
actively measuring the populace's mental states, personalities, and the probability that individuals will commit crimes
"
https://en.wikipedia.org/wiki/Psycho-Pass
AI may be worth the trade-off if violent crime can be almost eliminated.
or consider (non-fiction): body-language/facial detection at airports; what if they actually start catching terrorists?
There is a school of thought that says some crime is necessary for a healthy, functioning society. Personally, while I would hate to be the victim of violent crime (obviously), I actually do agree that cities with very low crime levels are often stultifyingly uncharismatic.
I do wonder if this type of technology is something we should slowly approach due to the very nature of the outcome of sentencing. We already incarcerate a lot of innocent people and I truly wonder if this is something we should tread lightly on.
Law and Oracle is an adaptation of PKD's short story "Minority Report" in which the Pre-Cog (short for pre-cognition) department polices based on glimpses of future crime. Orwell's "1984" explored the similar, but distinct, notion of 'thoughtcrime'. Both works examine the implications of such policing methods and are certainly worth reading.
The oppressive regime of 1984 does not use math or computers to look for "suspects". We might argue this was because Orwell had no idea about these methods (book was written in 1948, after all) but personally I doubt he would have used this because advocating decisions to an impersonal algorithm would make the bad guys slightly less bad: in the novel the main baddie says something like "do you want to see the future of Human Race? Think of a boot stomping on a human face, forever...".
I.e. power for them is a end in itself, and the only way to use it is to make someone else suffer. I don't see any place for "impersonal algorithms to better adjudicate anything" in this.
Or, if you are time impaired, there is of course Minority Report the film, starring Tom Cruise. Not as in depth, of course, but not a bad adaptation imho.
I find them more readable than the endless wall of indistinguishable text of pick-a-random-news-site. Of course all the text is readable, and there's innumerable ways of increasing the readability. But their art direction and effort to distinguish particular content helps it stand out and helps overcome the kind of fatigue that causes readers to drop off within a couple paragraphs and has plagued other content-oriented sites with horrifying "minutes to read" estimates.
I was actually pleasantly surprised that the weird style and graphics rendered since most of their article just display as plain text due to my uBlock Origin settings. And I sorta like the weird style, mainly because it's out of character with Bloomberg.
I know, it's almost as if they don't consider you the sole and undisputed arbiter of the limits of technology in creating social policy. What a bunch of psychopaths!
When someone engages in dehumanizing language towards a group of people and their baseless characterization is satirized for being reductive, you're saying that the problem is the irony, not the bigotry?
Obviously it's your treehouse and you have the right to set whatever rules you want, but this interpretation of civility seems tilted towards creating an echo-chamber that nurtures increasingly authoritarian views among an increasingly homogenous community.
Or is there no way to couch that opinion in such a way that it won't be considered uncivil?
Just for the record: I didn't play a "card", that was a genuine question.
I didn't want to escalate, I think the escalation was already done by the man who "jokingly" called his project the "Minority Report".
If you can't talk about pathological behavior with the appropriate words, it's getting difficult to discuss this.
The commenter you booted out was showing very similar behavior btw.
Not every psychopath/sociopath (I know it's getting muddy there) is out to eat people, but they all fundamentally lack empathy and usually compensate with power, and sarcastic, deprecating snark is just that.
I don't doubt your sincerity, but sincerely dropping a lit match is still dropping a lit match. Nothing good can come of blithely invoking categories like 'psychopath' in internet arguments. Simply doing that is escalation. Of course it was only a small infraction, which is why I didn't chide you directly. The main problem with what you posted is that it implicitly invited worse.
Your comment here makes me think that you underestimate what a big deal this is. You're basically arguing, "well, they are psychopaths" about people you disagree with. You can't conduct disagreements that way on an internet forum without leaving a trail of destruction behind you, and that's enough to show that it's a wrong approach.
Any time you find yourself taking a psychiatrically diagnostic position in an internet argument, your game piece is sitting on a square from which no comments should ever emerge. That's not because your views are wrong; it's because even assuming they're 100% right, damaging the commons does more harm than posting such a comment can do good.
I'd just like to cosign your rebukes to both of us, they're very reasonable. Showing that you understand why I reacted how I did but that it wasn't productive is a totally legitimate call-out. I'm working on being a less confrontational/snarky/douchey person and rationalizing everything I do is a big impediment to that.
Sorry for the trouble, and thank you for moderating in a thoughtful and engaged way.
If you have (created!) a job that closely resembles a work of dystopian fiction, laughing that off is absolutely lacking in human empathy. That's not even the first problem with this line of work, but since you're also laughing off the problem, it deserves a rebuttal.
If I said to you that I was going to create a network of surveillance devices that also serves as mindless entertainment and routinely broadcasts faith routines that non-participants will be punished for, and you told me that sounds like something out of 1984, and I told you were paranoid, you'd think I'm mad.
And the advance of technology unhindered is not a universal good. Algorithms only have better judgment than humans according to the constraints they were assigned. If there's a role for automation in criminal justice, that role must be constantly questioned and adjusted for human need, just as the role of human intervention should be. Because it's all human intervention.
Or it's that non-specialists with only a vague understanding of what you're doing don't know what you're talking about. I don't consider stem cell researchers to be lacking in empathy because some of the hyper-religious accuse them of dangerous experiments and "playing god", even though their work "resembles a work of dystopian fiction."
Of course we can question where technology is going, but saying "gee, group x commits crimes at 27 times the rate of group y, on average we should consider them to be more at-risk" isn't 1984, it's how car insurance, the job market, dating, the stock market, almost everything works.
It's not optimally fair by a long shot, but neither is the alternative of analyzing no data and making social policy out of how you wish things were instead of how the data tell you they actually are. The solution to data telling you unpleasant facts is to try to change those facts with policy, not put your head in the sand and bleat "totalitarianism" every time a fact enters the conversation.
In context, faith routines would be things like, in the book 1984, the Two Minutes Hate. In reality, it might be a (imlicitly mandatory, if not explicitly) routine such as pledging allegiance to a flag, or mandatory participation in a moment of prayer, or something similar.
The reality is humans are far worse. We are biased by all sorts of things. Unattractive people were found to get twice as long sentences as attractive ones. Judges were found to give much harsher sentences right before lunch time, when they were hungry. Doing interviews was found to decrease the performance of human judges, in domains like hiring and determining parolle. As opposed to just looking at the facts.
Even very simple statistical algorithms far outperform humans in almost every domain. As early as 1928, a simple statistical rule predicted recidivism better than prison psychologists. They predict the success of college students, job applicants, outcomes of medical treatment, etc, far better than human experts. Human experts never even beat the most basic statistical baseline.
You should never ever trust human judges. They are neither fair nor accurate. In such an important domain as this, where better predictions reduce the time people spend in prison and crime, there is no excuse not to use them. Anything that gets low risk people out of prison is good.
I believe that any rules that apply to algorithms should apply to humans too. We are algorithms too after all. If algorithms have to be blind to race and gender, so should human judges. If economic information is bad to use, humans should be blind to it also. If we have a right to see why an algorithm made a decision the way it did, we should be able to inspect human brains to. Perhaps put judges and parolle officers in an MRI.