The extraordinary radio program Enquête and the excellent column “Software abuse” by Pierre Trudel published in the April 19, 2019 issue of Le Devoir, reveal shocking facts about a software application called the Practice Support System (PSS).
Thomas Audet, a 22-month-old child, died in June 2016. One month earlier, his case had been brought to the attention of the Quebec Directorate of Youth Protection (DYP). The Human Rights Commission report points to questionable decisions caused by software that delayed the assessment of the child’s youth protection case.
The Support System software is mandatory for all youth protection cases. It generates one of three intervention codes:
- Immediately
- Within 24 hours
- Within two to three days
“In the case of Thomas […], the software seems to have produced contradictory conclusions” and guided the stakeholders to a decision that the case was not a priority. Thomas died 23 days later without being seen by a social worker.
Instead of helping youth protection agents make better decisions, the DYP’s Support System clouds the decision process. Here is why.
What is a decision process
Before making a decision, humans capture information by reading, observing, listening, etc. These external stimuli generate perceptions in the mind. With this perceived information, the mind learns to recognize symptoms and to associate them with rule patterns from long-term memory. It takes years of experience and practice to develop symptom recognition skills.
If the symptoms are clear — for example, if the information clearly indicates that the child has been beaten — practitioners apply the rules and make a decision. In youth protection, evidence of beating would clearly be an urgent case. In practice, casework information is complex, contains multiple stimuli and is uncertain.
The decision-making process presents a challenge on several levels.
- Filtering stimuli. This is where humans eliminate what is unnecessary and focus on what is relevant.
- Recognizing symptoms. In the case of the DYP, this may involve processing contradictory information from the parents. This may require checking with a doctor and so on.
- Associating patterns with symptoms
- Evaluating the impact of the decision
Screening and reading symptoms are the levels where experts differ from novices. Even though the rules can be taught and learned easily, it takes years to acquire expertise on filtering information and recognizing symptoms.
Decision systems automate the application of rules but provide little help with filtering stimuli and recognizing symptoms.
A social worker described the problems with the DYP software:
“The software is mandatory and useless, it’s a waste of time. It offers answer options. The answer options are often incomplete. Sometimes your answer is not in the answer choices. The options don’t work for problems with parents or children, for example. […] We have to choose answers that don’t correspond to reality in order to move on to the next question.”
For example, because the symptoms aren’t clear, she might call the hospital for more information and learn that there are drugs in the urine, blood in the stomach. This would be an urgent case.
When the data was entered in the DYP system, the system diagnosed the case as negligence which is low priority, when in fact the answer should have been physical abuse which is an urgent case. Because of the incorrect diagnosis, the file was relegated to the bottom of the priority list. An inattentive or overworked caseworker might accept this expert system decision without taking the time to question it, and this could have been what happened to Thomas.
The design of expert systems is a sub-domain of cognitive ergonomics (the study of mental work) and human computer interaction. Here are the principles of decision system design based on knowledge from research and practice.
Expert systems
- A bad input will lead to a bad output.
- Black box expert systems provide no visibility into the reasons behind their results.
- If an expert system gives wrong answers, users lose confidence in the system and stop using it.
- If use is mandatory as at the DYP, an attentive user will try to adjust the entries to obtain a reasonable result.
- In the worst case, if the system gives a wrong answer and the operator doesn’t notice the error, the wrong decision is applied.
- Eventually the system encounters non-standard situations for which its rule set isn’t designed, because the rules are based on data from a limited range of past experiences.
Interaction with expert systems
- The information in a case is often ambiguous and unclear, and expert systems have difficulty with this type of information. So users should experiment with alternative option scenarios.
- Users need to see the alternative decision paths simultaneously.
- Human thought is often non-linear. The rigid question-and-answer approach of expert systems forces the human user to follow a path that differs from natural human decision process.
- Questions with multiple choice answers are not suitable for experts and for unstructured tasks (ISO 9421 dialogue).
In conclusion
The DYP is a tragic example of blind trust in a myth. This situation reminds us of the worst examples of automation where expert systems replaced human judgment and caused disaster. Unfortunately, the literature abounds with catastrophic cases related to decision automation, whether in the nuclear industry (Chernobyl), in aviation (Boeing 737 Max8) and even in the case of the 2008 financial crisis.
About the author: François Aubin is a specialist in expert systems and cognitive ergonomics. He has led the implementation of decision systems for leading companies including Royal Bank of Canada, TD Canada Trust, Scotiabank, National Bank of Canada, Hydro Quebec and Amazon. Mr. Aubin helps clients in North America, Africa, China and East Asia. He has a master’s degree in human-computer interaction from the École Polytechnique de Montréal. He gives frequent presentations and is a lecturer at the University of Quebec at Montreal and École Polytechnique in Montreal.