As organizations increasingly replace human decision-making with algorithms, they may assume these computer programs lack our biases. But algorithms still reflect the real world, which means they can unintentionally perpetuate existing inequality. A study published Thursday in Science has found that a health care risk-prediction algorithm, a major example of tools used on more than 200 million people in the U.S., demonstrated racial bias—because it relied on a faulty metric for determining need.
This particular algorithm helps hospitals and insurance companies identify which patients will benefit from “high-risk care management” programs, which provide chronically ill people with access to specially trained nursing staff and allocate extra primary-care visits for closer monitoring. By singling out sicker patients for more organized and specific attention, these programs aim to preemptively stave off serious complications, reducing costs and increasing patient satisfaction. (author abstract)