Observation Without Judgment: The Hidden Perils Of Machine Learning

Because most machine-learning models cannot offer reasons for theirongoing judgments, there is no way to tell when they’ve misfired if onedoesn’t already have an independent judgment about the answers theyprovide. Misfires can be rare in a well-trained system. But they canalso be triggered intentionally by someone who knows just what kind ofdata to feed into that system. – The New Yorker