Understanding Machine Minds 🧠❓

What is "Explainable AI"?

January 02, 2024

Welcome to Answers on AI: making you smarter about artificial intelligence, one email at a time.

Before we kick it off, a quick request. This free newsletter is intended to help educate the masses so they can understand what’s happening in AI. It’s only able to grow thanks to readers like you, who share it with others. If you’d be willing to forward it to a friend, we’d greatly appreciate it. And we’re sure they’d appreciate it too!

First time reader? Sign up here to get Answers in your inbox. And now, on to the question…

What is Explainable AI and why is it important?

Today we'll delve into the concept of Explainable AI—a crucial aspect of the AI ecosystem—which sheds light on how artificial intelligence makes its decisions. Understanding its significance is essential as we navigate the increasingly complex world shaped by AI technologies. Here's what you need to comprehend:

  • 🕵️ Understanding the Black Box: As artificial intelligence (AI) systems become increasingly complex, their decision-making processes can seem as mysterious as a magician's secrets. Under the hood, the neural networks that power today’s impressive AI are tremendously large graphs of numbers with large numbers of mathematical steps—virtually impossible for a human to truly grasp. Explainable AI (XAI) aims to make the workings of AI more transparent, offering insights into how and why a system makes certain decisions. By providing clearer explanations, XAI can help us trust AI solutions, adopt them more widely, and ensure they align with ethical standards.

    • One area of active research is Concept Relevance Propagation (CRP), a new method for identifying which parts of neural networks (the foundation of how modern AI systems work) are actually responsible for decisions the AI makes.

  • 🤖 Why XAI Matters: Trust is the currency in the human-AI relationship. Without explanations, AI remains an enigmatic black box that many are hesitant to rely on, especially in critical areas like healthcare, finance, and autonomous vehicles. XAI not only fosters trust but also facilitates debugging and improves learning outcomes by allowing developers to understand the AI's "thought" process and identify areas for improvement.

    • Consider this article from Forbes, which argues that while AI is helpful to companies in identifying the possibility of customer churn, Explainable AI is a necessary step to understand why the models expect those particular users to churn. In other words, what are the attributes that make churn more likely?

  • ⚖️ Balancing Act: Striking the right balance between explainability and AI performance presents a unique challenge. The most accurate AI models, like deep learning systems, are often the least interpretable. Simplifying AI to make it more explainable can sometimes result in a loss of effectiveness. Moving forward, we need to innovate ways to retain high performance while making the AI's decision-making process as clear as possible.

  • 🔍 XAI as a Moral Compass: As AI becomes more prevalent, the ethical implications of its decisions come under greater scrutiny. Explainable AI serves as a moral compass, ensuring that the algorithms governing our lives do not perpetuate biases or unfair practices. In the future, we can expect XAI to play a crucial role in maintaining the ethical integrity of AI systems by allowing us to understand and correct biases in their decision-making.

    • Check out Pedro Ferreira’s recent article in Finance Magnates arguing that Explainable AI is critical in AI financial systems, not only for business reasons, but also for ethical ones.

What do the experts say?

“Explainability is undoubtedly crucial in certain cases. In health care, for example, deep learning models have been used in hospitals to predict sudden deteriorations in patient health, such as sepsis or heart failure. But while these models can analyze vast amounts of patient data—from vital signs to lab results—and alert doctors to potential problems, the interpretive leaps which they can uniquely provide are a function of complex computations. As a result, the exact pathways and combinations of data points they use to arrive at their conclusions may not be clear to clinicians. This "black box" nature can make it challenging for doctors to fully trust the model's predictions without understanding its reasoning, especially in life-or-death situations.”

— Hamilton Mann, from Do All AI Systems Need to Be Explainable? in Stanford Social Innovation Review

“AI… always has to be explainable. After all, if a human has the final sign-off on a critical business process, they need to understand what they are signing. That means the results need to be presented in a way that is easily intelligible. Still more importantly, every process needs to be auditable – and that will also necessitate human involvement.

While AI and automation are lightly regulated at the moment, there is every likelihood that this will change in the near future. It is possible that businesses will need to provide some kind of log or auditability for why decisions were made. This new area is not covered by legislation or frameworks at the moment, but it is critically important that businesses prepare themselves for what’s coming.”

— Christian Pedersen, from Explainable AI – why humans must always be able to understand what AI is thinking in Diginomica

Stay Tuned for More

In each issue, we bring you the most interesting and thought-provoking questions of the day to help make you smarter about AI. Stay tuned for more questions, and more importantly, more answers.

Share Answers on AI

Want to help your friends get smarter? Share Answers with them! We may even give you a shoutout or the ability to ask us your own AI question.