Explainable AI

Imagine this. You apply for a mortgage. You have a stable job, a decent credit history, no outstanding debts. A week later, the bank sends you a letter: application denied. No reason given. You call customer service. They tell you "the system decided." That is it.

You are left wondering. Was it my age? My neighbourhood? A data entry error? You have no way to know, no way to challenge the decision and no way to fix whatever the problem might be.

This is not a hypothetical scenario. It is happening right now, at scale, across all industries. AI models deny loans, flag medical scans, filter job applications and set insurance premiums. This is the black box problem and it is the reason Explainable AI exists.

As with my previous blog post on Federated Learning, during my Master's Thesis at USI and UniMiB, I spent months investigating how we can make AI models explain their decisions and what happens to privacy when we do. Today, I want to share what I learned about XAI: how it works, the different techniques available and why European regulators decided to make it law.

Opening the Black Box

Think of a deep neural network like a chef who produces extraordinary dishes but refuses to share the recipe. You taste the result and know it is good or bad. You might even know what went into it, but not how it was prepared. This is the black box problem. XAI is the set of tools that forces open that kitchen door.

The field has evolved through two generations. Early systems relied on logical rules , essentially writing down "if X then Y" , which worked in narrow domains but felt robotic and unhelpful to anyone who was not already an expert. The second generation shifted the focus to human-centric explanations: context-sensitive, adaptable to the audience, built to be understood rather than just technically accurate. A doctor reviewing an AI-assisted diagnosis needs different explanations than a patient receiving the results and both need something different from the regulator auditing the system.

• Domain experts want detailed feature analysis.
• Everyday users want plain language.
• Regulators want proof of fairness.

How Do You Actually Explain an AI Decision?

Let's go back to the mortgage example. The bank's model denied your application. How could XAI help? Which factors mattered most? Feature attribution methods answer this question by assigning importance scores to each input.

SHAP (SHapley Additive exPlanations), rooted in game theory, calculates each feature's contribution to the final prediction. Applied to your mortgage denial, SHAP might reveal: debt-to-income ratio contributed 40% to the denial, employment duration 25% and postcode 20%. Suddenly, the black box has some structure.

LIME takes a different approach. It perturbs the input , slightly changes your income, your credit score, your address and watches how the prediction shifts. From those shifts, it builds a simple, local approximation of the model's behaviour around your specific case.

Gradient-based methods (Saliency Maps) work specifically with neural networks, computing the sensitivity of the output to each input. For image models, this creates visual heatmaps showing which pixels the model focused on.

Counterfactual Explanations do not tell you why you were denied. They tell you what a world looks like where you were approved. "Your mortgage was denied. If your annual income were €5,000 higher, the decision would change to approved." That is a counterfactual. Formally, it is the solution to an optimization problem that balances staying close to the original input while flipping the model's decision: x' = argmin d(x, x') + λ · ℓ(f(x'), y')

During my thesis, I worked with VCNet, a conditional variational auto-encoder that generates counterfactuals in a single forward pass , fast enough for real-time applications, since it operates during the training of the model. Why are counterfactuals so compelling? Because they are actionable. They don't require you to understand gradients, Shapley values or neural network architectures. They simply tell you: change this and you get a different outcome.

The Law Now Requires It

For years, explainability was a "nice to have." A research topic. A checkbox in responsible AI guidelines that nobody enforced.

That changed. The GDPR (2016) introduced a radical idea: if an automated system makes a decision that significantly affects you, you have the right to understand the logic behind it. Not a vague summary. The actual logic. This applies to credit scoring, insurance pricing, hiring algorithms and any automated decision with material consequences. For AI practitioners, this one clause rewrote the rules. Deploying an unexplainable model on decisions affecting individuals now carries legal risk. Then came the EU AI Act, which entered into force on August 1, 2024. It goes further. It classifies AI systems by risk level and imposes mandatory requirements on high-risk applications:

• Transparency , Users must know they are interacting with AI.
• Accountability , Clear responsibility for AI-generated outcomes.
• Human Oversight , Humans must be able to monitor and override.
• Explainability , High-risk systems must provide meaningful, accessible explanations.

The Catch: Explainability Can Leak Privacy

This is the part most people miss and the core of my thesis research: making a model more transparent can make it more vulnerable. When you add counterfactual explanations to a machine learning system, you are giving users and potential attackers, more information to work with. My research showed that these additional information channels can be exploited:

• Linkage Attacks , Combining explanations with external datasets to re-identify individuals. "If the model says changing this specific postcode flips the decision, that narrows down who the applicant might be."
• Model Extraction , Using explanation outputs to reverse-engineer the model itself.
• Membership Inference , Exploiting counterfactual patterns to determine whether a specific person's data was in the training set.

This creates a genuine tension. The GDPR demands privacy. The AI Act demands explainability. And the two can pull in opposite directions. This can be overcome using techniques like Differential Privacy (injecting calibrated noise into model gradients) that mitigate these risks.

Conclusion

The mortgage applicant from the beginning of this article deserves an answer. Not a shrug from customer service, not a reference to "the algorithm." A clear, honest explanation of what happened and what, if anything, they can change. The tools to provide that answer exist today. The regulations to enforce it are already in effect. The remaining challenge, as my thesis explored, is making models transparent without making them vulnerable.

The winners here won't be whoever ships the most powerful model. They'll be whoever ships a model that explains itself clearly and honestly, without leaking the data it learned from.

References

[1] Molinari, L., "Privacy Leakage of Counterfactual Explanations in Federated Learning," Master's Thesis, USI/UniMiB, 2025. [2] Mueller, S.T., et al., "Explanation in Human-AI Systems: A Literature Meta-Review," 2019. [3] Wachter, S., Mittelstadt, B. and Russell, C., "Counterfactual Explanations Without Opening the Black Box," Harvard Journal of Law and Technology, 2017. [4] Guyomard, V., et al., "VCNet: A Self-Explaining Model for Realistic Counterfactual Generation," 2022. [5] European Commission, "Regulation (EU) 2016/679 (General Data Protection Regulation)," 2016. [6] European Commission, "Regulation (EU) 2024/1689 (Artificial Intelligence Act)," 2024. [7] Nguyen, T.T., et al., "A Survey of Privacy-Preserving Model Explanations," 2024.