From Explanations to Explainability Mediation: LLMs, XAI, and the Regulatory Future of Human-Centered AI
Abstract
AI systems are increasingly embedded in high-stakes decisions. Thus, explainability is no longer only a methodological challenge; it is becoming a socio-technical and regulatory requirement. Traditional XAI methods provide valuable technical artefacts, such as feature attributions, counterfactuals, saliency maps, surrogate models and logs, that could help understand an opaque model. However, these artefacts are often difficult to interpret for deployers, supervisors and, more so, affected individuals with limited literacy in AI. Moreover, as pointed out by various studies, few researchers pay attention to the last mile problem of XAI systems, i.e. the presentation of explanations to end-users in such a way as to be comprehensible, effective and trustworthy.
The recent emergence of Large Language Models (LLMs) opens a new possibility: using LLMs as translators of the technical artefacts into natural-language, interactive and audience-specific explanations. LLMs can help overcome the last mile of XAI systems by making explanations more accessible, contextual and usable, particularly for non-expert stakeholders. On the negative side, LLMs may introduce hallucinations, misleading simplifications, ambiguous narratives and a gap between plausibility and faithfulness to the model’s inferential process. Such a gap could be dangerous; human users could struggle to detect errors in convincing explanations. In AI-supported decision-making processes, LLM-generated explanations can also increase over-trust in AI recommendations rather than support appropriate reliance.
Overcoming these scientific challenges has become a legal requirement under the emerging European regulatory landscape, particularly the AI Act. The AI Act requires transparency for high-risk AI-based systems and the provision of information that enables deployers to interpret outputs, employers to perform effective human oversight in specific cases, and affected persons to exercise their right to clear and meaningful explanations of AI recommendations. Besides the AI Act, the GDPR demands a right to explanation and the people’s privacy, thus forbidding explanations that reveal sensitive data. XAI explanations must also take into account the risks of disclosing patent-protected information or trade secrets and jeopardising a company’s business.
Conclusions will outline a research agenda for LLM-mediated XAI explanations that must account for legal constraints and be adaptable to the cognitive processes and knowledge background of the various stakeholders of AI-based systems. Additionally, it is vital to develop evaluation frameworks that jointly measure fidelity, understandability, usefulness, privacy and regulatory compliance in an objective, universal manner.
Short Bio
Giulia Vilone is a Senior ML Engineer at Analog Devices International. Following her PhD on XAI and argumentation, she investigates real-world applications of XAI in human–machine interaction and the integration of technical and legal perspectives on explainability. Her research focuses on Edge AI, Computer Vision, Argumentation, Explainable AI, and Legal Compliance.