Algorithmic robustness and explainability more relevant than ever
by Raffaella Aghemo
A document issued in 2020 by the European Commission, entitled ‘Robustness and Explainability of Artificial Intelligence’, by Ronan Hamon, Henrik Junklewitz, and Ignacio Sanchez offered an overview, aimed at ‘strengthening’ the oversight of algorithmic systems, with some primary objectives:
provide a policy-oriented description of current perspectives on AI and its implications in society;
provide an objective view of the current AI landscape, focusing on robustness and explainability aspects. This includes a technical discussion of the current risks associated with artificial intelligence in terms of cybersecurity, security and data protection;
present scientific solutions currently under active development, in the AI community, to mitigate these risks;
present a list of recommendations for the attention of policymakers to establish a set of standardisation and certification tools for AI systems.
THREE themes need to be cultivated and implemented for proper algorithmic system functionality:
1. Model transparency in the lifecycle of the algorithmic system: this refers to the documentation of the AI processing chain, including the technical principles and description of the data used to design the model. This also includes elements relating to the interpretability and explainability of the models;
2. Reliability of models and identification of vulnerabilities: this concerns the ability of models to avoid failure or malfunctioning, either due to edge cases or due to malicious intentions;
3. Data protection in models and proper governance: the security of data used in artificial intelligence models must be preserved. In the case of sensitive data, e.g. personal data, risks should be managed by applying appropriate organisational and technical controls.
The steps for a proper roadmap are:
1. development of a methodology aimed at assessing the impacts, of AI systems, on society, built on the DPIAs, introduced in the GDPR, which would provide a risk assessment for users and organisations;
2. introduction of standardised methodologies to assess the robustness of AI models, in particular, to determine their scope of action with respect to the data being trained, the type of mathematical model or the context of use
3. raising awareness among AI practitioners by publishing best practices on known vulnerabilities of AI models and technical solutions to address them;
4. promotion of transparency in the design of machine learning models, emphasising the need for an approach to explainability for system designs that could lead to negative impacts on users’ fundamental rights.
The fear of ‘non — control’ of AI is pressing.
AI systems fall under the scope of the Cybersecurity Act, proposed by the European Commission in 2017, and which introduced an EU-wide cybersecurity certification framework for digital products, services and processes. In the same way that cybersecurity techniques have evolved over time to adapt to new devices and practices, the need to protect and improve the reliability of AI systems has become prevalent, as their opacity exposes them to strong flaws, both intentional and unintentional.
Data is a key aspect in artificial intelligence techniques and the widespread use of data management systems, which has been enabled by the digitisation of services, has led to the emergence of useful principles for properly storing and managing this data. The concept of Data Governance seems to describe the set of practices, procedures, standards and rules, to ensure that the data collected by organisations are well managed (legality, accountability, etc.). The GDPR takes an active stance on this sensitive issue in recitals 71, Art. 4 and Art. 13 and 14, and 22.
Explainability and interpretability become the two pillars underpinning the new algorithmic path, based on seven general key requirements:
1. human agency and oversight: protection of fundamental rights, interaction between humans and Artificial Intelligence Systems;
2. technical robustness and security: resilience, accuracy, reliability of AI systems;
3. privacy and data governance: data protection, data management, privacy rights;
4. transparency: traceability, explainability, communication;
5. diversity, non-discrimination and fairness: accessibility, lawfulness;
6. environmental and social well-being: sustainability, social and societal impact;
7. accountability: verifiability, reporting, responsibility.
Machine learning is still the most common method of algorithm development. Machine learning consists of a set of mathematical techniques at the intersection of algorithm, statistical learning and optimisation theory, which aim to extract information from a set of examples (images, sensor recordings, text, etc.).
Machine learning can be roughly divided into three paradigms: in the supervised setting, each example includes a label, which can be categorical or scalar: for a given input, the model aims to predict the right label. In the unsupervised (or self-supervised) setting, no label is provided; the model aims to learn a new representation that groups examples according to their similarity. The last category groups reinforcement learning techniques, in which an agent is trained to perform a complex sequence of actions autonomously in a complex environment in order to maximise a reward function. In today’s machine learning applications, supervised techniques are predominant and have been applied mainly to decision-making systems.
Three levels of transparency can be found in AI systems:
implementation, which is usually referred to as a white box system, as it is quite verifiable;
specification, which has led to a certain degree of implementation and here too there is a fair degree of transparency;
interpretability: this corresponds to understanding the mechanisms underlying the model (e.g. the logical principles behind data processing, the reason behind an output, etc.), which is not achieved in current algorithmic systems.
In the aforementioned work, however, it is stated that most real-world artificial intelligence systems used in production are not transparent, because the implementation and specifications are not publicly available (e.g. due to intellectual property issues) and/or because the model is too complicated and no simple interpretation of the results can be made.
Depending on the nature of the model, two paths are followed for interpretable AI:
- Post-hoc interpretability, which is used to extract explanations from the black box model that are not inherently interpretable, such as high-dimensional models (e.g. deep learning models) that include a huge number of parameters. Interpretation is performed by reverse engineering, selectively interrogating the model to reveal some of its properties.
- Interpretable models: these models are fully or partially designed to provide reliable and easy-to-understand explanations of the prediction they produce from the outset, but this is not always a feasible approach.
It must also be said that making these models more interpretable in turn seems almost inevitably to result in a loss of performance of the AI system.
On the reliability of the algorithmic system, several approaches influence it, but essentially two:
- the first is the importance of external validation, independent of the training phase, in order to limit overfitting, which occurs when the model does not learn any meaningful patterns, but only stores the input data, considerably reducing the generalisation power of the model;
- the second is the risk of spectrum bias, which refers to the presence of examples in the dataset that do not reflect the diversity and complexity of situations, i.e. the spectrum of examples does not reflect the actual spectrum.
There are also a number of vulnerabilities due to:
- data poisoning
- creation of contradictory examples
- model flaws
which can be mitigated through:
- data sanitisation
- robust learning (statistical techniques such as regularisation and robust inference)
- extensive testing
- formal verification (through two parameters, that of Unsatisfiability: checking whether for a given input, obtaining a certain output is (not) feasible; and that of Robustness: checking whether adding noise to a given input changes its output).
The right goal of protecting data sets, which are used for algorithmic training, is also recalled: ‘Two risks are considered:
1. sensitive data is directly accessible to an untrustworthy actor, due to malicious intent or vulnerabilities in the data infrastructure;
2. sensitive data may leak from the model after training. The second risk refers to the storage capacity of machine learning models. Models are in fact trained to extract patterns from data and usually store them as model parameters, e.g. in the form of weights. The purpose of the training phase is to make the model store generalisable patterns, which will be relevant for data that are not present in the training dataset. However, these patterns can be very similar to the training data and can be retrieved from the adversaries.”
Different solutions are proposed, which go through so-called differential privacy, which consists of adding noise to the training data to reduce the influence of each individual sample on the output; and through so-called distributed and federated learning, two different situations in which the learning of the model is not carried out by a single actor, but instead unpacked by a multitude of different subjects that may or may not be connected to each other. Or by training on encrypted data, which averts the risk of sensitive data being passed on to third parties in charge of implementing the algorithmic learning, albeit at great expense.
Hence the need for adequate system certifications, which also configure appropriate DPIAs, and which make use of TESTS, with the following objectives
- Identification of system vulnerabilities and potential impacts. These impacts may be on citizens and organisations, economically, socially and ecologically.
- Consideration of the scope of the data and potential borderline cases where the system may fail.
- Demonstration of performance on various data sets, including external data sets, which were not used in the training phase.
Standardisation systems should also be used to prevent failure risks and provide a taxonomy of known vulnerabilities.
Hence, in conclusion, the need for systematic transparency, accompanied by comprehensible explainability.
Therefore, let us not be fooled, or rather, ‘enchanted’ by solutions deemed to be ‘magic’ such as CHATGPT3, which is just an evolved model of what is called a ‘stochastic parrot’, an LLM large language model, a statistical language model, capable of assembling sentences and making us believe that the system is deeper and more ‘intelligent’ than it actually is! This term, which came out of an article ‘On the Dangers of Stochastic Parrots’, by Tminit Gebru and Margareth Mitchell, and two linguists from the University of Washington, Emily Bender and Angelina McMillan-Major, explains how LLMs, which are based on learning from large databases of texts, mainly taken from the internet, work. These systems have no understanding of the meaning of the words or expressions they generate, but identify recurring verbal patterns in the data and ‘repeat’ them. Never was the statement ‘so much data but so few views’ truer!
All Rights Reserved
Raffaella Aghemo, Lawyer
Subscribe to DDIntel Here.
Visit our website here: https://www.datadriveninvestor.com
Join our network here: https://datadriveninvestor.com/collaborate
