Securing AI Systems — Defensive Strategies

In the first paper on “Securing AI Systems — Risks in AI based Solutions” we got an overview of the Risk landscape in the solutions built using AI technologies. The risks were analyzed from both the security and safety aspects of AI usage. Also, the impact arising from regulatory violations were studied including the importance of getting aligned with AI principles. We took a deeper dive into the Adversarial Machine Learning (AML) Attacks in the second paper (Securing AI Systems — Researching on Security Threats) by emphasizing on the importance of doing research in enumerating the security threats and ways to proactively discover them in the Model internals. Although we discussed about the role of research in the Adversarial AI space, it is implied that performing continuous research with other segments of risks (eg: AI Safety or AI Principles) are helpful to understand the advancements happening there. We will continue the same rhythm of researching as we deal with the defensive strategies that are applied to safeguard AI solutions.

Analyzing Security Incidents

Strong processes are required to effectively implement the defensive controls for the security risks identified. This means using a combination of process engineering tools to identify risks and successfully mitigate them. One of the ways to identify these risks is by using the concept of FMEA (Failure Mode and Effect Analysis). FMEA is a process analysis tool to determine the components that are failing in the system, including their causes and effects. This in turn helps mitigate the risks by applying the right defenses. Applying this FMEA principle also helps in analyzing the security incidents in AI solutions to understand their failure modes.

Apparently, Microsoft and Harvard University have worked on the Failure Modes in Machine Learning. Their research focusses on how AI based solutions fail, whether by the hand of an adversary or due to the inherent design of a system. Based on their study, are two types of failures:

1. Intentional failures where the failure is caused by an active adversary attempting to subvert the system to attain their goals — either to misclassify the result, infer private training data, or to steal the underlying algorithm.
2. Unintentional failures where the failure is because an ML system produces a formally correct but completely unsafe outcome.

As you could notice from their work, many of the adversarial attacks that we have briefly discussed earlier including the perturbation attack, model poisoning, model stealing as well exploiting software dependencies are a result of the attacker’s ability to misalign the model’s objectives. All these falls under the Intentional failure type. It is highly recommend going through their document as it details the different attributes of Intentional Failures including the attack scenario, examples, the compromise impacting the core tenets of security, the attacker’s methods and numerous other research papers that are being referred.

Interestingly, as we dig deeper on the Failure modes in AI, a research firm named BNH.AI that specializes in the privacy, fairness, security, and transparency of AI and data has completed a thorough analysis of AI incidents. They have surveyed about 169 publicly reported AI incidents and the inferences are that, 95% of these incidents were unintentional failures which includes algorithmic discrimination, lack of transparency and accountability, privacy violations and others. This means only about 5% of AI incidents were caused by adversarial attacks or the Intentional failure type. The figure below gives a snapshot of these issues and is taken from the blog post on Bias in AI where the study made by BNH.AI has been summarized including the impact and implications of these AI incidents.

Scope of Defensive Controls

As we discuss about the defensive strategies, we will keep our focus only on the risks emerging out of these Intentional failure categories and the defenses to be planned to prevent them. This is because

1. As practitioners building AI solutions, it is important to know our enemies and the weaknesses existing in our system that could be exploited by these adversaries. Hence it is essential to build solutions with the right countermeasures.
2. Secondly, the variations in the attack patterns, innovative methods involved in crafting adversarial examples pose a huge challenge and we need to constantly defend the boundaries from potential threats caused by malicious actors.
3. Importantly, it is required that Solution providers should do their due diligence and be continuously compliant with international laws and regulations by being resilient to security attacks on the target AI systems.

Having said all these, as part of the concluding section, we will also brief the best practices that needs to be performed including the industry initiatives that are ongoing to prevent against Unintentional failures in AI systems. So, let’s look at some of the control mechanisms that help build the integrity in our AI models.

Adversarial Robustness

Adversarial Robustness is all about building the machine learning models with security, privacy and regulatory principles that will ensure the integrity of the ML models and the resiliency of complete AI solutions. So, essentially it deals with the implementation of countermeasures against all the Adversarial Machine Learning (AML) attacks.

So, what are those countermeasures? “A picture is worth a thousand words” — The figure below portrays the taxonomy of the adversarial defenses in machine learning models. Based on the paper published in IEEE Reviews, the team of researchers have categorized Adversarial Robustness into three classes (as seen in the figure).

AML Defenses — Implementation Experience

As much as we are interested in learning these remediation measures, we should also understand the practical challenges faced by data scientists in applying these remedies and the evaluation techniques for applying the right defenses. To begin with, we will briefly discuss the various taxonomies of countermeasures and how they are being approached.

A) Approaching Countermeasures

In the figure above, we observe three classes of defense methods that are designed to either modify the model/data on its parameters and its features or defense methods that focus on having additional models integrated to secure the mainstream models. This is one perspective of how the defense controls are discussed but as we leverage advanced algorithms like Deep Reinforcement Learning (DRL) that are being used by a lot of AI systems to achieve high performance, we need to think about adversarial defenses that are specific to attacks against the DRL models. The work published by Journal of IEEE Transactions on Artificial Intelligence has both the Adversarial attacks and taxonomy of defenses for DRL models (figure below) discussed in detail, that is worth looking at.

Finally, in addition to implementing the security controls to the models, going by the principles of Adversarial Robustness, we need to invest in Privacy Preserving techniques as well. These are nicely explained in the paper on Privacy and Security Issues in Deep Learning: A Survey. The techniques include Differential Privacy, Homomorphic Encryption, Secure Multi-party Computation and Trusted Execution Environment.

B) Challenges and Lessons Learnt

It is great to see the numerous approaches published for adversarial defenses focusing on the realms of security, privacy, and advanced algorithms. However, not all these approaches are feasible and have some challenges that we should know beforehand. For example, as mentioned in Empirical defenses for adversarial examples, Adversarial training based defenses (where the model is re-trained by augmenting adversarial examples in the training set) is observed to be very slow when implemented. Also, research shows that accuracy of robust models implementing adversarial training drops substantially than that of non-robust models.

Few other research papers have also summarized their experience on the adversarial countermeasures including the one on Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Survey. Among the many lessons learnt, the authors highlight that it is generally hard to apply a defense method that is effective on one model or dataset, to other complex models or datasets. Hence, they infer that Scalability and transferability of the defense approach is a major concern. It is recommended to go over the papers cited above for more in depth knowledge.

C) Evaluating Recommendations

So, one question that every practitioner has while implementing these Adversarial countermeasures are “how to apply the right one?” with many unknown challenges in them. A joint effort made by Google Brain, MIT and University of Tübingen on their paper On Evaluating Adversarial Robustness, provides practical advice for evaluating defenses that are intended to be robust to adversarial examples. Some of the principles to perform these defense evaluations include:

1. Implement threat model specifically centered around the attacker centric approach so that the defense controls planned are most effective.
2. Evaluate by applying diverse set of attacks. This includes testing the robustness to random noise, evaluating the attack hyperparameters, etc.
3. Perform ablation analysis by removing a combination of defense components and verifying that the attack succeeds on a similar but undefended model. This clarifies the point of combining multiple defense strategies which would help in model robustness.

The paper has the complete list of evaluation recommendations that are being categorized as shortcomings, that need to be avoided when evaluating defenses.

In the last few sections discussing the countermeasures, we have been focusing on methods that prevent Adversarial attacks on the models. But, as we know, an AI based solution also has the application components built around the model to be leveraged by the business or integrated with another system. Hence, it is equally important to plan and implement the application layer security on top of the model. Application Security is a well-known domain to majority of us, so in this paper, we only wanted to highlight the importance of the subject and the same kind of analysis must be done to work on those countermeasures.

Major Roles in AI Programs

BSA | The Software Alliance which is an advocate for the global software industry, have discussed in their publication about the distinction between AI Developers and AI Deployers. According to their analysis, organizations building AI systems can act as AI Developer and/or AI Deployer. It is essential to distinguish between these two roles so that legal frameworks could accurately assign obligations to the company based on its role in the AI ecosystem. As a result, companies are better able to fulfill the legal requirements and better protect consumers. The figure below discusses the two different roles (and the corresponding examples) that companies take in building AI systems. Going by this idea, we will define the security measures to be considered and accountabilities of these AI roles (AI Developer and AI Deployer).

The defense mechanisms described in the sections above are largely aligned with the AI Developers. This means security techniques and principles that need to be implemented by

a) Data Scientists working on data preparation and exploration, and internal and external data acquisitions

b) AI/ML Scientists tasked with building AI models and evaluating them against different metrics.

Additionally, there are AI Deployers that use these AI systems (i.e.) companies that work towards productionalizing the model by integrating them with the CI pipelines to operationalize, monitor, and evaluate them for its performance. These are done by AI/ML Engineers. We haven’t looked at the defense strategies specific to this aspect yet and we will examine them now, including the security activities that they need to put their attention on.

Security Accountabilities & its Importance for Companies using AI Systems

So, in the remainder of this paper, we will focus on the defensive strategies that AI Deployers (the companies that are using the AI Models) must follow. There are strong motives and valuable takeaways that are imperative for AI Deployers to implement the defense strategies. Sharing a few of those, based on the evidence observed from the research artifacts and the rapid developments happening in the AI community:

A) Shared Accountability Model

We saw in the previous section, that companies playing both AI Developer and AI Deployer roles have legal frameworks assigned to them and they must satisfy those requirements. This means AI Developers practice Due Care to build-in security mechanisms in the models and the companies that are AI Deployers must do the Due Diligence by properly investigating all possible weaknesses on the AI models including the risk mitigation before offering the service to clients.

All these efforts prevent security breaches from happening and prepares us to recover the AI systems from possible failures. Overall, this synergy of working together by the two AI roles helps build Trustworthy AI Systems. Additionally, NIST, a leading standard recommends accountability to be shared across the actors in the AI Lifecycle and requires every actor in the AI Lifecycle to play their part to ensure the security of the complete AI system to be very effective. As you could see from the figure below, many of these actors are part of the AI Deployment.

B) AI Assurance Ecosystem

In addition to understanding how the accountability in AI Lifecycle has got to be planned, the NIST’s Risk Management Framework has many other helpful information, that is worth referring to. Their AI RMF document also highlights the importance of Test, Evaluation, Verification, and Validation (TEVV) processes throughout an AI lifecycle and generalizes the operational context of an AI system. The TEVV tasks are given for every phase of the AI lifecycle.

Although these tasks are collaboratively carried out by many actors (as given in the figure), they certainly need the help of AI Assurance tools to operate efficiently and achieve automation. There are a few promising AI Assurance solutions, but many of the commercial vendors are just warming up to tackle the very many attacks and failures including the threats emerging from the Generative AI models. This results in companies that are using the AI models (AI Deployers) to gear up and come up with their homegrown tools/solutions/strategies to assist the AI actors in the TEVV processes described above.

C) The Evolving Risk Landscape

With the advent of the Generative AI Models, more and more companies are leveraging them for developing their business. Enterprise applications benefit from the integration with Large Language Models (LLM) and Image-generation models by improving performance and enhancing the user experience. Apparently, a venture capital firm a16z was one among the first to share a high-level view of the Generative AI technology stack in their publication. The figure below clearly shows the rise of Model providers and applications built on top of them.

So, we could infer that the AI Deployers that are consuming the AI Models as a Service, see extraordinary value in them but there are numerous challenges here. In addition to the potential threats from the AI Models and the software built around it, there are safety hazards and intellectual property infringement that need to be addressed as well. With all these unknown challenges and dynamic attack surface, companies using the AI Models must take accountability and work towards the implementation of defensive strategies.

Establishing Security competencies for AI Programs

We discussed many compelling reasons for AI Deployers to invest their efforts on the defensive controls. But determining the countermeasures may not be as straight forward as we guess. Just like the companies that built these models, the companies that are leveraging them have several different elements to consider for the defenses implementation to be successful. This is because the project using the AI Model might have varying deployment scenarios, uncertainties with technological and platform integration, budget constraints at program level, changing dynamics of the vendor/stakeholder relationship and many other businesses level pain points that impacts the productiveness of the defensive controls.

Considering these challenges, we will delve more into the strategic defenses as opposed to the objective-typed controls, so it is valuable for all of us under all circumstances. Based on that point, we will focus on the security competencies that are essential for AI Programs. It has been proven that having these competencies will evaluate the effectiveness of the defenses built into the model by the AI Developers. The upcoming sections are centered around the takeaways for AI Deployers.

a) AI Red Teaming

“The best defense is a good offense” — Hiring Red-teaming experts helps enterprise strengthen the resiliency of the overall AI solutions by realizing the effect of Adversarial attacks that are performed using real-world simulations. As the adoption of GenAI and LLMs are on the rise, red-teaming efforts become vital to evaluate the safety of the models for the presence of harmful content and unethical output, in addition to the security and privacy aspects. Anthropic, an AI start up firm focusing on AI Safety and reliability in their research work have discussed the types of harms discovered by red teaming. The picture above is a visualization of the red team attacks that shows the clusters of attack types. When it comes to the planning and procedures involved, I would recommend going through the resource from Microsoft that briefs the approach to be taken in red teaming an LLM. Although our example focused on LLMs and Generative Models, red teaming activities would benefit solutions developed and deployed using Discriminative models as well.

b) Threat Modeling

Another favorite subject for many security professionals who come from the traditional software security world. Threat Modeling is extremely valuable for companies that buy AI-ML solutions and deploy them. The snapshot below gives an overview of the methods followed to Threat Model an AI system to get the attacker’s point of view. But as we attempt to address the potential threats in the complete system, we need to approach threat modeling from multiple dimensions. This includes the threats emerging from the design patterns followed during model operation scenarios, interoperability with existing systems, software technology stack, data ingestion methods, etc. Additionally, there are also threats arising out of process lapse, non-conformance aspects, regulatory violation, amongst others. Hence, it is essential to take a Combinatorial approach to Threat Modeling. If the principles are correctly applied, Threat Modeling activity adds a lot of value to AI Deployers including evaluating the defenses, prioritizing them, understanding the root cause of the threats, establishing the feedback loops with AI Developers, and importantly contributes to the maturity of the overall AI Lifecycle’s risk posture.

c) Compliance Audits

Compliance is a common practice that applies to all the roles across the AI Lifecycle. Specifically, from an AI Deployers’ standpoint, there are few aspects of compliance that are of foremost value. The first among them is the Data Privacy Laws — As AI based applications require the users to input the PII and PHI data, the auditors must investigate the processing of personal data and weather it is going to be shared with external entities. This could be evaluated by doing a Data Protection Impact Assessment (DPIA) and there are many resources, guidelines, and templates that are recommended by IAPP and GDPR to get more insights. Secondly, there are AI Standards, where we are getting to witness countries regulating the use of AI. The AI Act, proposed by the European commission and the AI and Data Act from the Government of Canada are some of the examples. There are also Government Protocols and AI Regulations for Industry Verticals that require the AI Deployers to comply with the mandates given by the authorities. Hence, it is strongly advised to have a team of Internal auditors diligently assess the AI based solutions.

Continuous Compliance and Best Practices

We discussed many strategies, competencies, compliance measures, countermeasures that are specific to adversarial attacks in the previous sections (and there are numerous others too), but the effectiveness of defenses will be long-lasting only when performed continuously. This means doing a threat modeling for every milestone release, periodically auditing programs for regulatory adherence, and so on. All these efforts result in AI solutions being continuously compliant. One way of accomplishing this is by implementing the Principles of MLSecOps, where the security process gets embedded into the complete AI Lifecycle, thereby securing the data pipeline, models deployed, and the software stack built. Hence this practice is followed from the inception stages till the sunsetting and performed repeatedly for every release.

In addition to the processes and practices, Security Champions acknowledge the value of investing their time in shaping up the culture of the organization to successfully accomplish their objectives by educating and empowering the product teams. This would mean rolling out training programs and awareness sessions on the secure use of technologies to prevent Insiders risks from happening. Based on our learnings from the recent security incidents, where GenAI based code generation and chat applications’ improper use have resulted in sensitive data being leaked out. Hence alongside technical controls, administrative controls need to be carefully planned and executed.

Industry Initiatives and Ongoing Efforts

Unlike the initial days of software development, where reliability, trust, and regulations were an afterthought, the AI community has adopted and prioritized Trustworthy AI by embedding security and privacy principles in the AI model development efforts at an early stage. Today, we witness a lot of entities highlighting the benefits of Trustworthy AI and many organizations are in the process of diligently implementing its key pillars namely Security, Privacy Accountability, Transparency, Explainability and others. The Linux Foundation has done an excellent job of capturing the Principles of Trusted AI. There are other organizations that have done a fair job of defining the execution strategies of these AI Principles. On a similar mission to help practitioners build Responsible AI systems is the Responsible AI Institute that has published many artifacts including the how-to’s that will be very valuable in building the AI solutions with the appropriate laws, regulations, and best practices.

Conclusion

We looked at many strategies and approaches, but where do we get started? As quoted by the well-known American writer, Mark Twain, “The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks and starting on the first one.” Likewise, we approach the defenses to AI Systems by breaking up the complex strategies, performing prototypes to diligently analyze the working patterns, and that in turn will help us determine the defensive measures lasting effect.

Throughout the series, we noticed one attribute contributing in a significant way in guiding the AI Security practice — Research and Development. The R&D community has played a huge role in building offensive and defensive principles of Securing AI system. A lot of research papers were referenced in all the three articles. It is highly recommended for every AI practitioner to invest more time and resources in researching on the latest trends, security attacks, defensive tools and importantly keeping a constant watch on the regulatory updates. All these helps them to be better prepared for the future and always manage uncertainties, to build safe and secure AI Solutions.

Subscribe to DDIntel Here.

Visit our website here: https://www.datadriveninvestor.com

Join our network here: https://datadriveninvestor.com/collaborate