11 Oct 2023

Assurance through Adversarial Attacks

This blog explores adversarial techniques to explain their value in detecting hidden vulnerabilities. Adversarial methods offer insight into strengthening AI against potential threats, safeguarding its use in critical sectors and underpinning AI trustworthiness for end users.

What is an Adversarial Attack?

Adversarial attacks in AI involve intentionally creating inputs that cause AI models to make mistakes. These attacks are not random; they are carefully crafted to exploit specific vulnerabilities in an AI system. Adversarial inputs in Computer Vision can often go undetected because they are normally not visible to the human eye; in language, adversarial input looks like gibberish. By understanding and simulating these attacks, we can gain invaluable insights into how AI models might fail in the real world, particularly under malicious conditions.


Why Use Adversarial Attacks for AI Assurance?

  1. Identifying Vulnerabilities: Adversarial attacks help uncover hidden weaknesses in AI models, especially those that might not be evident during standard testing procedures.

  2. Enhancing Robustness: By integrating these attacks into the training process, AI models can be made more robust against potential manipulations and unforeseen scenarios.

  3. Ensuring Reliability: For AI systems, especially in high-stakes domains like Defense or healthcare, reliability is paramount. Adversarial testing provides a deeper level of assurance.

  4. Regulatory Compliance: With increasing scrutiny and regulatory demands around AI, adversarial testing helps ensure compliance with standards and guidelines, particularly in terms of fairness and transparency.

Mechanism and Types of Attacks


The core mechanism of an adversarial attack involves leveraging the way AI models process input data. AI models, especially deep learning models, make decisions based on intricate patterns they learn from data. Adversarial attacks subtly alter these patterns in a way that's imperceptible to humans but highly significant for the model.

Types of Attacks:

    • Evasion Attacks: These occur during the inference phase, where slight, often imperceptible, alterations to input data lead the model to make incorrect predictions or classifications.

    • Poisoning Attacks: These happen during the training phase, where the training data is subtly manipulated to corrupt the learning process.

    • Model Inversion Attacks: Aimed at reconstructing sensitive or private data used in training the model, thereby compromising data privacy.

Why Are Adversarial Attacks Effective?

  1. High Dimensionality: AI models, particularly deep neural networks, operate in high-dimensional spaces. Small changes in high-dimensional input can lead to vastly different outputs, which attackers exploit.

  2. Linear Behaviour: Many AI models, despite their complexity, exhibit linear behaviour in high-dimensional spaces. This means that small changes in input can linearly accumulate, leading to significant errors.

  3. Transferability: An adversarial example created for one model often misleads other models, even if they have different architectures or were trained on different datasets. This transferability makes attacks more scalable and dangerous.

  4. Lack of Robustness: Traditional AI models are optimized for average-case performance (e.g., accuracy) and not for worst-case scenarios, making them vulnerable to these attacks.
Pig Perturbation

Adversarial Attacks in Practice

  • Image Recognition: Slight alterations to pixels in an image can make an AI incorrectly identify objects.

  • Natural Language Processing (NLP): Subtle changes in text input can alter the sentiment or meaning interpreted by the AI.

  • Reinforcement Learning: Altering the environment or reward signals can lead to incorrect or suboptimal policy development.

Mitigation Strategies

  1. Adversarial Training: Involves training the model with adversarial examples to improve its resilience.

  2. Model Regularisation: Techniques like dropout or weight decay can make the model less sensitive to small perturbations.

  3. Input Pre-processing: Applying transformations to input data can sometimes neutralise the effect of adversarial modifications.

  4. Robust Architecture Design: Developing models with inherent resistance to adversarial attacks.

Adversarial attacks highlight a critical aspect of AI security and reliability. Understanding and mitigating these attacks are fundamental to developing robust and trustworthy AI systems. As AI continues to integrate into various sectors, the ability to defend against adversarial attacks will become a key differentiator in the efficacy and safety of AI applications.