AI systems are at risk of being trained to produce specific, bad responses. In this topic, we examine this risk and what you can do about it.
Data Breach Risks > Data Poisoning
Data poisoning is a type of attack where bad information is fed into a system with the goal of altering the system’s behaviour. This type of attack has been used to reroute network traffic through an attacker’s server [34, 35] and influence how search engines rank web pages [36].
In the past, the effect of the bad data in a system was noticeable, fairly localized and easy to remove to restore proper operation.
How is AI changing things?
A data poisoning attack targets the integrity of an AI model and can have catastrophic and sometimes invisible effects. By providing bad information that leads the AI model to be biased, inaccurate or by creating a backdoor, the entire model is then compromised. It’s difficult to detect the poisoned information because of the enormous amount of information used to train models and the only mitigation for this type of attack is to retrain the model on new, unpoisoned datasets.
Here’s how this attack works [37]:
- A malicious actor identifies the model they want to attack and finds the sources of data that are used to train it.
- The malicious actor then finds a way to inject “poisoned” information into that data. One typical way is to create publicly facing information that includes an attack
- As web crawlers collect data for inclusion in open source datasets, they come across the malicious actor’s information and ingest it
- That poisoned dataset is used to train the model, which bakes the vulnerability into the AI
What can you do about it?
The key to protecting yourself from the risk of data poisoning is to know what information your AI model has been trained on. The techniques being used to ensure no malicious data is consumed during training include:
- Traceability. Ensures that the source of every piece of data is known
- Authority Scores. Prioritizes legitimate data over public data
- Supply Chain Audits. Ensures that vendors providing data have rigorous standards for evaluating their training information
- Perplexity Monitoring. Perplexity indicates when an irregular kind of data has been identified
- Examination. Involves examining data for instructions that could be malicious
Once the system has been trained operational oversight is required to protect the system and its resources from malicious actors. These include:
- Data Detection and Response (DDR) where real-time prompts are examined for intent.
- AI Firewalls, which protect the tool from poisoned documents being added by any actor.
- Adversarial Red Teams, who regularly test the system for backdoors.
In Summary: Prevent your AI from getting access to malicious data to mitigate the risk of data poisoning.
Hi
Support this Work
You can purchase the book on Amazon:
You can listen to the podcasts for free through Patreon, Apple Podcasts or YouTube Music: