Red Team and AI Security: A New Frontier in Cyber Defense pt1

Hello, fellow squirrels. Today, I will be opening the nutshell about Red Team and AI Security!

This post will focus more on the AI Security side since we need some base knowledge on what they are working with, just like squirrels need a tree to get to the acorns.

What is Red Team?

Red Team is a group of people who are security experts and/or testers that simulate real attacks on a company’s system to find flaws. Red teaming is the name for the process or practice of these simulated attacks.

Why is AI Security Different From Other Types of Security And Why Does it Matter?

AI security differs in three main ways.

One way is the AI’s dynamic factor, meaning the system is constantly changing as it runs. Traditional software is generally predictable: the same input will always give you the same output, no matter the context. AI systems are chance‑based systems that are also context‑dependent. This means an AI system’s behavior will change based on the phrasing, the order of messages, the emotional tone, and/or subtle token changes. You might never get the same output even if the inputs are the same. This means you can’t accurately predict how the model will act or respond. Failures in the safety protocols might only show up in certain contexts, making them hard to find. It also gives attackers something else to exploit: the behavior of the AI, not just bugs in the codebase. AI introduces a moving attack surface that is constantly evolving and being fine‑tuned.

The second way it’s different is that no two systems and AI models are exactly the same. You can have two AI systems trained with similar data and they can still behave completely differently. This can happen for many reasons: different architectures, variations in the training data, safety layers, context‑window behavior (how much data it can take in at once), and more. Two seeds from the same tree can sprout two different acorn trees. This means an attack might work on System A but won’t work on System B.

The third way it’s different is that it is a lot harder to identify security issues by reading the code. Though this applies to all coding, AI amplifies it. Many security problems only appear while the code is running, when certain components interact, or when users behave in unexpected ways. Looking through code cannot show you logic flaws, what happens when two or more processes try to access or change the same data (this is called a race condition), data‑flow problems, and many other issues. AI is also not purely code, so some problems and vulnerabilities can only be found through testing.

That’s all for today’s cyber-stuffed nutshell! So, remember in this digital forest, we the squirrels need to stay alert and protect our stash, while chipmunks try to sneak in and swipe it when we are not looking.

Cool thing to look at:

AI Security Training: Case Studies and Tools for Generative AI | Microsoft Learn

requie/AI-Red-Teaming-Guide: A comprehensive guide to adversarial testing and security evaluation of AI systems, helping organizations identify vulnerabilities before attackers exploit them.

https://mindgard.ai/blog/red-teaming-exercises-processes-examples

Here is a fun thing to try: Gandalf | Lakera – Test your AI hacking skills

Link to Aardvark Labs where there is an implementation with a security issue:

Step-by-Step Guide: Create Agents in AL Code