Azure AI Red Teaming Agent + Arize Tracing

This notebook walks through how to trace Azure AI red teaming agent scans

Prerequisites:

1. Arize AX account (Sign up for free)
2. Azure AI foundry account and project created (Sign up here)

Information on AI Red Teaming agent: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/ai-red-teaming-agent?source=recommendations

Prerequisites

• Python 3.10-3.12
• Azure AI Foundry project
• OpenAI API key (or Azure OpenAI endpoint)
• Required packages installed

1. Create the Azure AI Red Teaming Agent

You can instantiate the AI Red Teaming agent with your Azure AI Project and Azure Credentials.

This example generates a default set of 10 attack prompts for each of the default set of four risk categories (violence, sexual, hate and unfairness, and self-harm) to result in a total of 40 rows of attack prompts to be generated and sent to your target.

https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#results-from-your-automated-scans

Optionally, you can specify which risk categories of content risks you want to cover with risk_categories parameter and define the number of prompts covering each risk category with num_objectives parameter.

2. Trace the Agent

3. Create a target for the red teaming agent

4. Run the red teaming scan

When the scan is finished, you can specify an output_path to capture a JSON file that represents a scorecard of your results for using in your own reporting tool or compliance platform.

Additional Examples

Available Attack Strategies

To discover available attack strategies in your environment:

5. Now view the traces from your red teaming scans in Arize!