How would you threat-model a generative AI feature?

Maps assets, trust boundaries, untrusted content, model influence, actions, and abuse cases before selecting controls.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I start with the user goal, data, identities, model, retrieval sources, tools, outputs, and side effects, then draw the trust boundaries between them. I mark every place untrusted content can influence the model and every place model output can influence code, a person, or an external system. For each path I ask what could expose data, corrupt a decision, misuse authority, exhaust resources, or mislead a user.

Full answer below

Practice focus

AI threat modeling, trust boundaries, attack paths, impact analysis, and proportional controls.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Overview
An AI threat model extends ordinary application security to cover probabilistic behavior and model-mediated data flow.
I start with the user goal, data, identities, model, retrieval sources, tools, outputs, and side effects, then draw the trust boundaries between them. I mark every place untrusted content can influence the model and every place model output can influence code, a person, or an external system. For each path I ask what could expose data, corrupt a decision, misuse authority, exhaust resources, or mislead a user.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
The threat model distinguishes direct user input from indirect content such as documents, email, web pages, tool results, memory, and third-party prompts. It records the authority available at each step, the consequence of a wrong action, and which controls are deterministic versus probabilistic. I prioritize concrete abuse cases by impact and reachability, then test whether identity, authorization, validation, isolation, budgets, and approval still contain the damage when a model safeguard fails.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Listing OWASP categories without mapping the actual system.
Threat-modeling only the public prompt box.
Assuming a model safety layer contains every downstream impact.

How would you threat-model a generative AI feature?

Practice focus

Answer path

Overview

Review signals

Follow-up prompts

Continue practice

Next in path

Tech stacks

Skill areas

Human editorial direction

Was this drill useful?

Found this answer useful?