> ## Documentation Index
> Fetch the complete documentation index at: https://safety.flamy.net/llms.txt
> Use this file to discover all available pages before exploring further.

# How moderation works

> Defense in depth across chat input, generation, catalog filters, alerts, and human review.

# How moderation works

> Defense in depth across input, generation, catalog, alerts, and review.

Flamify's content moderation is built as
[**defense in depth**](https://en.wikipedia.org/wiki/Defense_in_depth_\(computing\)):
multiple independent layers, each tuned to a different class of signal. Layers
reinforce each other; no single layer is trusted alone.

## The layers

### 1. Input — chat message moderation

Every user chat message is checked **before** it is sent to the AI worker.

**Stage 1 — keyword / regex matching**

Active forbidden topics are matched against the message using word-boundary
keywords and regex patterns (typically 1–3 ms). Topics are admin-managed and
versioned in production.

**Stage 2 — LLM verification**

When Stage 1 hits and LLM verification is enabled for that topic, a dedicated
moderation model confirms whether the message is a genuine policy violation.
The verifier is instructed to distinguish real violations from normal adult
roleplay, BDSM language, and fictional framing. Per-topic confidence
thresholds reduce false positives on noisy categories.

**Enforcement actions**

| Action   | What happens                                                                                                                                                                   |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **STOP** | User message is saved; a fixed policy response is returned; the chat channel is **blocked**; the trust team is alerted (Slack).                                                |
| **HOLD** | User message is saved; a warning is returned; **no AI generation** runs for that turn; the channel stays open. High-confidence HOLDs may escalate to Slack with rate limiting. |

Critical categories (minors / CSAM signals, self-harm, incest, zoophilia,
weapons instructions, drug manufacture, extortion, grooming/manipulation,
extreme gore) default to **STOP**. Softer categories (therapy substitution,
eating-disorder promotion signals, IRL meeting requests, AI-identity
confusion) default to **HOLD**.

The same pipeline runs on message **edits**. Regenerate reuses the prior
user text and does not introduce new unchecked input.

### 2. Character and catalog gates

* Characters cannot be created with age under **18** (schema-enforced,
  `ge=18`).
* Public catalog visibility requires an approved / published status.
* Characters carry an explicit **`is_safe`** flag used by Safe Mode brand
  surfaces (see [Safe filters](/moderation/safe-filters)). The flag is set by
  operators for app-store / PG-13 distribution — it is not inferred solely
  from generative output.

### 3. Generation-time controls

**Chat → AI**

When a brand has Safe Mode enabled, AI requests are marked `is_safemode=true`
and routed with a Safe Mode chat configuration (model / parameter profile
intended for SFW surfaces).

**Media generation**

Under Safe Mode, the media pipeline:

* Strips NSFW tokens from free-text prompts (`nude`, `naked`, `nsfw`, …).
* Rejects explicit scene types.
* Prepends NSFW terms to the **negative** prompt (`nsfw, nudity, explicit, …`).

**Appearance / image negatives**

Character appearance prompts always include age blockers in the negative
prompt (`child`, `underage`, `teen`, …), independent of Safe Mode. The main
adult product intentionally allows adult nudity outside Safe Mode; age
blockers are never omitted.

### 4. Catalog and media distribution filters (Safe Mode)

Brand-aware Safe Mode (`APP_PALM`) filters discovery surfaces so that PG-13 /
app-store fronts only show characters marked `is_safe` and only mild media
tiers. Restricted media unlocks are blocked. Details:
[Safe filters](/moderation/safe-filters).

### 5. Alerts and human review

* Every STOP violation creates an alert log and notifies the trust channel.
* HOLD violations are logged; Slack noise is rate-limited and suppressed for
  expected adult-RP false-alarm categories where appropriate.
* Office / admin tools expose forbidden-topic CRUD and alert-log review so
  operators can tighten keywords, thresholds, and response templates without
  a code deploy.
* User and admin reports on media / characters trigger re-review against the
  **current** policy.

### 6. Community and operator reports

Users and operators can [report](/reporting/how-to-report) problematic media,
characters, or behaviour. Reports are prioritised by severity — suspected
minor-related content first.

## Why this shape

| Layer                  | Strength                                                 |
| ---------------------- | -------------------------------------------------------- |
| Keyword / regex        | Fast, deterministic, cheap at volume                     |
| LLM verifier           | Context-aware; reduces false positives on adult RP       |
| Schema / catalog gates | Prevents underage characters and unsafe distribution     |
| Safe Mode media policy | Keeps app-store surfaces SFW even if a prompt tries NSFW |
| Alerts + humans        | Judgement, pattern spotting, policy iteration            |

Classifiers handle volume. Humans handle ambiguity. Safe Mode handles
distribution. Together they cover more ground than any single control.

## What classifiers see vs what humans see

Automated classifiers process chat messages in-line because that is required
to enforce [Prohibited content](/policies/prohibited-content). Classifier
processing does not mean a human reads every private chat.

**Human reviewers** see content that has been:

1. Flagged by a STOP / high-priority HOLD path,
2. Reported by a user or operator, or
3. Escalated as part of a trust-team investigation.

Account data (email, payment information, IP) is gated and accessed only when
an investigation requires it.

## Fail behaviour

* If the moderation LLM is unreachable or rate-limited, the system may
  **fail open** on that stage (message allowed) to preserve availability —
  keyword Stage 1 still ran.
* Certain client/config errors on the verifier path fail **closed**
  (conservative block).
* Safe Mode media policy fails **closed** on explicit scenes (generation
  rejected).

We document this honestly: availability and safety are both goals; critical
STOP categories remain keyword-backed even when the LLM stage is degraded.

## What gets stopped most often

In rough order of severity priority (not necessarily volume):

1. Sexual content involving minors / grooming signals
2. Self-harm and suicide ideation that triggers crisis routing
3. Real-world weapons / explosives instructions
4. Incest and zoophilia framings
5. Extortion / blackmail language
6. Drug manufacture / sourcing instructions

Each of these maps to [Prohibited content](/policies/prohibited-content).