Observational evidence normalization

# Observational evidence normalization

## Goal

Combine the authorized no-contact research streams into one atomic, deduplicated, source-aware evidence ledger without flattening different evidence types into interchangeable “feedback.”

This prompt performs local synthesis only. Do not browse, collect new sources, contact anyone, generate synthetic evidence, or recommend product work.

## Required prior artifacts

Read the current:

* `docs/product-intelligence/product-job-baseline.*`
* `docs/product-intelligence/research-policy.*`
* `docs/product-intelligence/source-map.*`
* `docs/product-intelligence/search-demand.*` when present
* `docs/product-intelligence/experience-signals.*` when present
* `docs/product-intelligence/alternatives-workarounds.*` when present
* `docs/product-intelligence/competitor-observations.*` when present

The baseline, policy, and source-map decisions must authorize continuation. Consume a collection stream only from its `PUBLISH...` or `PUBLISH NARROWLY` decision and inherit its exact boundary. Exclude a stream on `SKIP STREAM`; do not reinterpret it as negative evidence. A collection `HOLD` stops normalization and forces a return to the named policy or source-map boundary.

At least one stream must publish. One usable source class permits only `NORMALIZE NARROWLY`. Preserve prior `docs/product-intelligence/evidence-ledger.*` IDs and history.

## Evidence contract

Use exactly `OBSERVED`, `DERIVED`, `ASSUMED`, and `UNKNOWN`.

Keep these dimensions separate:

* evidence label
* source kind: `public_behavior`, `public_statement`, `official_claim`, or `owned_aggregate`
* directness to the claimed fact
* independence from other sources
* relevance to the current product boundary
* recency and sample coverage

Normalization changes format, not evidence strength. A public statement stays an observation of a statement. An official claim stays a claim. A search phrase stays an observed search-surface artifact.

## Step 1: Validate inherited contracts

Check that each consumed artifact:

* uses stable IDs and valid source references
* stays inside the authorized sources, dates, fields, and samples
* records observed-at timestamps and limitations
* distinguishes statements, behaviors, official claims, and derivations
* contains no unnecessary personal data
* preserves its decision boundary

Quarantine malformed, out-of-policy, uncited, or overclaimed entries. Do not repair them by inventing support.

## Step 2: Create the canonical atomic schema

Normalize every inherited item into a stable record containing:

* canonical observation ID and original stream/item ID
* source and query IDs
* source kind and evidence label
* actor, situation, job, outcome, and product-boundary relation
* raw observation or minimal excerpt reference
* normalized statement
* event or publication date and observed-at timestamp
* directness, relevance, recency, independence, and coverage fields
* assumptions, unknowns, limitations, and counterevidence links

Never fill a missing field from a different source unless the relationship is explicitly derived and cited.

## Step 3: Deduplicate at three levels

Identify:

1. exact duplicates: copied text or identical source artifacts
2. event duplicates: several sources describing the same underlying event, issue, or announcement
3. conceptual similarity: different observations about a similar job or failure

Count exact and event duplicates once for independence while retaining every citation. Conceptual similarity is not duplication; keep the items distinct and link them for later synthesis.

Do not assume anonymous sources are independent. Do not treat several surfaces owned by one organization as independent customer evidence.

## Step 4: Reconcile conflicts without erasing them

Create explicit conflict sets when sources disagree about behavior, consequence, prevalence, success, pricing, or current state.

For each conflict show all observation IDs, source types, dates, possible scope differences, and what remains `UNKNOWN`. Resolve only when an authoritative source and rule genuinely govern the field.

Positive and null evidence must remain available alongside complaints and failures.

## Step 5: Build coverage and bias registers

Summarize the ledger by stream, source class, actor, workflow, date, geography or language when available, and evidence directness. Use raw counts with explicit denominators.

Record platform, search-ranking, moderation, survivorship, complaint, recency, technical-user, competitor, and publication bias. Do not calculate a population prevalence estimate from convenience samples.

## Decision

Choose exactly one:

* `NORMALIZE`: multiple usable evidence streams form a traceable ledger
* `NORMALIZE NARROWLY`: a bounded ledger is usable only for named questions, actors, or source classes
* `HOLD`: contract failures, duplication, conflicts, or coverage gaps prevent responsible synthesis

Lead with:

> As of [cutoff], the canonical ledger contains [N] atomic observations from [N] usable streams and [N] independent source classes, with [N] duplicate groups, [N] conflicts, and coverage [limits], decision [NORMALIZE / NORMALIZE NARROWLY / HOLD].

## Required outputs

Create or update only:

### 1. `docs/product-intelligence/evidence-ledger.md`

Decision, inherited boundaries, validation report, canonical ledger summary, duplicate and conflict registers, coverage, bias, quarantined items, assumptions, unknowns, and sources.

### 2. `docs/product-intelligence/evidence-ledger.yaml`

`version`, `status`, `decision`, `evidence_cutoff`, `inherited_decisions`, `observations`, `duplicate_groups`, `conflicts`, `coverage`, `biases`, `quarantined_items`, `assumptions`, `unknowns`, `sources`, `next_step`.

### 3. `docs/product-intelligence/evidence-ledger-changelog.md`

Append only. Record version, cutoff, decision, streams consumed or excluded, IDs added or merged, conflicts changed, and reason.

## Boundaries

* Do not browse, collect new evidence, contact anyone, or generate synthetic records.
* Do not upgrade evidence during normalization.
* Do not infer independence, prevalence, or identity.
* Do not discard counterevidence, conflicts, excluded streams, or failed searches.
* Do not retain unnecessary personal data.
* Do not recommend opportunities or solutions.

## Done when

* Every canonical observation traces to an authorized original item and source.
* Exact, event, and conceptual relationships are distinct.
* Conflicts, counterevidence, coverage, and bias remain machine-readable.
* No synthetic content appears in the observational ledger.
* Markdown, YAML, and changelog agree.