Prompt Engineering

Summary

Prompt engineering at this scale is primarily a failure-diagnosis discipline: the most expensive mistakes are prompts that appear to work in happy-path testing but silently fail on edge cases — broad keywords triggering false positives, AI ignoring explicit constraints, context fragmentation when assembling prompts from multiple files. The single most consequential architectural decision is whether to use one comprehensive prompt or split into category-specific prompts; LabelCheck tried both and the split approach failed due to AI context fragmentation, requiring a full rollback. Numeric constraints (word count, section minimums, character limits) reliably require both prompt enforcement and post-processing validation — the model will drift without the safety net. Decoupling extraction from synthesis, then caching extractions keyed to schema version and file mtime, eliminates re-extraction cost entirely during prompt iteration and is the highest-leverage infrastructure investment for any multi-stage LLM pipeline.

TL;DR

What we've learned
- Category-specific prompts looked like a 60-70% size win but were ultimately removed from LabelCheck because AI context fragmentation caused incomplete analysis; a single comprehensive prompt outperformed the split approach in practice.
- Numeric output constraints (word count, section counts, character limits) require both prompt instructions and post-processing validation — prompts alone drift, as ContentCommand discovered when a 1,500-word target produced 4,288 words.
- Extraction caching keyed to schema version + file mtime drops Haiku API calls to zero for prompt-only iterations in Meridian — the highest-ROI infrastructure change for iterative prompt development.
- Broad keywords in pattern matching cause false positives at production scale; "help," "now," and "today" are too common for intent detection and must be replaced with specific phrases.
- LLM-judged rubric evaluation (10 criteria, 0-10 scale) enables quantified prompt iteration; Meridian's judge call completed in 20.4 seconds and returned an 88/100 average on a 9-fragment legal topic.

External insights

No external sources ingested yet for this topic.

Common Failure Modes

1. Category-specific prompt assembly causes AI context fragmentation

In LabelCheck, category-specific prompts were built by assembling content from multiple markdown files at runtime. The approach showed a 60-70% prompt size reduction (30KB → 7-8KB per category) and 5-10 second performance gains in benchmarks. Then it was removed entirely.

The failure: when prompts are assembled from multiple files, the AI receives a fragmented context that causes incomplete analysis — sections reference concepts defined elsewhere in the original monolith but absent in the assembled version. The model doesn't error; it silently omits analysis.

The fix was a full rollback to a single comprehensive TypeScript prompt. This is a resolved contradiction in the evidence: the enhancement work ^[1] preceded the removal decision ^[2], and the removal was empirical, not theoretical.

Lesson: Benchmark prompt size and latency, but validate completeness of analysis output before shipping a split-prompt architecture.

2. Numeric output constraints drift without post-processing enforcement

ContentCommand hit this directly: a 1,500-word target produced 4,288 words, inflating the Surfer SEO score from a real 52 to a misleading 82. The prompt specified a word count range; the model ignored it under longer-context generation pressure.

The fix requires two layers:

// Layer 1: Prompt constraint (necessary but not sufficient)
// "Generate content between 1,275 and 1,650 words (85-110% of 1,500 target).
//  If you exceed 1,650 words, cut the lowest-value paragraphs. CRITICAL: 
//  do not pad to hit minimums."

// Layer 2: Post-processing validation
function enforceWordCount(content: string, target: number): string {
  const words = content.split(/\s+/).length;
  const min = Math.floor(target * 0.85);
  const max = Math.ceil(target * 1.10);
  if (words > max) {
    // trim or flag for re-generation
  }
  return content;
}

The same pattern applies to context limits in Meridian: Haiku-based planning passes need a 40K character hard cap, writing passes need 80K — enforced in code, not just in the prompt.
^[3]

3. Broad keywords cause false positives in pattern matching

In AsymXray's call intent analysis, keywords like "help", "now", "today", and "right now" were triggering on greetings and casual conversation, not genuine urgency or intent signals. The model matched the word, not the semantic context.

Fix: replace single-word triggers with specific multi-word phrases that carry unambiguous intent signal. Phrase weighting was also increased from ×2 to ×3 to improve recall on genuine matches without widening the false-positive surface.

# Before: too broad
URGENCY_KEYWORDS = ["now", "today", "help", "urgent"]

# After: specific phrases only
URGENCY_PHRASES = [
    "need this fixed today",
    "can't wait until",
    "emergency situation",
    "production is down",
]

^[4]

4. AI ignores soft constraint language; requires imperative commands

In LabelCheck, ambiguity detection was specified with language like "should check" and "consider flagging." It didn't trigger reliably. The fix was replacing soft language with explicit imperative commands:

# Before (unreliable)
"You should check for ambiguity when the product category is unclear."

# After (reliable)
"STOP AND CHECK: Before proceeding, you MUST evaluate category confidence.
If confidence is below 85%, you MUST set ambiguity_detected: true.
This is CRITICAL — do not skip this step."

Consistent across LabelCheck: "MUST", "CRITICAL", "STOP AND CHECK" reliably trigger behavior that "should" and "consider" do not.
^[5]

5. AI prioritizes functional signals over structural indicators in classification

In LabelCheck, the model was classifying products based on health claims and functional ingredients rather than the definitive regulatory indicator: whether the label contains a Supplement Facts panel or a Nutrition Facts panel. A product with a Nutrition Facts panel was being classified as a supplement because it contained functional ingredients.

The fix is explicit rule prioritization in the prompt:

"CLASSIFICATION RULE (highest priority, overrides all other signals):
1. If the label contains a 'Supplement Facts' panel → DIETARY SUPPLEMENT
2. If the label contains a 'Nutrition Facts' panel → CONVENTIONAL FOOD
Panel type is the definitive regulatory indicator per 21 CFR 101.
Do NOT override this based on ingredients or health claims."

^[6]

6. JSON parsing fails silently when model wraps output in markdown fences

ClientBrain's sentiment analysis was falling back to default values without error. Root cause: Claude Haiku was wrapping JSON responses in ```json fences, and JSON.parse() was receiving the fence characters, throwing a parse error caught by a silent fallback.

// Fragile
const result = JSON.parse(response.content);

// Fix: strip fences before parsing
function extractJSON(raw: string): unknown {
  // Find first { and last } to handle nested content
  const start = raw.indexOf('{');
  const end = raw.lastIndexOf('}');
  if (start === -1 || end === -1) throw new Error('No JSON object found');
  return JSON.parse(raw.slice(start, end + 1));
}

The indexOf('{') / lastIndexOf('}') approach handles nested objects better than regex fence-stripping, which breaks on JSON containing backtick characters.
^[7]

7. Reference database gaps cause false positive compliance errors

LabelCheck's GRAS database was missing 50+ bioavailable vitamin and mineral synonyms — methylcobalamin, adenosylcobalamin, hydroxocobalamin, pyridoxal-5-phosphate (P-5-P), and chelated mineral forms. A fortified coffee product using methylcobalamin was flagged as non-compliant despite being fully legal.

This is a data problem masquerading as a prompt problem. The fix was expanding the synonym database, not changing the prompt. Before attributing false positives to prompt logic, audit the reference data the prompt is checking against.
^[8]

8. Initial analysis misses claims that appear only in follow-up chat

In LabelCheck, the initial AI analysis scanned the label upload. When users asked follow-up questions in chat, they sometimes introduced marketing language not present in the original label — and the initial analysis pass had no visibility into it. Problematic terms in chat were going undetected.

Fix: extend the analysis scope to include follow-up chat content, or run a secondary detection pass on chat messages. The red flag marketing term list was also expanded from 4 terms to 20+ to improve recall.
^[9]

What Works

Decoupling extraction from synthesis, then caching extractions

In Meridian, separating the extraction pass (reading source documents, pulling structured data) from the synthesis pass (writing the article) enables caching extractions keyed to schema version + file mtime. When you're iterating on the synthesis prompt, extraction API calls drop to zero. The fixture-based regression harness costs ~$0.20 and runs in ~15 minutes for 6 representative topics.

This is the highest-leverage infrastructure investment for any multi-stage LLM pipeline. The cost of prompt iteration without it is proportional to corpus size; with it, iteration cost is flat.
^[10]

LLM-judged rubric evaluation for quantified prompt iteration

Meridian uses a 10-criteria rubric (0-10 scale per criterion) evaluated by a judge LLM call to score synthesis output. This replaces eyeballing diffs between prompt versions. A judge call on a 9-fragment legal topic completed in 20.4 seconds and returned an 88/100 average.

The rubric criteria should match the generation criteria — ContentCommand validated this separately: quality scoring that doesn't evaluate against the same standards used in generation produces misleading scores.
^[11]

Extracting prompts to external markdown files

In LabelCheck, extracting prompts from TypeScript to external .md files reduced analysis-prompts.ts from 360 lines to 32 lines (91% reduction) and a route handler from 2,087 to 871 lines (58% reduction). The prompts become editable by non-developers and diffable in git without TypeScript noise.

The tradeoff: prompts in external files are harder to type-check and easier to accidentally break with whitespace changes. Add a prompt-loading test that validates the file exists and parses to a non-empty string.
^[12]

RAG lite: cheap OCR pre-classification to filter regulatory documents

LabelCheck's full analysis prompt included ~50 regulatory documents. After adding a GPT-4o-mini OCR pre-classification step (cost: ~$0.0001 per analysis, detail: low), the relevant document set drops to 15-25 based on product category. This reduces prompt size by 60-70% without the context fragmentation risk of splitting the analysis prompt itself.

The key distinction: RAG lite filters the inputs to a single comprehensive prompt. It does not split the prompt. This is what the category-specific prompt approach got wrong.
^[13]

Explicit enumeration of acceptable vs. prohibited patterns

In both LabelCheck and ContentCommand, replacing general guidance with explicit categorized lists of acceptable and prohibited examples improved accuracy. For LabelCheck claims analysis: listing nutrient content claims, structure/function claims, and FDA-authorized health claims as acceptable (not just listing prohibited ones) reduced false positives on legitimate supplement language.

# Less effective
"Flag any health claims that may be problematic."

# More effective
"ACCEPTABLE (do not flag):
- Nutrient content claims: 'High in Vitamin C', 'Good source of calcium'
- Structure/function WITH disclaimer: 'Supports immune health*'
- FDA-authorized health claims: 'May reduce risk of heart disease'

PROHIBITED (always flag):
- Disease claims: 'Treats diabetes', 'Cures cancer'
- Structure/function on conventional foods (no disclaimer makes it legal)"

^[14]

Feature flags with graceful fallbacks for prompt experimentation

LabelCheck used feature flags to gate category-specific prompts, falling back to the monolithic prompt when the flag was off or when forcedCategory wasn't explicitly provided. This meant the experimental path never broke production. When the category-specific approach was ultimately removed, the rollback was a flag flip, not a code revert under pressure.

^[15]

Gotchas and Edge Cases

AI-generated insights require post-processing filters for client-specific metric exclusions. In AsymXray, ROAS metrics were appearing in insights for lead-gen clients who don't run ROAS campaigns. The fix was a post-processing filter, not a prompt change — and the filter had to be extended to cover "Local Business" and "Awareness" objectives, not just "Lead Generation." Crazy Lenny's surfaced this when their Marketing Objective was set to "Local Business."
^[16]

Structure/function claims are prohibited on conventional foods entirely, not just missing a disclaimer. The prompt originally flagged structure/function claims on conventional foods as "missing disclaimer." The correct flag is "prohibited." Supplements allow structure/function claims with the FDA disclaimer; conventional foods don't allow them at all under 21 CFR 101. This is a regulatory distinction that must be encoded explicitly — the model will default to the softer interpretation.
^[17]

AI struggles to distinguish Statement of Identity from marketing taglines without explicit guidance. On complex packaging, the model was extracting taglines ("The Ultimate Recovery Formula") as the product name instead of the regulated Statement of Identity. Fix: provide contextual clues — font size hierarchy, net quantity placement, Nutrition/Supplement Facts panel location, manufacturer address, and barcode placement all help identify the Principal Display Panel per 21 CFR 101.1.
^[18]

Supplement ingredients require extraction from two sources. LabelCheck found that extracting only from the Supplement Facts panel missed active ingredients listed solely in the ingredient list. The prompt must explicitly instruct extraction from both sources and deduplication.
^[19]

Allergen detection requires absolute rules, not probabilistic ones. Prompts that say "flag if allergens may be present" produce false positives on hypothetical scenarios. The correct instruction: flag non_compliant only when allergens are confirmed present in the ingredient list, not inferred from manufacturing context.
^[19]

Context overflow in Haiku-based pipelines is silent. In Meridian, the _index.md registry grew to 1.7MB before context overflow was diagnosed. The symptom was degraded output quality, not an error. Fix: compact registries to slug→alias lists instead of full YAML, and enforce hard character limits in code (40K for planning pass, 80K for writing pass).
^[20]

Markdown output formatting requires a client-side safety net. ContentCommand's key takeaways section was rendering as prose instead of bullet lists despite prompt instructions. The fix required both strengthened prompt language and a fixBulletLists() post-processing function. LLM variance on markdown syntax is real and persistent.
^[21]

Where Docs Disagree With Practice

Category-specific prompts: documented as a best practice, failed in production. The standard advice for large prompt optimization is to split by category and load only the relevant subset. In LabelCheck, this produced measurable size and latency improvements in benchmarks but caused incomplete analysis in production due to context fragmentation. The monolithic prompt, despite being larger, produced more complete and consistent output. The benchmark metric (prompt size) was not the right proxy for the outcome metric (analysis completeness).
^[2]

Soft constraint language in prompts: documented as sufficient, insufficient in practice. OpenAI and Anthropic documentation presents constraint language like "you should" and "please ensure" as effective. In LabelCheck's ambiguity detection, soft language consistently failed to trigger the behavior. "MUST", "CRITICAL", and "STOP AND CHECK" were required. This may be model-specific (Haiku vs. Sonnet) but was consistent enough across LabelCheck's use cases to treat as a general rule.
^[22]

Allergen flagging: "flag potential allergens" produces false positives. The intuitive prompt instruction is to flag anything that could be an allergen concern. In practice, this causes the model to flag hypothetical cross-contamination scenarios, manufacturing environment risks, and "may contain" statements as non_compliant when they're actually compliant disclosures. The correct framing is confirmatory, not precautionary.
^[19]

Tool and Version Notes

• GPT-4o-mini with detail: low — Used in LabelCheck for OCR pre-classification at ~$0.0001 per call. Sufficient for category detection from label images; not sufficient for full compliance analysis.

• Claude Haiku (planning/extraction pass) — Context limit behavior: 40K characters for planning pass, 80K for writing pass in Meridian. Silent degradation above these limits, not hard errors. Observed in Meridian.

• Claude Haiku JSON output — Wraps responses in ```json markdown fences inconsistently. Always strip fences or use indexOf('{') / lastIndexOf('}') extraction. Observed in ClientBrain.

• Frase SERP API — Used in ContentCommand to derive target word count (110% of SERP average, clamped 1,200-4,000 words) and LSI keyword lists. Enriches briefs with competitor-derived data rather than static targets.

• LLM-as-judge evaluation — Meridian's judge call on a 9-fragment topic: 20.4 seconds, 88/100 average. Viable for regression testing at ~$0.20 per 6-topic run. Not viable for per-request quality gating due to latency.

Related Topics

Sources

Synthesized from 52 fragments: 49 git commits across AsymXray, ClientBrain, ContentCommand, Crazy Lenny's, LabelCheck, and Meridian; 0 external sources; 0 post-mortems. Date range: unknown to unknown.