System Evaluation Prompt

Tier 1 prompt used for all evaluations. Current: v1

# MagicBlocks Conversation Evaluation v2 — Cache-Optimized

## Architecture: Three-Tier Caching

Gemini's context caching charges you once for the cached prefix, then only for the new tokens on each request. The goal: maximize the **stable prefix** that stays identical across calls.

```
┌─────────────────────────────────────────────────┐
│  TIER 1: SYSTEM INSTRUCTION (cached globally)   │
│  - Role, all 13 eval dimensions, base taxonomy  │
│  - Identical across ALL agents, ALL convos      │
│  - Cache TTL: 7 days+ (update only on release)  │
│  - ~1,200 tokens                                │
├─────────────────────────────────────────────────┤
│  TIER 2: AGENT CONFIG (cached per-agent)        │
│  - Rules, watchlist, custom topics              │
│  - Changes only when admin updates agent        │
│  - Cache TTL: 1-24 hours                        │
│  - ~200-800 tokens (varies by config)           │
├─────────────────────────────────────────────────┤
│  TIER 3: CONVERSATION (never cached)            │
│  - Transcript + per-message metadata            │
│  - Unique every call                            │
│  - Variable size                                │
└─────────────────────────────────────────────────┘
```

### How to implement with Gemini API:

**Option A — Single cache per agent (recommended):**
Create one `CachedContent` per agent that includes Tier 1 + Tier 2. Refresh it when the agent config changes. Pass Tier 3 as the user message.

**Option B — Two-level cache (if Gemini supports nested caching in future):**
Global cache for Tier 1. Per-agent cache for Tier 2. Tier 3 as user message.

**Option C — System instruction + user message (simplest):**
Tier 1 as `system_instruction`. Tier 2 + Tier 3 concatenated as user message. You lose per-agent caching but Tier 1 still caches via implicit prompt caching.

---

## TIER 1: System Instruction

This is the stable, globally-cached portion. It contains ALL eval logic and the base taxonomy. **No agent-specific or conversation-specific content.**

```
You are a conversation evaluator for AI sales agents. Analyze completed conversations across 13 dimensions. Be evidence-based — cite message numbers. If something cannot be determined, say so.

Per-message metadata (sentiment, hallucination flags, rule violations) may be provided. Treat these as ground truth — reference them but do not re-evaluate.

== DIMENSIONS ==

1. SUMMARY
Write a CRM-ready briefing so a human picking up this lead can feel like they were in the conversation. Scale length proportionally to conversation complexity — a simple 3-message exchange gets 2-3 sentences, a complex multi-topic negotiation gets several paragraphs. Never pad short conversations and never truncate important ones.

Provide:
- OVERVIEW: 1-2 sentence high-level summary.
- SITUATION: Who is this person, what do they want, how did they arrive (channel/source if known). Context a rep needs before calling.
- DISCUSSION: What the AI covered in sequence — key exchanges, questions, info provided. Include specifics (amounts, dates, product names).
- NEEDS: List the lead's primary needs, challenges, or pain points.
- SOLUTIONS OFFERED: List the products, services, or options the agent presented.
- CUSTOMER FEEDBACK: List lead reactions — interests, concerns, enthusiasm.
- TEMPERATURE: Where the lead's head is at right now. Buying signals, hesitations, engagement level. Be honest.
- NEXT STEPS: Each follow-up action with owner (lead/agent/specific person) and timeline.

Also extract structured key_facts as name/value pairs for CRM field mapping (budget, timeline, company, decision-makers, loan amount, property type, etc).

2. COMPLIANCE
Holistic review against configured rules. Look beyond per-message compliance for cumulative/pattern violations — e.g. each message is polite but 5 follow-ups in a row creates pressure. Identify: rule broken, violation type (per_message|cumulative|spirit_of_rule), evidence, severity. Rate overall: compliant|minor_violations|major_violations.

3. TECHNICAL ISSUES
Agent errors, malformed responses, system messages leaking through, integration failures, broken links, formatting issues, latency/timeouts.

4. DUPLICATE MESSAGES
Exact or paraphrased repeats from the agent. Note which messages and count.

5. LOOPS
Agent stuck: repeated questions despite receiving answers, circular conversation, unable to process input, regressing to earlier stage. Rate: none|minor (brief, self-corrected)|major (sustained, degraded experience).

6. TOPICS
Classify each distinct topic raised. Use base taxonomy + any custom categories provided.
Base: pricing_and_costs, product_features, availability_and_timing, process_and_requirements, qualification_questions, competitor_comparison, support_or_complaint, scheduling_and_next_steps, general_inquiry, off_topic, other.
Note: who raised it (lead|agent), resolved (yes|no).

8. CUSTOMER UNABLE TO ANSWER
Lead couldn't answer agent's question: explicit "don't know", deflection, or vague non-answer. Note: question asked, response type, how agent reacted.

10. AI NPS
Score 0-10. Consider: need addressed, efficiency, competence, tone, satisfaction at close. Provide rationale.

11. HUMAN REVIEW FLAG
Flag if agent optimizer should review. NOT about lead handoff — about improving the agent. Flag on: compliance violations, technical issues, loops, NPS<5, ineffective objection handling, hallucination flags, off-script behavior, watchlist triggers, significant knowledge gaps, anything unusual. Provide reasons and priority (low|medium|high).

12. MISSING KNOWLEDGE
Distinct from #8 (which tracks when the LEAD can't answer). This tracks when the AGENT lacks knowledge. Identify questions the lead asked that the agent couldn't answer or answered poorly due to missing information — NOT due to agent error. Generalize each into an FAQ-style question suitable for adding to the agent's knowledge base. Filter out person-specific queries (e.g. "when will MY loan close") — only capture reusable knowledge gaps (e.g. "What are the current interest rates for 30-year fixed mortgages?"). Include: generalized question, original context from transcript, suggested knowledge category.

13. CUSTOM WATCHLIST
Check conversation against each admin-defined watchlist item. Report: detected (yes|no), evidence if yes.
```

**Token count: ~640 tokens** (up from ~580 due to Missing Knowledge dimension, still ~33% down from v1's ~950)

---

## TIER 2: Agent Config Block

Injected per-agent. This is the part that changes when the admin updates rules, watchlist, or topic categories.

```
== AGENT CONFIG ==
Name: {{agent_name}}
Purpose: {{agent_purpose}}
Industry: {{agent_industry}}

== RULES ==
{{agent_rules}}

== WATCHLIST ==
{{custom_watchlist_items}}

== CUSTOM TOPICS ==
{{custom_topic_categories}}
```

---

## TIER 3: Conversation Payload (User Message)

Unique per-conversation. Sent as the user message content.

```
== TRANSCRIPT ==
{{conversation_transcript}}

== MESSAGE METADATA ==
{{per_message_metadata}}
```

---

## Structured Output Schema

Passed via `generation_config.response_schema`. The schema itself is cached as part of the API request config — it doesn't consume input tokens.

Changes from v1: removed redundant `description` fields where the property name is self-explanatory, trimmed enum descriptions, and let the schema do the work instead of duplicating constraints in the prompt.

```json
{
  "type": "object",
  "properties": {
    "summary": {
      "type": "object",
      "properties": {
        "overview": {
          "type": "string",
          "description": "1-2 sentence high-level summary of the conversation."
        },
        "situation": {
          "type": "string",
          "description": "Who is this lead, what they want, how they arrived. Context a rep needs before calling."
        },
        "discussion": {
          "type": "string",
          "description": "What the AI covered in sequence — key exchanges, questions, info provided. Include specifics (amounts, dates, names)."
        },
        "needs": {
          "type": "array",
          "description": "Primary needs, challenges, or pain points the lead mentioned.",
          "items": { "type": "string" }
        },
        "solutions_offered": {
          "type": "array",
          "description": "Products, services, or options the agent presented.",
          "items": { "type": "string" }
        },
        "customer_feedback": {
          "type": "array",
          "description": "Lead reactions — interests, concerns, objections, enthusiasm.",
          "items": { "type": "string" }
        },
        "temperature": {
          "type": "string",
          "description": "Lead's current state — buying signals, hesitations, concerns, engagement level. Be honest."
        },
        "next_steps": {
          "type": "array",
          "description": "Each agreed follow-up action.",
          "items": {
            "type": "object",
            "properties": {
              "action": { "type": "string" },
              "owner": { "type": "string", "description": "Who is responsible: lead, agent, or specific person/team." },
              "timeline": { "type": "string", "description": "When — deadline, scheduled date, or ASAP." }
            },
            "required": ["action", "owner"]
          }
        },
        "key_facts": {
          "type": "array",
          "description": "Structured name/value pairs for CRM field mapping.",
          "items": {
            "type": "object",
            "properties": {
              "fact": { "type": "string" },
              "value": { "type": "string" }
            },
            "required": ["fact", "value"]
          }
        }
      },
      "required": ["overview", "situation", "discussion", "needs", "solutions_offered", "customer_feedback", "temperature", "next_steps", "key_facts"]
    },

"compliance": {
      "type": "object",
      "properties": {
        "rating": { "type": "string", "enum": ["compliant", "minor_violations", "major_violations"] },
        "violations": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "rule": { "type": "string" },
              "type": { "type": "string", "enum": ["per_message", "cumulative", "spirit_of_rule"] },
              "evidence": { "type": "string" },
              "severity": { "type": "string", "enum": ["low", "medium", "high"] }
            },
            "required": ["rule", "type", "evidence", "severity"]
          }
        },
        "rationale": { "type": "string" }
      },
      "required": ["rating", "violations", "rationale"]
    },

"technical_issues": {
      "type": "object",
      "properties": {
        "found": { "type": "boolean" },
        "issues": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "type": { "type": "string", "enum": ["agent_error", "integration_failure", "formatting", "latency", "other"] },
              "description": { "type": "string" },
              "evidence": { "type": "string" }
            },
            "required": ["type", "description"]
          }
        }
      },
      "required": ["found", "issues"]
    },

"duplicates": {
      "type": "object",
      "properties": {
        "found": { "type": "boolean" },
        "instances": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "type": { "type": "string", "enum": ["exact", "paraphrased"] },
              "messages": { "type": "string" },
              "count": { "type": "integer" }
            },
            "required": ["type", "messages", "count"]
          }
        }
      },
      "required": ["found", "instances"]
    },

"loops": {
      "type": "object",
      "properties": {
        "severity": { "type": "string", "enum": ["none", "minor", "major"] },
        "instances": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "type": { "type": "string", "enum": ["repeated_question", "circular", "unprocessed_input", "stage_regression"] },
              "description": { "type": "string" },
              "messages": { "type": "string" },
              "self_corrected": { "type": "boolean" }
            },
            "required": ["type", "description", "messages", "self_corrected"]
          }
        }
      },
      "required": ["severity", "instances"]
    },

"topics": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "topic": { "type": "string" },
          "category": { "type": "string" },
          "raised_by": { "type": "string", "enum": ["lead", "agent"] },
          "resolved": { "type": "boolean" }
        },
        "required": ["topic", "category", "raised_by", "resolved"]
      }
    },

"objections": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "type": { "type": "string", "enum": ["price", "timing", "trust", "complexity", "competitor", "authority", "fit", "inertia", "other"] },
          "description": { "type": "string" },
          "handling": { "type": "string" },
          "effectiveness": { "type": "string", "enum": ["effective", "partially_effective", "ineffective", "not_addressed"] }
        },
        "required": ["type", "description", "handling", "effectiveness"]
      }
    },

"unable_to_answer": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "question": { "type": "string" },
          "response_type": { "type": "string", "enum": ["explicit_dont_know", "deflection", "vague"] },
          "agent_reaction": { "type": "string" }
        },
        "required": ["question", "response_type", "agent_reaction"]
      }
    },

"sentiment": {
      "type": "object",
      "properties": {
        "start": { "type": "string", "enum": ["positive", "neutral", "negative"] },
        "end": { "type": "string", "enum": ["positive", "neutral", "negative"] },
        "trajectory": { "type": "string", "enum": ["improving", "stable_positive", "stable_neutral", "stable_negative", "declining", "volatile"] },
        "inflections": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "shift": { "type": "string" },
              "cause": { "type": "string" },
              "message": { "type": "string" }
            },
            "required": ["shift", "cause"]
          }
        }
      },
      "required": ["start", "end", "trajectory", "inflections"]
    },

"nps": {
      "type": "object",
      "properties": {
        "score": { "type": "integer" },
        "rationale": { "type": "string" }
      },
      "required": ["score", "rationale"]
    },

"review_flag": {
      "type": "object",
      "properties": {
        "flagged": { "type": "boolean" },
        "reasons": { "type": "array", "items": { "type": "string" } },
        "priority": { "type": "string", "enum": ["low", "medium", "high"] }
      },
      "required": ["flagged", "reasons", "priority"]
    },

"missing_knowledge": {
      "type": "array",
      "description": "Knowledge gaps suitable for RAG database. Generalize into FAQ-style questions. Exclude person-specific queries.",
      "items": {
        "type": "object",
        "properties": {
          "question": { "type": "string", "description": "Generalized FAQ-style question (not person-specific)" },
          "original_context": { "type": "string", "description": "What the lead actually asked and how the agent responded" },
          "category": { "type": "string", "description": "Suggested knowledge category (e.g. pricing, eligibility, process, product_details, policy)" }
        },
        "required": ["question", "original_context", "category"]
      }
    },

"watchlist": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "item": { "type": "string" },
          "detected": { "type": "boolean" },
          "evidence": { "type": "string" }
        },
        "required": ["item", "detected"]
      }
    }
  },
  "required": ["summary", "compliance", "technical_issues", "duplicates", "loops", "topics", "objections", "unable_to_answer", "sentiment", "nps", "review_flag", "missing_knowledge", "watchlist"]
}
```

---

## Token Budget Comparison

| Component | v1 | v2 | Savings |
|---|---|---|---|
| System instruction (static) | ~950 tokens | ~640 tokens | **33% reduction** |
| Agent config (semi-static) | embedded in prompt | ~50-100 tokens (template) | cleanly separated |
| Schema descriptions | heavy | minimal (schema self-documents) | ~30% reduction |
| Total input per call (static portion) | ~950 cached | ~640 cached | **~310 fewer cached tokens** |

### What changed:
- **Removed prose explanations** where the dimension name + enum values already communicate intent
- **Removed duplicate constraints** — if the schema enforces `enum: ["low", "medium", "high"]`, the prompt doesn't need to restate it
- **Shortened property names** — `conversation_summary` → `summary`, `handling_effectiveness` → `effectiveness`, `human_review_flag` → `review_flag`
- **Replaced generic overview with structured narrative sections** (situation/discussion/temperature/next_steps) — gives CRM-ready briefings that scale with conversation complexity
- **Removed the "do not speculate" padding** — the structured output schema constrains this naturally
- **Combined Tier 1 into one continuous instruction block** — no markdown headers, no separators, no fluff

### Output token estimate:
- Short clean conversation (no issues): ~500-800 output tokens
- Medium conversation (some issues): ~1,000-2,000 output tokens
- Long complex conversation (multiple issues): ~2,000-3,500 output tokens
- Set `max_output_tokens: 4096` (summary section now scales with conversation length)

---

## Gemini API Call Structure

```python
import google.generativeai as genai

# --- SETUP (once per agent, or on config change) ---

# Option A: Explicit cache
cached = genai.caching.CachedContent.create(
    model="models/gemini-2.0-flash",  # or gemini-2.0-pro
    system_instruction=TIER_1_SYSTEM_INSTRUCTION,
    contents=[{
        "role": "user",
        "parts": [TIER_2_AGENT_CONFIG]
    }],
    ttl=datetime.timedelta(hours=24),
    display_name=f"eval-{agent_id}"
)

# --- PER CONVERSATION (the only thing that changes) ---

model = genai.GenerativeModel.from_cached_content(cached)

response = model.generate_content(
    # Tier 3: just the transcript + metadata
    TIER_3_CONVERSATION_PAYLOAD,
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json",
        response_schema=EVAL_SCHEMA,
        max_output_tokens=4096,
        temperature=0.1  # low temp for consistent evals
    )
)

result = json.loads(response.text)
```

```python
# Option C: Simpler approach (implicit caching via system_instruction)
# No explicit cache management needed — Gemini auto-caches system_instruction

model = genai.GenerativeModel(
    model_name="models/gemini-2.0-flash",
    system_instruction=TIER_1_SYSTEM_INSTRUCTION
)

response = model.generate_content(
    # Tier 2 + Tier 3 concatenated
    TIER_2_AGENT_CONFIG + "\n\n" + TIER_3_CONVERSATION_PAYLOAD,
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json",
        response_schema=EVAL_SCHEMA,
        max_output_tokens=4096,
        temperature=0.1
    )
)
```

---

## Notes

- **Schema is NOT in the prompt.** It's in `generation_config.response_schema`. This means schema tokens don't count toward input billing and don't interfere with caching.
- **Temperature 0.1** — evals should be deterministic. Low temp reduces variance between runs on the same conversation.
- **Flash vs Pro** — start with `gemini-2.0-flash` for cost. Move to `pro` only if eval quality on edge cases is insufficient. Flash handles structured extraction well.
- **Watchlist and custom topics scale without schema changes** — they're injected as text in Tier 2 and the schema uses open `string` types for category/item fields.

Change Note (required)

Version History

Current

Initial system prompt seed from conversation-eval-v2.md

system:seedApr 9, 2026, 03:03 AM

Quality Safeguards

Test required before deploy
Schema validation on output
Side-by-side comparison
Version history + instant rollback
Audit trail for all changes