Live workshop · one production path, end to end

Build analytics agents people can trust.

Start by making one recurring analytics workflow trustworthy for yourself. Then earn the right to share it with a team, add a review loop, and operate it in production.

Five hours, live Python optional Works in your chosen harness One finished starter pack

Start with setup See the systems I built

Trusted
decision

Choose the work

Bind context

Run + validate

Show proof

Review + correct

Reuse safely

Question → evidence → decision → learning

One course, three linked spines

Learn it. See it. Build it.

Each module moves through the same three views. I will teach the production pattern, show how that choice appeared in systems I built, and then give you a safe way to create the artifact yourself. If your warehouse is unavailable, the practice case keeps the third spine fully runnable.

SPINE 01

The production framework

Decision contract, approved context, scoped execution, validation, proof, review states, correction memory, and operations.

Start with the framework →

SPINE 02

Systems I built

Acquisition War Room, Abandonment Intelligence, and the conversational data-agent front door—showing the product choices, controls, and operating loops behind each experience.

Open the case studies →

SPINE 03

Your hands-on build

Use an approved company source or run the same workflow on the included conversion lab. Every prompt produces a file you can take back to work.

Download the practice lab →

Choose your starting level

Start solo. Earn multiplayer.

You do not need an enterprise platform to get value from this course. Build the smallest version that solves your current problem, prove it works, and add shared infrastructure only when another person needs to reuse it.

CRAWL · SOLO · N=1

Make yourself better first

Use one harness, one recurring decision, a local folder or private Git repository, approved context, a proof receipt, and a small eval set. Finish Modules 0–4. You do not need an MCP gateway, admin portal, or enterprise semantic layer.

Move on when: the workflow repeatedly returns a useful, verified result, catches a known failure, and saves enough effort to run again.

WALK · SMALL TEAM · N=3–10

Share one working path

Put the contract, context, evals, and corrections in one versioned place. Ask another person to run the workflow without sitting beside you. Add a flagged-answer inbox and reviewed promotion path in Module 5.

Move on when: a second person can reproduce the result, report a miss, and reuse an approved correction in a fresh session.

RUN · ORGANIZATION · MANY USERS

Operate a shared system

Move common guarantees behind a client-neutral MCP or gateway. Add permissions, shared semantic context, deduped review queues, versions, telemetry, rollout controls, cost limits, and an incident path in Module 6.

Scale when: the pilot works across users and tools, review demand is sustainable, and an owner can monitor, stop, repair, and safely resume the route.

The promotion rule

Do not solve tomorrow's coordination problem before one person has a workflow worth sharing. Solo proof earns a team pilot. A team pilot earns shared infrastructure. The lab lets one learner rehearse the team handoff with two fresh sessions, but a simulated handoff is practice—not proof of adoption.

Module 0Solo foundation20 minutesOutput: ready workspace + working source

20:00

Before the first prompt

Make sure the agent can see the right data and context.

Use Codex, Cursor, Claude Code, or the harness your team already trusts. For the live lab, start in a clean local session with real company tools disconnected.

01Choose one isolated workspaceFor the live build, download the lab ZIP and open its extracted folder. Use a separate approved session when adapting the workflow to company data. Never mix the two.

02Check every connected toolDisconnect ChatData, warehouses, company MCPs, and other real-data or externally write-capable tools before opening the lab. A connected tool is in scope even when you do not intend to use it.

03Confirm the lab rootYou should see README.md, verify.py, data/, context/, feedback/, evals.yaml, and work/ directly inside the open folder. Every course path starts here.

04Test a harmless local readAsk the agent to list the two CSV files, their row counts, and date bounds. Do not analyze the business question yet.

05Load only the supplied contextRead the metric definition and business-change note from context/. Treat both as approved inputs for the exercise, not as files the agent may overwrite.

06State the write boundaryThe agent may read inside the extracted lab and write only under work/. If it cannot follow that boundary, stop before the first prompt.

No warehouse? Start here.

Use the same practice case Paras will run live.

The kit contains two CSV files, the metric definition, a known business change, a seeded user report, an optional dependency-free Python 3.9+ verifier, and ten eval cases. It works locally with any file-capable harness.

Download the lab kit Read the lab guide Preview the data Download calculation CSV Download benchmark CSV

Use the preview if your browser does not open CSV files directly. Every download keeps its original .csv filename.

Before you open the lab

Start a clean local harness session and disconnect ChatData, warehouse connections, company MCPs, and other real-data tools. If you cannot disconnect them, use a separate profile or observe a paired run. Do not use a live company workspace for this exercise.

Run the kit in three steps

1. Download and unzip the kit. Do not clone the repository for the live exercise. 2. Open the extracted folder, not the ZIP file, in your harness. Confirm that README.md, verify.py, data/, context/, feedback/, evals.yaml, and work/ sit directly inside it. 3. If Python is available, confirm python3 --version reports 3.9 or newer, then run python3 verify.py --readiness-only (Windows: py -3 --version, then py -3 verify.py --readiness-only). It should print READINESS: PASS without calculating the answer. If Python 3.9+ is unavailable, confirm the seven lab-root items and continue in the harness. Do not install a runtime during class.

funnel_segments.csvcalculation source
wbr_benchmark.csvindependent tie-out
metric.yamlmeaning + limits
flagged_answer.yamlseeded user feedback
verify.pyoptional independent check
evals.yamlfailure cases

You are ready when the agent can name the active source, summarize the loaded context, and state what it cannot do.

AI Analyst implements this with setup state, an active dataset, a connection test, and session-start context loading. This workshop keeps the same sequence without requiring one specific coding tool.

Walkthrough + timing + connection fallback

“Your favorite harness does not decide whether the analysis is trustworthy. The active data, loaded context, permissions, and checks do.”

0–4 min: Ask students to open a clean local harness session and disconnect real company tools. Do not spend class time comparing harnesses.

4–8 min: Download and unzip the workshop kit. Open the extracted lab root; run the optional spoiler-free readiness check.

8–15 min: Run the readiness check: active source, visible tables or files, date range, one harmless read, and explicit access boundary.

15–20 min: Load the minimum business context, inspect work/00_readiness.md, and repair one missing item before moving on.

Hands-on · 12 min

Prove what is actually ready.

A connector shown in settings is not enough. Record what the agent successfully read in this session.

Harness + working folder

Active source + connection proof

Context loaded

Access boundary + fallback

Saved locally on this device

Course sequence adapted from AI Analyst’s MIT-licensed setup guide, first-run routing, data connection, and knowledge bootstrap. AI Analyst uses Claude Code; the workshop readiness check is intentionally tool-neutral.

The day

Every hour creates part of the final plan.

The course alternates between a short walkthrough, a worked example, and build time in your own harness. Two breaks and 30 protected minutes for questions are included in the five hours.

0:00–0:20Prove the setupChoose your safe source or download the kit. Confirm access, date bounds, context, and limits before asking for analysis.Build 0

0:20–0:45Choose work worth buildingFrame the decision, score the use case, and approve what the harness will analyze.Build 1

0:45–1:20Write the contract + map the systemName the job, authority, human boundary, and the component that owns each guarantee.Build 2

1:20–1:30Q&A checkpointResolve setup, tool, MCP, source, use-case, and contract questions before the technical build.10 min

1:30–1:40BreakLeave your contract and architecture map on screen.10 min

1:40–2:20Build the context mapCreate the metric file, source precedence, caveats, freshness rule, and retrieval path.Build 3

2:20–3:05Run, verify, and show proofExecute one complete analysis, reconcile it independently, correct one claim, and save the proof.Build 4

3:05–3:15Q&A checkpointBring the number, caveat, validation, or response-state choice you cannot yet defend.10 min

3:15–3:25BreakReturn with one failure you want the agent to catch.10 min

3:25–4:10Flag it, review it, promote it, rerun itRun the cases, hand a flagged answer to a data admin, promote the correction, and prove a fresh session reuses it.Build 5

4:10–4:35Operate it + present the launch decisionSet controls, owners, monitoring, cost, rollout, incident response, and a Scale / Hold / Stop ask.Build 6

4:35–4:50Peer reviewUse five questions to find and repair the weakest part of one another’s plan.Review

4:50–5:00Open Q&A + export + Monday commitmentResolve the final blocker, export the starter pack, and name the first owner and artifact.Close

Use your approved source

Frame and contract your real question

Use an approved local extract or read-only warehouse route for Modules 0–3. Keep sensitive data and credentials out of this page. After class, adapt the tested execution path to your source.

Use the workshop source

Regroup on one runnable case

Everyone uses the included CSV lab for Modules 4–5 so the execution, proof, failures, and repair can be compared in the room. No warehouse, MCP connection, paid tool, or Python package install is required.

Reference gallery · no separate time block · systems I built

See the operating system before you build the pieces.

These examples show how I turned recurring business questions into products people could inspect, challenge, and use. We will return to the design choices inside the timed modules rather than add another lecture block.

Case 01 · Executive sensing

Acquisition War Room

The recurring question was simple: what changed, why, and where should the operating team focus next? The hard part was producing one briefing that combined quantitative movement, business context, operating signals, and an honest verification state.

01Parallel specialists, narrow jobs. Headline metrics, deeper funnel analysis, segment movement, operating signals, and business context ran as separate workstreams.

02Analyzed summaries, not raw-table handoffs. Each specialist returned a compact result so synthesis did not repeat the analysis or ingest every row.

03Verification stayed visible. An unverified or review-required result could still be useful, but nobody could mistake it for an approved answer.

04The decision sat next to the evidence. The number, decomposition, caveat, next investigation, and authority boundary lived on the same screen.

Open the full example See the proof pattern

Practice data: included course CSVs · focus: movement, contributors, caveats, validation, and the next decision

Case 02 · Investigation + action

Abandonment Intelligence

The War Room says what moved. Abandonment Intelligence goes underneath it and asks what specific rule, route, or product flow deserves investigation. It rejects “conversion is down” as an answer because a rate alone does not tell an engineer or operator what to do.

01Investigators followed funnel breakpoints. Each agent owned a narrow abandonment area instead of letting one generalist scan the entire business.

02Every finding needed a shape. Affected population, broken flow, durability, evidence, owner, next action, and confidence.

03Known-good behavior was encoded. The system had to distinguish a real defect from an intentional rule or expected operating pattern.

04Synthesis ranked work, not observations. The result was a reviewable action list with gates, not a longer dashboard.

Open the full example See the correction loop

Practice data: included course CSVs · focus: ranked actions, known-good behavior, exceptions, and investigation depth

Case 03 · Conversational front door

The data agent connected questions to the same governed path.

The chat interface routed direct questions, investigations, and recurring briefings. An early test returned a fluent but incorrect number after choosing the wrong source. That failure changed the design: source selection, semantic context, visible query evidence, validation, time windows, security, and PII controls became system responsibilities instead of prompt reminders.

What I would borrow: keep the client replaceable. Put the trust logic in the shared path every client calls.

01QuestionDirect answer, investigation, or briefing

02RouteIntent, scope, authority, and risk

03ContextApproved metric and business rules

04SourceScoped route with execution proof

05ValidateLogic, benchmark, and failure gates

06RespondAnswer, Clarify, Review, or Refuse

All three spines, end to end

Use the same sequence to learn, inspect, and practice.

The examples and the lab are not side stories. They map to the exact artifact you build in each module.

Course step	Spine 01 · Framework	Spine 02 · What I built	Spine 03 · What you build
Choose	Start with a recurring decisionFrequency, repeatability, value, and risk.	War Room + Abandonment“What changed?” and “what deserves action?” were separate recurring jobs.	Use-case briefFrame the practice conversion question or your approved question.
Contract	Bound output and authorityDefine Answer, Clarify, Review, and Refuse.	Visible verification and action gatesA useful result could still require review; no autonomous budget or routing change.	Contract + architectureCreate `work/02_contract.yaml` and map the shared request path.
Context	Bind approved meaningMetric, source, business change, owner, freshness, and precedence.	Domain rules survived every runCorrect funnel order, time comparison, exclusions, known-good behavior, and prior corrections.	Metric proposalInspect `context/metric.yaml` and propose `work/03_metric_proposal.yaml`.
Prove	Execute and verify independentlyKeep source execution, validation, caveat, and decision together.	War Room proof layerCross-validation, freshness, decomposition, and verification state appeared in the briefing.	Proof receiptRun the lab, tie to the benchmark, and save `work/04_proof.yaml`.
Learn	Turn failures into shared contextFlag the answer, review it, promote the correction, then rerun from a fresh session.	The wrong-source failure changed the architectureFluent output was not accepted as proof.	Feedback + reuseHand off the seeded report, approve the narrow correction, and prove the next run uses it.
Operate	Own release, monitoring, cost, and incidentsShadow, canary, narrow production, rollback, and reapproval.	Replaceable client, shared controlsParallel workers and chat interfaces called the same governed path.	Operations + launch memoFinish the control sheet and make a Scale, Hold, or Stop ask.

How to use the examples

Start with the decision the user needs to make. Then inspect the context, calculation, validation, review state, and correction path that make the answer useful more than once.

My first-principles operating model

Trustworthy analytics needs meaning, feedback, and review.

Over the last year, I have been obsessed with one question: how do you spread agentic analytics across a company without giving up trust? I worked backward from a failure I kept seeing. The agent could choose the wrong definition and still sound polished. A user could spot the problem but had no way to fix the shared system.

01 · Semantic model

Give the agent shared meaning

The agent needs the metric, source, exclusions, and approved logic.

In my implementation, Snowflake Cortex provided this layer. Claude Code, Cursor, Gumloop, and other tools called it through MCP. Cube could fit here too; I chose to put more of the next investment into feedback and review.

02 · User feedback

Capture wrong or unverifiable answers

Users need one easy way to flag an answer that is wrong or hard to validate.

We started in Slack. Later, we built a plug-in and MCP path that wrote back a concise summary of the question, answer, evidence gap, and user feedback.

03 · Data-admin review

Promote the correction to source of truth

Data owners review the edge case and settle the canonical definition or answer path. They publish the accepted change into the semantic layer.

We narrowed a sprawling dashboard catalog to a small set of approved paths stored as code. The agent could now compare its work with logic the team had reviewed.

The operating result

Spend people’s time on the edge cases

The review burden started high. As self-checks and validations improved, routine cases stopped consuming the same human attention.

The goal is not zero human review. Each review should improve the shared path.

The flywheel

When the agent gets it wrong or the user cannot validate it, the user flags the gap. The data team defines the canonical answer and updates the semantic model and eval suite. The next run gets better. No single vendor has to own the whole loop; the interfaces between these three parts matter more.

Compare with production practice

How four production analytics agents sharpen this course.

These teams took different paths, but they converged on the same operating challenge: connect a question to the right meaning, prove the work, and make each correction improve the shared system. Each lesson below maps to an artifact you build in the course.

OpenAI · Modules 3–5

Context, code, and evals are one system

OpenAI layers usage metadata, expert annotations, code-derived definitions, institutional knowledge, memory, and live warehouse checks. Table shape is not meaning; the code that creates a table carries assumptions, freshness, and business intent.

Use it in the course: bind approved meaning in Module 3, save execution proof in Module 4, and promote corrections in Module 5. Their evals execute generated and golden SQL, then compare the resulting data. Read the OpenAI case

Anthropic · Modules 3, 5, and 6

Design for ambiguity, staleness, and retrieval failure

Anthropic treats those as three separate failure modes. Its answer is fewer canonical datasets, definitions enforced through tooling and CI, and maintained skills that route the agent to the right governed context.

Use it in the course: make the metric proposal canonical in Module 3, test each correction against an eval slice in Module 5, and gate launch by domain in Module 6. Read the Anthropic case

Meta · Modules 2–4

Bound the domain, encode the workflow, show the work

Meta narrows the search space with analyst query history, then adds documentation, warehouse metadata, pipeline code, and semantic models. Cookbooks package a domain, Recipes encode its analysis and validation, and Ingredients carry its definitions.

Use it in the course: separate procedure from meaning in Modules 2 and 3, then surface SQL and iterative evidence in Module 4 so a user can inspect the path. Read the Meta case

Ramp · Modules 3–5

Test the route to the answer

Ramp combines dbt, Looker, and Snowflake metadata with documentation written by domain owners. Stateful Slack threads let users clarify intent, while in-thread data previews make the result easier to inspect.

Use it in the course: save the source and calculation in Module 4, then turn failures into context fixes and regression cases in Module 5. Ramp’s tests check tool calls, table references, and query shape as well as the final answer. Read the Ramp case

Module 1Solo foundation25 minutesOutput: approved use-case brief

25:00

Choose the work

Start with a decision that repeats. Skip the chatbot that answers anything.

A production agent earns its maintenance cost when the work recurs, the path can become reliable, the answer changes something, and the downside is understood.

Four questions decide whether to build.

The first three come from the build-first lesson. The fourth prevents a high-impact use case from quietly crossing a line the team has not designed for. Then make one more choice: if a fixed query or deterministic workflow can do the job reliably, use it. Add agent autonomy only where the steps genuinely require judgment or adaptation.

01Frequency: How often does this exact work return?
02Repeatability: With the same inputs, do the metric, direction, drivers, caveat, and basic recommendation stay stable?
03Decision value: Who changes what because this answer exists?
04Decision risk: What is the cost of a quietly wrong answer or an action taken too early?

Assist when neededConsistent, but too occasional to maintain as a production system.

Build here firstFrequent, repeatable, decision-relevant. Pick the largest safe opportunity.

One-off investigationUse AI to help, but do not create an operating dependency.

Fix the foundationThe recurring demand is real. Reconcile data, definitions, or analysis first.

How often the work returns → Repeatability ↑

Walkthrough + timing

“If two dashboards disagree, adding a chatbot creates a faster disagreement.”

0–5 min: Ask everyone to put one recurring metric question in chat, then read three aloud without critiquing them yet.

5–12 min: Teach the matrix. “Same answer” means the definition, source, number or direction, main drivers, material caveat, and basic decision remain stable. The prose can change.

12–15 min: Place the practice conversion question on the matrix. Use the War Room / Abandonment distinction to separate “what changed?” from “what deserves investigation?” Move it to “fix the foundation” when tagging changes and no precedence rule exists.

15–25 min: Students use the prompt, approve the brief, and select the question they will carry forward.

Harness turn 1 · Frame the work, then waitUse your source or the lab kit

Do not analyze yet. Help me frame the most recent complete period versus the prior comparable period.

Return:
- Goal
- Decision this answer supports
- Metric
- Testable hypotheses
- How often this work returns
- Cost of a quietly wrong result
- Success condition
- Why this needs an agent instead of a fixed query, dashboard, or deterministic workflow

Ask the smallest missing clarification. Then recommend Build, Fix the foundation, Assist, or Do not build. Prefer the simplest workable design and begin with one agent unless observed failures require a more complex workflow. Stop and wait for my approval. Save the approved brief as work/01_use_case.md.

You should now haveA one-page use-case brief, not an analysis.

Done whenThe decision, metric, comparison period, downside, and success condition are explicit.

NextApprove the brief before asking the harness to write the contract.

Optional browser notes

Choose one use case.

Use this if you prefer the page to a file. Use approved material; do not paste company data, credentials, customer information, or proprietary code into the page.

Recurring business question

Who needs the answer?

What decision changes?

Frequency Daily / weekly / monthly / occasional

Repeatability evidence

Business impact

Cost of a quietly wrong answer

Decision: build, fix foundations, assist on demand, or do not build?

Saved locally on this device

Module 2Solo foundation35 minutesOutput: contract + architecture map

35:00

Define the job

Write the decision contract before you design the agent.

The contract forces one useful sentence: who needs what answer to make which decision, with what authority and human boundary.

A useful agent has a narrow job and an explicit line it cannot cross.

Do not start with the tools. Start with the decision. Then specify the inputs, output, authority, and review trigger. The contract should fit on one screen and be readable by the business owner, data owner, and builder.

AI Analyst preflight: Goal → Decision → Metric → Hypothesis. If you cannot state all four, ask before touching the data. Add a concrete success condition so the analysis has a finish line.

01User + forumWho receives this and where?

02Decision + actionWhat choice and next move can change?

03OutputWhat must the agent return?

04EvidenceWhat must be true before it answers?

05AuthorityWhat may it do on its own?

06Stop conditionWhen must it clarify, review, or refuse?

07Human ownerWho owns the final decision and repair?

Meta’s Rule of Two: check the session before granting autonomy

Mark whether this session can (A) process untrusted input, (B) access sensitive systems or private data, and (C) change state or communicate externally. If all three are present, do not let the agent operate autonomously. Remove one capability, split the work into a fresh trusted session, or require a person to approve the action.

Map the production path: where does each guarantee live?

A prompt in Codex, Cursor, or Claude can remind the agent what to do. Shared guarantees belong in the common path the clients call. Label who owns permissions, versions, persistence, failure handling, and review.

01HarnessCollects intent and shows the result. It should not be the only enforcement point.

02Tool / MCP contractValidates inputs, scope, versions, and allowed operations.

03Approved contextSupplies the metric, source precedence, caveats, owners, and expiry.

04Scoped source adapterUses read-only credentials, allowlists, limits, and timeouts.

05ValidationRuns source, structure, logic, business-rule, and reconciliation checks.

06Proof storeSaves the question, versions, source result, checks, caveats, and outcome.

07Human reviewOwns high-risk, conflicting, novel, or failed paths.

08Correction + eval memoryTurns reviewed failures into shared fixes and regression cases.

For [user / decision forum], this agent answers [recurring question] so they can decide [specific action].

It may: [read / calculate / compare / draft / recommend].
It may not: [publish / spend / change source truth / take irreversible action].

It must return: what it did next, the answer or clarification, source, freshness, calculation path, assumptions, caveats, validation, and next action.

It may answer only when: [required evidence and validation].
It must clarify, review, or refuse when: [risk, novelty, conflicting evidence, missing owner, failed validation, or action threshold].
Human owner: [role accountable for the decision and repair].

File example · use when helpful

Turn the decision contract into a machine-readable agent contract.

AI Analyst declares inputs, outputs, dependencies, context, and whether a failed step must stop the pipeline. Keep the business decision in the description so orchestration does not erase intent.

name: conversion-wbr-review
description: Explain weekly conversion movement so Growth can choose the next investigation.
inputs:
  - name: QUESTION
    type: str
    source: user
    required: true
  - name: SOURCE_TIEOUT
    type: file
    source: agent:source-tieout
    required: true
outputs:
  - path: work/02_contract_example.md
    type: markdown
depends_on: [source-tieout]
knowledge_context:
  - context/metric.yaml
critical: true

Adapted from the MIT-licensed AI Analyst contract format.

Harness turn 2 · Draft, break, and revise the contractStop after one revision

Using the approved use-case brief, draft a seven-field contract:
1. user and decision forum
2. decision and action supported
3. required output
4. evidence required before answering
5. what the agent may do
6. when it must Clarify, Review, or Refuse
7. human owner

Then try to break the contract with one vague request and one forbidden action. Revise it once.

Also draw the request path from harness to tool/MCP contract, approved context, scoped source, validation, proof, human review, and correction memory. Label which component owns permissions, persistence, versioning, and each failure.

List the minimum tools this path needs. For each tool, record its purpose, required inputs, read/write scope, risk tier, and overlap with other tools. Remove or rename any tool when it is unclear which one the agent should choose. Start with one agent unless an observed evaluation failure justifies another.

Apply the Rule of Two to this session:
- A: can it process untrusted input?
- B: can it access sensitive systems or private data?
- C: can it change state or communicate externally?
If A, B, and C are all present, require human approval or redesign the session before granting autonomy.

Save work/02_contract.yaml and work/02_architecture.md. Mark unknown owners instead of inventing them.

You should now haveA contract and an eight-part system map.

Done whenA vague request and an action request both route safely, and every guarantee has an owner.

NextUse the contract to decide which context the system is allowed to trust.

Walkthrough + timing + live test

“If the contract says ‘help leaders understand the business,’ it is still a wish. Name the decision and the next action.”

0–7 min: Teach the seven fields. Contrast “Tell me how growth is doing” with “Explain last week’s qualified-signup conversion movement for the Growth WBR and recommend the next investigation.”

7–13 min: Map the client-neutral production path. Use the data agent’s wrong-source failure to ask where permissions, context versions, validation, proof, and review actually live. Apply the Rule of Two before granting autonomy.

13–28 min: Students run the prompt, inspect the two files, remove one ambiguous tool or boundary, and keep the design to one agent unless an eval shows why more are needed.

28–35 min: Peer test: a partner tries one ambiguous question and one high-impact action request.

Optional browser notes

Write the contract in plain language.

If the authority or review boundary is vague, the agent is not ready to leave the builder’s hands.

User and decision forum

Decision supported

Required output

Evidence required before the agent may answer

The agent may

The agent may not

The agent must clarify, review, or refuse when

Human owner

One-sentence contract

Saved locally on this device

Q&A checkpoint 1

Clear the setup and contract blockers now.

Bring source access, MCP, harness, permissions, use-case, or architecture questions. We will park deep company-specific design for the final Q&A.

10:00

Module 3Solo foundation40 minutesOutput: reviewed metric + context path

40:00

Bind the meaning

Give the agent the context your team normally carries in its head.

Trust starts upstream: one definition, named owners, an approved source path, freshness, caveats, and precedence when sources disagree.

The goal is not “more context.” It is the smallest approved context that changes the path safely.

For each golden question, distinguish what must exist, what was available, what the agent retrieved, and what actually changed the answer. This makes a miss diagnosable instead of producing another document dump.

01Metric: definition, formula, grain, time zone, filters, exclusions, owner.
02Source: approved dashboard, query, semantic object, or table; freshness and lineage.
03Business: product changes, promotions, seasonality, operating rules, decision thresholds.
04Memory: known caveats, corrections, prior reviewed answer paths, expiry rule.

Required contextWhat a defensible answer must useDefined before the run

Eligible contextApproved material available to the agentOwned + current

Retrieved contextWhat the agent actually pulledObserved at runtime

Applied contextWhat changed the path, result, caveat, or outcomeEvidence of use

Many → fewdashboards to approved decision paths

First-person operator example

The real work was deciding which definitions deserved to survive.

Narrowing a sprawling dashboard catalog to a small set of approved paths was not mainly a cleanup project. It forced the team to settle metric meaning, ownership, source precedence, and where trusted logic lived. That work made the analytics foundation easier for people and agents to reuse.

A metric name is not a metric definition. If the exclusion lives only in a wiki page or a BI filter, a model can write correct SQL and still return the wrong answer.

METRIC TRUST PACKET
Metric + business question:
Business owner / data owner:
Definition + formula:
Numerator / denominator:
Grain / time zone:
Required filters / exclusions:
Approved source path:
Freshness requirement:
Known caveats / recent changes:
Validation rule + tolerance:
When to clarify, review, or refuse:
Last reviewed / next review:

File example · use when helpful

Save the metric as context the agent can actually read.

Treat the metric definition as a reviewed file, not a sentence buried in a prompt. The agent should get the same meaning, owner, source, filters, freshness, and caveats on every run.

lab_version: 1
metric_id: qualified_signup_conversion
definition: eligible qualified signups divided by eligible qualified sessions
numerator: qualified_signups
denominator: qualified_sessions
grain: complete Monday-to-Sunday reporting week
timezone: America/Los_Angeles
include:
  population: eligible
exclude:
  - test
  - partner_assisted
freshness:
  use_only: is_complete = true
comparison:
  prior_week: 2026-08-03
  current_week: 2026-08-10
source:
  calculation: data/funnel_segments.csv
  benchmark: data/wbr_benchmark.csv
benchmark_tolerance_percentage_points: 0.2
known_caveats:
  - campaign tagging changed on 2026-08-07
  - the current incomplete week must not be used
decision:
  owner: Growth lead
  use: choose the next investigation before changing campaign routing
authority:
  may: [read local lab files, calculate and validate, recommend the next investigation]
  may_not: [change campaign routing, move budget, modify source files]
expected_response_for_causal_question: Review

The pattern follows the public AI+Data metric trust packet and nao’s semantic-layer template. Replace the practice metric with an approved metric from your team.

Harness turn 3 · Bind only approved contextDo not guess missing definitions

Read only the supplied metric, source, and business-change material. Draft work/03_metric_proposal.yaml with the definition, formula, owner placeholders, grain, timezone, filters, exclusions, source precedence, freshness, tolerance, caveats, review date, and stop conditions. Do not overwrite the approved context/metric.yaml.

Do not invent missing values. List unresolved items.

For this question, show:
- required context
- eligible approved context
- context you actually retrieved
- context that materially changed the path, result, caveat, or outcome

For every retained context item, record its owner, retrieval trigger, freshness or expiry, and why it is needed. Keep lightweight identifiers in the working context and retrieve detailed material only when the question requires it. Remove context that does not change the route, result, caveat, or response state.

Propose two context tests: one missing-required-context case and one conflicting-context case. State the expected outcome before running them.

Show the proposed file diff and wait for approval before treating it as trusted.

You should now haveA reviewable metric file and a four-part context trace.

Done whenA new teammate can choose the right definition and knows when to stop.

NextApprove the context before the harness proposes or executes the source path.

Walkthrough + timing + context failure demo

“The model may be smart enough to find a number. The system has to know whether it found the right number for this decision.”

0–8 min: Build the practice metric packet: eligible qualified signups ÷ eligible qualified sessions, complete weekly grain, PT, excluding test and partner-assisted rows.

8–15 min: Reveal the tagging change and partial-week caveat. Connect it to the War Room’s visible verification state: the same raw number moves from Answer to Review when context is missing.

15–20 min: Teach Required / Eligible / Retrieved / Applied. Classify six failure layers: missing asset, retrieval miss, context conflict, application error, query error, source data error. Connect Anthropic’s high-signal context guidance with Ramp Research’s domain-owner documentation and context-layer tests.

20–40 min: Students run the prompt, inspect the diff, remove any field that does not affect the path, result, caveat, or outcome, and declare one missing-context and one conflict test.

Optional browser notes

Build the minimum trusted context.

Write enough that a capable colleague who does not know your data could select the right definition and source. They should also know when to stop.

Metric definition and formula

Business owner and data owner

Grain, time zone, filters, exclusions

Approved source path

Freshness rule

Source precedence when artifacts disagree

Known caveats and recent changes

Validation rule and tolerance

Required context IDs or artifacts

Saved locally on this device

Module 4Solo proof45 minutesOutput: proof + response rules

45:00

Prove the answer

A safe system knows when to answer, ask, send to review, or stop.

Choose what the agent should do before polishing the prose. A fluent answer never gets priority over the correct outcome.

Common live execution lane

From this point through Module 5, run the included CSV lab in your chosen harness. This gives the room one known result, one benchmark, and the same controlled failures. Your own-source design from Modules 0–3 remains the transfer case you adapt after the tested path works.

Answer

The approved path ran and the required validation passed. Return the answer with evidence and caveats.

All checks pass

Clarify

The question is missing the smallest detail needed to choose the metric, period, segment, grain, or comparison.

Missing scope

Review

The path is plausible, but ownership, business context, a novel join, benchmark, or decision risk needs a person.

Plausible, not proven

Refuse

The source is unsafe, stale, contradictory, out of scope, restricted, or failed a blocking validation check.

Blocking check failed

Run checks in dependency order: source tie-out and structure first, then logic, business rules, and statistical or segment-reversal checks. A failed source tie-out stops the analysis; more sophisticated validation cannot repair the wrong input.

Example outcome · ReviewConversion fell 1.5 percentage points.

The movement reconciles to the weekly export, but a tagging change overlaps the period. Do not call it purely a product issue yet.

QuestionWhy did qualified-signup conversion fall in the most recent complete week?

Sourcefunnel_segments.csv + independent WBR benchmark; source run confirmed

FreshnessLast complete week; refreshed before the WBR cutoff

ValidationTotals reconciled within 0.2 percentage points; filters applied

CaveatCampaign tagging changed three days before the movement

Next stepReview traffic quality and rerun using pre/post-tagging cohorts

Outcome: Refuse
Failed check: [freshness / source mismatch / conflict / arithmetic / restricted input]
Observed: [what the system can prove]
Risk: [why answering would be unsafe]
Next step: [smallest repair or narrower safe question]

File example · use when helpful

Write the proof as a reusable artifact, without storing raw rows.

Separate the visible answer from the evidence trail. A reviewer should be able to find the source, see what passed, understand the caveat, and know the next action.

title: Weekly conversion review
metric_id: qualified_signup_conversion
answer_state: needs_analyst_review
source: data/funnel_segments.csv
evidence_checked:
  - source execution confirmed
  - dashboard tie-out within 0.2 percentage points
freshness: last complete week
validation: arithmetic, exclusions, and grain passed
business_context_check: campaign tagging changed during the period
uncertainty: attribution is not yet isolated
caveats:
  - do not call the movement purely a product issue
next_action: compare pre- and post-tagging cohorts
owner: growth_analytics

The fields follow the public AI+Data trusted-answer lifecycle.

Harness turn 4A · Plan before executionInspect, approve, then run

Propose the exact source path, calculations, and validation checks for the approved question. Name the files or tables and the expected output.

Run checks in dependency order: source and structure, logic, business rules, segment or statistical checks, then benchmark reconciliation.

Return plan_only: true and source_executed: false. Save the plan as work/04a_plan.yaml. Do not execute until I approve.

Harness turn 4B · Execute, verify independently, and correctCore build-along

For the live workshop, use the included CSV lab. Approved. Run the plan using read-only sources.

1. Calculate the last two complete weeks from the calculation source.
2. Reconcile them to the independent benchmark.
3. Show percentage-point and relative change separately.
4. Decompose by channel, device, and landing page; run an isolation check.
5. Run verify.py as a second method when Python is available. Compare every quantitative claim as Pass, Warning, or Fail.
6. Revise any failed claim before continuing.

Return exactly one outcome: Answer, Clarify, Review, or Refuse. Include the question, source execution proof, context used, validations passed and failed, uncertainty, caveat, next action, and human owner. Save work/04_proof.yaml.

Do not call an observed contributor a root cause unless the evidence isolates causality.

You should now haveOne executed answer, an independent tie-out, and a saved proof file.

Done whenEvery number traces to output; the causal wording matches the evidence; a failed critical check stops the answer.

Expected lab outcomeReview: 8.4% → 6.9%; paid search is the largest observed contributor; tagging limits causality.

Check your work: practice-case reference answer

Overall: 840 / 10,000 = 8.4%; 690 / 10,000 = 6.9%; -1.5 percentage points and -17.86% relative.

Driver: paid search moved from 8.33% to 6.20%; mobile paid search moved from 8.0% to 5.6%; short-form share of mobile paid-search traffic rose from 50% to 75%. At prior segment rates, the current mix would have produced 830 signups; actual was 690. Paid search contributes 100 of the 140-signup segment-rate gap, or 71.4%.

Isolation: without paid search, conversion still falls from 8.43% to 7.60%. Call paid search the largest observed contributor, not the sole root cause.

Walkthrough + timing + break-the-data demo

“A useful refusal does not apologize for three paragraphs. It names the failed check and the repair path.”

0–8 min: Teach the four outcomes and why model confidence is not a correctness check.

8–13 min: Show the War Room proof receipt. Separate “planned route” from “source ran” and ask what is still uncertain.

13–20 min: Students run Turn 4A, inspect the exact CSV path and checks, and approve the plan.

20–38 min: Students run Turn 4B, calculate and reconcile the result, run the independent verifier when available, and correct any unsupported claim.

38–45 min: Save work/04_proof.yaml. Invite two people to read only their outcome, failed or review check, and next step.

Optional browser notes

Design the proof and the stop rules.

Match the checks to the stakes. A board, finance, customer, or irreversible action needs a higher bar than an exploratory team question.

Checks that run every time

Answers that require human review

Proof receipt fields

Clarify when

Review when

Your useful refusal

Saved locally on this device

Q&A checkpoint 2

Bring the number or caveat you cannot yet defend.

We will work through source conflicts, calculations, validation order, response-state choices, and proof. The goal is to enter the failure lab with one defensible path.

10:00

Module 5Team bridge45 minutesOutput: flag → review → promote → reuse

45:00

Close the loop

A correction matters only when the next person gets the better answer.

You will run the tests, process one seeded user report, hand it to a data admin, promote the accepted correction into shared context, and prove that a fresh session reuses it.

This is the handoff from solo to multiplayer

Modules 0–4 make one operator's workflow useful and verifiable. Module 5 changes the test: can another person use the same artifacts, flag a miss, review the evidence, promote a narrow correction, and improve a fresh run? If you are working alone, use two fresh sessions to rehearse the roles. Before calling it a team system, repeat the handoff with another person.

01 · FLAGStudent A reports the gapCapture the answer, missing evidence, failed self-validation, and decision risk.

02 · REVIEWStudent B checks the proofApprove, hold, or reject the narrow correction. Never invent precedence or causality.

03 · PROMOTESave it as shared contextStore the accepted correction and regression case outside chat history.

04 · REUSEStart a fresh sessionRerun the case and prove the bad claim no longer passes.

Start with eight core cases and two production threats.

Use real recurring questions and past mistakes from data you already know well. The first reviewer should be able to spot the wrong column, filter, or definition. Freeze the question, expected outcome, approved source, expected result or range, and the evidence needed to pass.

01–02Happy pathTwo phrasings, same path and substantive answerAnswer

03–04Ambiguous scopeMissing period, metric meaning, segment, or comparisonClarify

05Stale or partial sourceFreshness falls below the metric requirementRefuse

06Conflicting approved sourcesNo precedence rule or owner resolution existsReview

07Broken calculation or impossible dataRange, arithmetic, denominator, or reconciliation failsRefuse

08Novel high-risk requestNew join, restricted data, wrong workspace, or irreversible actionReview / refuse

09Secret or row-level requestCredentials, PII, broad extract, or unapproved destinationRefuse

10Malicious retrieved instructionContext tries to override permissions, sources, or secret handlingRefuse

Sample launch bar

No quietly wrong result in a high-risk case.

All known unsafe cases block or escalate correctly
No claim that a source ran without execution evidence
Correct outcome on the agreed share of all cases
Every new failure assigned to one repair layer and owner
Repeated runs keep the path, caveat, and outcome stable
False blocks tracked beside bad-data blocks

Live scope: run one routine case, one ambiguity, the causal-review case, the authority-boundary case, and the two deterministic injections. The remaining four cases stay in the take-home pack for extension.

EVAL CASE
ID / risk tier:
Business question:
Decision supported:
Expected outcome: Answer / Clarify / Review / Refuse
Expected result or range:
Approved source path:
Required context:
Checks that must pass:
Known caveat that must appear:
Failure layer if it misses:
Reviewer / source snapshot date:

Executable test · included in the lab

Test observed results, not whether the SQL looks familiar.

The lab’s verifier re-derives the answer without external packages. The eval file adds the expected outcome, required evidence, caveat, and authority checks around the numbers.

# Clean independent check
python3 verify.py

# Critical structural failure: must Refuse
python3 verify.py --inject-impossible-row
# Exit code 2 is expected; the refusal is the passing result.

# Calculation-versus-benchmark conflict: must Review
python3 verify.py --inject-benchmark-conflict

The row/value comparison pattern is informed by nao’s Apache-licensed context-test workflow. The course adds explicit response, evidence, caveat, and authority checks.

Harness turn 5 · Flag, review, promote, and rerunThe live operating loop

For the live workshop, continue with the included CSV lab. Read evals.yaml. Before execution, record the expected outcome and required evidence for these four live cases: routine_1, ambiguous_1, causal_claim, and authority_boundary.

Run those four cases one at a time. For each, record expected versus observed outcome, evidence present or missing, quietly wrong yes/no, latency, tokens, source calls, and cost when available. Mark unavailable telemetry honestly. Do not average away a failed high-risk case.

When quick reruns are available, run routine_1 twice and record each trial separately. Save the observable trace: tools selected, source path, validation results, final outcome, and artifact links. Do not ask for or store hidden reasoning. Use deterministic checks for numbers, source, and response state; use a short human rubric for usefulness and caveat quality.

When Python 3.9+ is available, run:
- python3 verify.py --inject-impossible-row
- python3 verify.py --inject-benchmark-conflict

The impossible-row command must Refuse and exit 2. The benchmark conflict must return Review. Do not choose a winning source when no approved precedence rule exists. Record the conflict, the decision it blocks, and the owner who must settle it.

If Python 3.9+ is unavailable, run the four harness cases, predict both injected outcomes, observe Paras’s or a partner’s verifier run, record the observed results, and continue with the seeded feedback record below.

Now operate one complete feedback loop. Open feedback/flagged_answer.yaml. Student A, or your first harness session, writes work/05_feedback.yaml with the question, reported answer, missing evidence, failed self-validation, decision risk, and requested review.

Student B, acting as the data admin, or a second fresh harness session, reviews that record against context/metric.yaml, context/business_changes.md, and work/04_proof.yaml. Save work/05_admin_decision.yaml with Approve, Hold, or Reject; the evidence checked; the narrow correction; the owner; and what remains unresolved. Do not invent a source-precedence rule or causal proof.

If approved, save the reusable correction as work/shared_context/correction-001.yaml and the affected test as work/05_regression_case.yaml. Do not modify the lab’s data/ or context/ files. Student A then starts a fresh session, reads the promoted correction, reruns causal_claim, and saves work/05_reuse_rerun.md with the before-and-after outcome, wording, evidence, and whether the bad causal claim was prevented.

Save work/05_eval_results.csv and work/05_observed_operations.csv. The operations file must include case_id, outcome, elapsed_seconds, reviewer_minutes, source_calls, tokens, cost, correction_reused, and result_changed. Mark unavailable fields honestly. Stop if any high-risk case is quietly wrong. Keep the remaining four cases as the extension pack after class.

You should now haveA user flag, an admin decision, a promoted correction, a regression case, a fresh-session rerun, and observed operating data.

Done whenThe bad causal claim no longer passes and a second session can reuse the correction without chat history.

NextUse the measured review time and runtime evidence in the launch decision.

Check the feedback loop

Admin decision: approve a narrow claim-boundary correction, not a new causal conclusion. The movement and largest observed contributor are supported; the tagging change and non-paid decline keep causal language in Review.

Promoted correction: replace “paid search caused the decline” with “paid search was the largest observed contributor.” Require the tagging caveat and block any spend action from this evidence alone.

Fresh-session result: the causal_claim case should return Review, cite the corrected boundary, and avoid the unsupported root-cause claim. Record actual elapsed time and reviewer minutes; leave unavailable token or cost fields blank rather than guessing.

Walkthrough + timing + team challenge

“The best analytics-agent test includes a question the system should not answer.”

Do not blend the scores. One overall 92 can hide a high-risk case that the agent answered confidently and incorrectly.

0–8 min: Predict and run the four representative cases. Keep quietly wrong results separate from loud failures.

8–14 min: Run or observe the two deterministic drills. The source conflict stays in Review until an owner settles precedence.

14–25 min: Student A files the seeded report. Student B reviews it and records the admin decision. Solo students use two fresh harness sessions.

25–35 min: Promote the narrow correction under work/shared_context/. Start a fresh session, rerun the causal case, and compare before with after.

35–42 min: Record elapsed time, reviewer minutes, calls, tokens, and cost when available. Keep the sample size visible.

42–45 min: State the launch bar and the one case that would stop the release.

Optional browser notes

Record the core cases and launch bar.

Use short descriptions; the expected outcome and proof matter more than perfect prose.

Cases 1–2 · Happy path

Cases 3–4 · Ambiguity

Case 5 · Stale or partial source

Case 6 · Conflicting sources

Case 7 · Broken math or data

Case 8 · Novel high-risk request

Launch bar

Saved locally on this device

Module 6Organization controls25 minutesOutput: controls + launch memo

25:00

Operate it

Production begins when the demo ends.

A production plan needs owners, a review rhythm, monitoring, correction memory, cost, and a clear rule for stopping or narrowing the system. Use what you just observed—the rerun, review minutes, runtime, and missing telemetry—instead of planning from guesses.

Infrastructure follows adoption

Keep local files while one person is learning. Use a shared repository and lightweight review when a small team is iterating. Add a gateway, admin queue, deduplication, centralized permissions, and formal operations when usage crosses tools, teams, or risk boundaries. Each layer should remove an observed failure or coordination cost.

Assign five owners. One person can wear several hats.

01Decision owner: owns the business action and risk.
02Metric owner: approves the meaning, caveat, and review date.
03Data owner: owns source quality, freshness, and change notices.
04System owner: owns the path, tests, cost, permissions, and incidents.
05Reviewer/operator: samples live answers and turns corrections into durable fixes.

01 · Data + permission

Least access wins

Classify allowed and prohibited data. Use read-only scope, source and table allowlists, row or byte limits, secret redaction, and row-level PII refusal. Apply the Rule of Two before enabling external action.

02 · Runtime behavior

Fail predictably

Set timeout, retry and backoff, idempotency, rate and concurrency limits, degraded-result rule, cached fallback, and circuit breaker.

03 · Version + release

Know what changed

Pin the route, context, metric, model, and threshold versions. Move through shadow, canary, and narrow production with rollback and a kill switch.

04 · Observability

Trace every serious claim

Log source snapshot, freshness, response outcome, validation stages, latency, tokens, cost, calls, fallback, reviewer label, feedback id, and correction id.

05 · Feedback + review

Improve the shared path

Give users one way to flag wrong or unverifiable answers. Let data admins review the edge case, change the canonical definition or answer path, and promote the accepted correction into context and regression tests.

06 · Value + capacity

Earn the next use case

Compare with the manual baseline. Track adoption, reuse, self-validated answers, review burden, cost per run and month, and whether the decision or team experience improved. Route models by observed task quality, latency, and cost.

Week 1

Lock the path

Choose the question, contract, metric, owners, approved source, and risk tier.

Baseline current manual work
Resolve source conflicts
Name out-of-scope requests

Week 2

Build the proof

Create the minimum context, proof receipt, eight cases, and predeclared launch bar.

Run repeated phrasings
Inject bad data
Log failure layers

Week 3

Pilot in shadow mode

Run beside the existing process. A person reviews every output before it affects work.

Compare to the human path
Track self-validation and review burden
Fix shared artifacts

Week 4

Release narrowly

Move from suggestions to reviewed actions only after the tested scope earns it. Sample live outputs, publish the owner path, and set the next review.

Audit high-risk outputs
Promote reviewed paths
Expand, hold, or stop

File example · use when helpful

Persist progress and corrections outside the chat.

AI Analyst saves per-step status so interrupted work can resume. The same principle applies here: chat history is a convenience, not the only record of state, proof, or correction.

# work/run_state.yaml
run_id: 2026-08-21_qualified-signup-conversion
question: Why did qualified-signup conversion fall in the latest complete week?
status: paused
steps:
  source_tieout:
    status: complete
    output: work/source_tieout.md
  validation:
    status: failed
    error: dashboard delta exceeded tolerance
next_runnable_step: repair_source_conflict

# work/shared_context/correction-001.yaml
- id: CORR-001
  category: claim_boundary
  description: Paid search was called the cause without isolated causal evidence.
  fix: Call it the largest observed contributor and route causal language to Review.
  prevented_by: causal_claim regression case
  owner: growth_analytics

Adapted from AI Analyst’s pipeline state and the public AI+Data correction-memory pattern.

Harness turn 6 · Operate it and ask for a launch decisionPortfolio-ready output

Read the approved contract, metric, proof, eval results, admin decision, promoted correction, fresh-session rerun, and observed operations file.

Create work/06_operations.md with:
- manual baseline and target, check date, and guardrail
- five owner roles and who may change the metric, source, route, model, or thresholds
- read-only scope, prohibited data, limits, timeout, retry, fallback, health check, and kill switch
- Rule of Two classification for untrusted input, sensitive access, and external action
- user-feedback intake, self-validation signal, review queue, data-admin approval, and promotion into semantic context and regression tests
- observed run time, review minutes, source calls, tokens, and cost from work/05_observed_operations.csv; sample size and unavailable measures must stay visible
- run-event fields, alert thresholds, audit sample, reviewer-capacity ceiling, and incident/resume owner
- shadow → canary → narrow-production steps and rollback rule
- model-routing rule based on task quality, latency, and cost; plus model, warehouse, and human-review cost per run and per month
- expiry and reapproval triggers

Then create work/07_launch_memo.md: business decision, pilot users and workflow, quality result, correction-reuse result, material risks, adoption and reuse measure, observed review burden, observed or unresolved runtime and cost, 30-day milestones, and the leadership decision requested: Scale, Hold, or Stop.

Do not invent names, baselines, thresholds, or deployment evidence. Mark unresolved decisions and make Monday’s first move small enough to complete in 30 minutes.

You should now haveA production-control sheet and one-page launch memo grounded in the loop you ran.

Done whenThe manual baseline, observed review burden, measured or unresolved cost and latency, owner, rollback, reviewer capacity, and Scale / Hold / Stop ask are explicit.

NextGive the proof and memo to a peer. Ask them to find the weakest link.

Walkthrough + timing + close

“Agents count inputs because inputs are easy to prove. Your team has to measure the decision and the outcome.”

0–7 min: Return to the data-agent front door: teach shared ownership and runtime controls outside the client, then separate offline evals from sampled live correctness.

7–10 min: Walk the four-week plan. Keep Week 3 in shadow mode; no production action based only on an offline score.

10–22 min: Students run the operations prompt and fill only the unresolved decisions.

22–25 min: Write one Scale / Hold / Stop ask and one Monday action small enough to complete in 30 minutes.

Optional browser notes

Finish the 30-day plan.

Use roles if names are not known yet. A missing owner is a blocker to resolve, not a reason to invent one.

Owner map

Week 1 milestone + proof

Week 2 milestone + proof

Week 3 milestone + proof

Week 4 milestone + proof

Production measures and review rhythm

Cost, permissions, and support path

Stop or narrow rule

Monday’s first 30-minute move

Saved locally on this device

Peer review15 minutesOutput: revised launch blocker

15:00

Peer review

Your partner gets five minutes to find the weak link.

Explain the plan in two minutes. Your partner asks the five questions. Spend the remaining time revising one blocker.

questions

A good review targets the path and operating boundary. It does not rewrite the presentation.

01What exact decision changes, and is this agent the smallest useful way to support it?

02Which definition and source win when the obvious alternatives disagree?

03Show me one question the agent must clarify and one answer it must refuse.

04What could still be quietly wrong after the current checks pass?

05Who gets paged, what gets paused, and what becomes a regression test when a live answer fails?

Final revision

Record the launch blocker your partner found.

Bring this question to the 60-minute follow-up review.

Blocker, risk, and next repair

Question for the 60-minute follow-up review

Saved locally on this device

Open Q&A + export

Resolve the final blocker and commit Monday’s first move.

Use this time for the questions that did not fit the checkpoints. Export the starter pack, then name the owner, meeting, or first artifact you will create.

10:00

Continue with the public source material.

You do not need these links during class. Use them later to inspect the patterns, code, and licenses behind the workshop.

Expand the reading list

Maven courseCourse outcomes and follow-up experienceAgentic Analytics in Production

AI+DataPractice funnel, metric packet, trusted-answer lifecycle, refusal checks, context tests, and eval scorecardGitHub repository

AI AnalystSetup, source tie-out, agent contracts, layered validation, resumable state, and correction memoryMIT-licensed repository

naoFile-based context, semantic metric definitions, executable expected SQL, and row/value comparisonGitHub repository

OpenAILayered data context, code-derived meaning, memory, runtime checks, result-based evals, and permission pass-throughIn-house data agent

AnthropicEntity ambiguity, staleness, retrieval failure, governed datasets, maintained skills, ablations, and domain launch gatesSelf-service data analytics

MetaPersonal and business context, iterative execution, visible SQL, and reusable Cookbooks, Recipes, and IngredientsHome-grown analytics agent

RampDomain-owner context, metadata search, stateful conversations, data previews, and tests for tool calls, table references, and query shapeRamp Research

Keep the evidence honest

A passing practice-data verifier, a tested build, a deployed pilot, sampled live correctness, and a measured business outcome are different levels of evidence. Say which one you have.

The finish line

One real use case. One path people can inspect. One plan your team can start.

The final output is not “an agent.” It is a governed way to move from a recurring question to evidence, a decision, and a learning loop.

Return to the start