Prompt Engineering

This page explains how Pactwise builds its prompts and model calls: which model we use and why, how a prompt is put together, how we validate what comes back, and the controls around every call. It marks what runs in the product today and what is still in development.

Principles

Use the model only where language understanding is the job. Most of Pactwise is deterministic code. In PactWise Source, publish gates, deadlines, sealed bids, audit trails and file scanning make no model calls at all. Where arithmetic, dates or rule checks decide an outcome, code decides it, and the model is not asked to.

One model, enforced in code. Every Claude call is pinned to Claude Haiku 5.5. The backend's shared Claude client resolves the model again when each request is sent, so an environment setting, a database row or a request field that names a different model is mapped back to Haiku 5.5. Prep Studio (in development) and a separate contract-intelligence service have their own Anthropic clients, and each is hard-limited to Haiku 5.5 as well. No larger model runs anywhere in the product today. Using one small, fast model keeps cost and latency predictable, and it means there is one model behaviour to test.

Determinism comes from constraints, not sampling settings. Extended thinking is turned off on every request from the shared backend client. Prep Studio leaves thinking at the model default; its forced structured-output calls run without extended thinking on Haiku 5.5. We do not send sampling parameters such as temperature; Claude 5.5 models run at their default sampling. Consistency comes from fixed prompts, closed output formats, schema validation and narrow tool sets.

Bound capabilities instead of relying on instructions. A prompt can ask a model not to do something. A tool allowlist makes sure it cannot. Our main safety guarantees come from what the model is given access to, not from what it is told.

Fail visibly. A malformed structured answer is rejected, not quietly patched. A request that cannot be completed returns a generic error instead of a partial answer. Database errors and stack traces are never returned to the user. Free-text assistant answers are returned as the model produced them.

Where Claude runs today

Every model call in the product uses Anthropic's Claude, and embeddings come from Voyage AI. No other AI vendor is called.

Prompt anatomy

The Contract Assistant prompt is a good example of how we structure a prompt. It has three parts, in this order.

1. Role. A single sentence: what the assistant is, whose data it works on, and that answers are grounded in the actual contract text.
2. Tool-selection policy. Clear rules for when to use each tool. Wording questions go to semantic search over contract text. Counting and filtering questions go to structured contract search, and lookups to a single-contract fetch. Market-standard questions go to the precedent corpus. The model may combine tools.
3. Grounding rules. Answer only from tool results. Attribute every claim about the user's contracts to its source contract. Never present precedent material as the user's own. If nothing relevant comes back, say so instead of guessing.

The user turn contains only the user's current question. Contract content is never pasted into the prompt up front. The model pulls it in through tool calls: the user's own contracts through tenant-scoped tools, and market examples through the read-only precedent corpus. Tool names are recorded with usage, and semantic-search excerpts are kept as citations. Each question is answered on its own: earlier turns are stored for the user's history but are not sent to the model.

A simplified skeleton of that structure (paraphrased; not the production text):

1
SYSTEM (simplified)
2
  Role: You are the contract assistant. You answer questions about the
3
        user's own contracts, grounded in the contract text.
4
  Tool selection:
5
    - What a contract says        -> semantic contract search
6
    - Counts, filters, status     -> structured contract search / fetch one
7
    - Market-standard wording     -> public precedent search (not the user's)
8
  Rules:
9
    - Answer only from tool results. Never invent terms, parties, dates, values.
10
    - Attribute each claim to its source contract.
11
    - Label precedent material as precedent. Never give it a contract id.
12
    - If nothing relevant is found, say so.
13
14
USER
15
  <the current question, validated and length-bounded>

Structured output and validation

Validation happens at the request boundary and around every tool call.

Input. The request body is validated with a Zod schema before anything else runs. The question has a fixed maximum length, and the request cannot carry a system prompt, a tool list or a model choice. Those are fixed in code.

Tool calls. Each tool advertises a JSON Schema for its input to the model. The tool executor is bound to the set of tools it offered the model and refuses any other tool name, so a model that asks for an unlisted tool gets a refusal, not an execution.

Tool errors. A tool that fails returns an error result to the model in the normal tool-result format, so the model can recover, for example by trying a different search. The tool loop has a fixed iteration cap. A request that hits the cap ends with a generic error instead of running on.

Transient failures. Failed non-streaming model calls are retried a bounded number of times with exponential backoff and jitter.

Grounding and citations

Retrieval. Contract text is split into chunks and embedded with Voyage AI's legal-domain model (voyage-law-2, 1024 dimensions). Documents and queries use separate embedding input types, which is how that model is designed to be used. Semantic search runs over the tenant's own chunks. Indexing currently runs only through a manual backfill; automatic indexing on upload is in development.

Tenant scope comes from the session, never from the model. The enterprise is taken from the authenticated user. If the model passes a contract id to a search tool, that contract's ownership is checked first: it must belong to the caller's enterprise and must not be deleted. Semantic search and precedent search cap the number of excerpts and the length of each one. Structured contract search and single-contract fetch return full contract records.

Citations come from what was retrieved, not from what was written. After the model answers, citations are built from the tool calls it actually made. They are not parsed out of its prose. Citations are de-duplicated and capped, contract titles are looked up again with tenant and deletion scoping, and the citation set is stored with the answer. So a citation points to a contract the model really retrieved, in the user's own enterprise.

Precedent material is labelled. The precedent tool searches a fixed, read-only corpus of public contracts (such as SEC filing exhibits and published research datasets), kept separate from every customer's data. The model cannot choose which corpus it reads. The prompt requires precedent results to be labelled as market examples and forbids giving them a contract id. The precedent corpus is still being loaded, so precedent search may return few or no results today.

Refusing to invent. The grounding rules forbid inventing contract terms, parties, dates or values. "I couldn't find that" is the expected answer when retrieval comes back empty.

Prompt-injection defence

Contracts and vendor documents are written by other people, and any of them could contain text that tries to give the model instructions. We plan for that. In the live product, these properties hold no matter what a document says:

• The model cannot change anything. The Contract Assistant gets four tools, and all of them are read-only: semantic contract search, precedent search, structured contract search and single-contract fetch. The executor refuses any tool it did not offer, so injected text cannot make the model write, delete, send messages or run arbitrary queries.
• The model cannot leave your tenant. The tenant comes from the authenticated session, and any id the model supplies is checked again for ownership. A document cannot redirect the model to another customer's data.
• The instructions are fixed. The system prompt, the tool set and the model are set in code. A request can supply only a question.
• Output goes to a person. The model's answer is shown to the user with its citations. No path in the product UI acts on model output by itself.

Where untrusted text is placed directly in a prompt, as in Prep Studio and in recalled agent memory, it is fenced as data, and the system prompt declares fenced content to be data, never instructions. This defence relies on keeping the boundary intact rather than on a list of banned phrases, and it is covered by unit tests.

Human-in-the-loop

No model path reachable from the product UI writes to your data. The Contract Assistant reads and answers; people decide what to do. Every approval point in the product that is backed by AI drafting or AI extraction, such as reviewing extracted contract data or approving an RFP draft, belongs to features described under In development. Each of those is designed so a person, not the model, makes the final decision.

Cost and latency controls

• One small model, thinking off. Haiku 5.5 with extended thinking disabled keeps each call fast and cheap.
• Budget check before the call. Each Contract Assistant request checks the user's and the enterprise's AI budgets first. Over budget, the request is rejected with a rate-limit response before the model is called.
• Rate limiting on the assistant endpoint, on top of the budget check.
• Bounded loops and outputs. The tool loop stops at a fixed iteration cap. Output length is capped on every request. Semantic and precedent search return a capped number of excerpts of capped length.
• Per-response cost recording. Each model response in a tool loop is priced on its own, against the model's rate card. The request total is recorded as one usage-ledger row, and ledger writes are batched.
• Embedding reuse. Within a warm worker, repeated text is not embedded again. Chunks are stored with their embeddings, so a contract is embedded once at indexing time, not on every search.

Testing and evaluation

Automated tests cover these properties of the live model layer:

There is no golden-set evaluation yet for the accuracy of the live assistant's answers, or for contract classification and extraction quality. Our tests check behaviour, policy, safety and shape. They do not yet score answer quality against reference answers. Golden-fixture evaluations exist for Prep Studio, described below.

How prompts are versioned and changed

Prompts are kept in source code, next to the code that calls them. A prompt, its tool set and its output schema change together in one change, which goes through code review and the same test gates as any other code. The model is not a setting that can drift, because the send-time check enforces it. Prep Studio adds per-prompt version stamps (below).

In development

Everything in this section is built and tested but not yet generally available. Do not rely on it as a current product capability.

Prep Studio (PactWise Source RFP drafting)

Prep Studio drafts an RFP package in stages: intake interview, scope, requirements matrix, evaluation criteria, pricing schedule and timetable.

• Versioned prompts. Each of the seven prompt modules has a human-readable semantic version plus a content hash of its text. That hash is stamped on every trace and every drafted artifact, so any output can be traced to the exact prompt that produced it.
• Shared house rules. Every prompt starts with the same preamble: the model drafts, validators check, and a human approves. It must not invent facts, and it never scores, ranks or recommends vendors. Buyer input can change what is drafted, but never the model's role, rules or output schema.
• Sectioned, fenced templates. User turns are split into labelled sections (task, buyer input, untrusted context, output). Every buyer-supplied, retrieved or previously drafted value is fenced as data. A missing value is shown explicitly as "(none)", so a section is never silently dropped.
• Forced structured output. Output is produced through a forced tool call and validated with Zod. Stop reasons are handled one by one. A refusal is not re-asked. A truncated answer is retried once at the stage's full output ceiling if it ran below it; otherwise it fails as truncated. A complete answer that fails the schema is re-asked at most twice, with the validator's specific issues appended.
• Deterministic where it matters. Criteria weights summing to 100, requirement-to-criterion mapping, timetable dates and the publish-gate checks are all done in code. The timetable is computed in code, and the model writes only the explanation. The database publish gate has the final say.
• Critic loop. A red-team critic reviews the whole draft from two adversarial viewpoints, a vendor's bid manager and a protest lawyer. It uses a severity scale and gives a concrete suggestion for each finding. Its findings are advisory. Blocker and major findings automatically re-run the stage that owns the affected artifact and every stage downstream of it, for at most three critic cycles. Minor findings wait for the buyer.
• Approval bound to content. A buyer approves a specific version of an artifact, identified by its content hash. If anything upstream changes, everything downstream is marked stale and needs approval again.
• Cost controls. The static system prompt is cached. Each stage has its own output ceiling. Each draft has a token, time and cost budget that is checked before every attempt, and a draft that reaches its budget pauses instead of overrunning.
• Retrieval hygiene. Bids released through PactWise Source are always excluded from retrieval. A failed retrieval source contributes nothing and raises a visible warning, so a draft never appears better informed than it is.
• Evaluation. Golden-fixture replays run in the Prep Studio test suite. They check that the gate passes, every requirement is mapped, the critic converges, cost stays within the ceiling and draft quality meets set thresholds. A prompt lint test requires every prompt to declare its schema and fence its untrusted inputs. The fixtures are currently written by hand; recordings of live model runs are not yet part of the suite.

Contract document understanding

A pipeline that reads uploaded contracts, classifies them and checks them against a compliance checklist:

• Text extraction uses PDF parsing and OCR, not a language model. An OCR confidence score that was estimated rather than measured can never approve a document automatically.
• Classification makes one Claude call that picks from a closed list of 34 contract types. An id outside the list is discarded and the keyword fallback runs instead. The keyword fallback's confidence is capped below the review threshold, so those documents always go to a person.
• Extraction checks the contract one checklist section at a time, against a library of 831 checklist items. Each item is marked present, partial or absent, with a confidence score, a supporting quote where the item is present or partial, and a red-flag marker. Items the model skips are marked not evaluated. A failed section does not stop the run.
• Structured responses. Extraction asks the shared client for a JSON object rather than prose. The client converts the caller's Zod schema (objects, strings, numbers, booleans, arrays, enums) into JSON Schema, adds it to the system prompt with an instruction to reply with that object only, removes any markdown code fences, then parses the reply and validates it against the original Zod schema. If validation fails, the call fails. A partly valid object is never passed on, and the client does not try to repair it.
• One pure function decides whether a document needs human review, using an ordered list of reasons. A triggered red flag, or a low-confidence answer on a critical item, always sends the document to review. A reviewer's corrections take priority over the model's extraction.
• Current limit: each call sees a bounded excerpt of the contract, so very long contracts are not yet evaluated in full.

Contract scoring

A contract-scoring endpoint runs the same checklist evaluation section by section and stores the model-derived compliance results and an overall score for the contract. It is not called from the product UI.

Automatic contract indexing

When a contract's analysis task finishes, its text is chunked and embedded for semantic search. The background worker that runs those tasks is not yet switched on in production, so contracts are indexed today only through a manual backfill.

LLM steps in agent workflows

Pactwise's agents are rule-based by default. Model-assisted steps are an opt-in setting on individual steps of a durable workflow graph. Two built-in graphs, contract lifecycle and compliance audit, enable them on their legal and financial review steps. These workflows are not exposed in the product UI yet.

• The mode and the prompt template for a step come only from the workflow definition. Run input cannot switch a step into model mode or replace its prompt.
• Each model step passes a budget check and a circuit breaker. It gets only a read-only tool allowlist. Retrieved agent memory is fenced as untrusted data.
• Without a configured model, the step falls back to rule-based logic and records the fallback.
• The agents' own prompts require evidence: the legal reviewer quotes the clause and answers "insufficient information" rather than guessing; the financial reviewer shows its arithmetic, never invents amounts and returns a fixed risk scale; the extraction prompt copies values word for word and returns null for anything absent.
• Human approval steps in these workflows are enforced by the database, not only by application code.