Skip to main content

​
What are Libraries?

Libraries are reusable components — prompts or scripts — that you can define once and reference from multiple automation workflows. Each library item can have test cases so you can validate its behavior before deploying it.

​
Accessing Libraries

In the Automations section of your organization, click Libraries in the left sidebar. Library items are displayed in a grid, with each card showing the item name, a preview of its content, and the number of test cases.

​
Managing Library Items

Each card has a ⋮ (more actions) menu in the top-right corner with the following options: To open a library item, click the Open button at the bottom of its card.

​
Duplicating a Library Item

Choose Duplicate from a card’s ⋮ menu to create an independent copy of the item. The copy is named after the original with a (Copy) suffix — for example, Royalty Extractor becomes Royalty Extractor (Copy), and further copies are numbered (Copy 2), (Copy 3), and so on. The new item opens automatically so you can rename it or start editing right away.
Duplicating is the quickest way to create a variant of an existing prompt or script — copy a working item, then tweak the content or test data without touching the original.

​
Library Detail View

Opening a library item displays the detail view, which is split into two panels:
  • Left panel — the prompt or script content, plus an Output Schema builder for prompt-type items
  • Right panel — tabs for Test Data and Test Runs

​
Output Schema (Prompt Libraries)

For prompt-type library items, an Output Schema builder appears below the prompt editor. This works identically to the Transformer step’s Output Schema: you define the shape of the result object the model should return, and it is compiled into a strict schema at runtime. Defining a schema on a library prompt is useful when the prompt is shared across multiple Transformer steps that all expect the same output structure — you define the schema once on the library item, and it is applied wherever the library is loaded.

​
Syncing a Prompt to a Pipeline

You can push a prompt library item directly onto a Transformer step in one of your automations. This lets you refine and test a prompt in the library, then apply it to the workflow that uses it without copying and pasting.
Sync to pipeline is available for prompt library items only. Open the ⋮ menu on a prompt card and choose Sync to pipeline.
1

Open the Sync Dialog

On a prompt card, open the ⋮ menu and click Sync to pipeline. The dialog header shows whether you are pushing to your Test or Live environment, based on your current execution mode.
2

Choose a Composer

Select the composer that contains the target automation.
3

Choose an Automation

Select an automation within that composer. If the composer has no automations, a message is shown instead.
4

Choose a Transformer Block

Select the Transformer block to update. If the automation has no Transformer blocks, the dialog tells you there is nothing to sync into.
5

Review the Changes

A side-by-side comparison shows the block’s current prompt and output schema next to the new values from your library item, so you can confirm exactly what will change.
6

Push to Block

Click Push to block. The prompt and output schema are written to the Transformer block, and a new revision of the target automation is saved automatically.
Pushing replaces the Transformer block’s existing prompt and output schema. The target automation is saved as a new revision, so you can restore the previous version if needed.

​
Test Data

The Test Data tab lets you define test cases for a library item. Each test case has a name, a set of input variables, and an expected output.

​
Linking an LLM Block

For prompt-type library items, the top of the Test Data tab shows a linked LLM block card. Linking connects the prompt to a specific Transformer block in one of your automations. This unlocks the From a run test-creation method, which pulls real test data from that block’s runs.
1

Open the Link Dialog

If no block is linked yet, click Link block on the card. To change an existing link, click the pencil icon on the linked block card.
2

Choose a Composer

Select the composer that contains the target automation.
3

Choose an Automation

Select an automation within that composer.
4

Choose a Transformer Block

Select the transformer block this prompt represents, then click Link block.
Once linked, the card shows the prompt name alongside the linked block, with the composer and automation path beneath. Click the unlink icon to remove the link.
Linking is available for prompt library items only. The From a run option in Add Test appears only after a block is linked and that block has at least one run.

​
Adding a Test Case

Click Add Test to choose how to create a new test case: Each test case appears as a collapsible card.

​
Creating a Test from a Run

When you choose From a run, a picker opens so you can seed a test from a past run of the linked block:
1

Pick a Run

Select one of the linked block’s past runs. Each entry shows the run label, the environment (Test or Live), when it ran, and how many LLM blocks it captured.
2

Pick an LLM Block

Choose which transformer block from that run to seed the test from. Blocks without a captured snapshot can’t be selected.
3

Review the Captured Data

The picker shows the block’s resolved variables, the resolved prompt, and the actual output from that run. Optionally give the test a name.
4

Add the Test

Click Add Test. The captured variables become the test’s input variables, and the actual output becomes its expected output.
Linking a block and creating tests From a run is the fastest way to build realistic test data — you capture exactly the variables and output a block produced in a real workflow.

​
Naming a Test

Each test case has an editable name shown in its header. Click the pencil icon on a test card to edit the name inline. Press Enter to save or Escape to cancel.

​
Variables

Variables are the inputs used to resolve {{template}} placeholders in your prompt before execution. Each variable has a key — the template path (e.g. trigger.contract.name) — and a value.
1

Open the Variables Editor

In the test case card, click Manage Variables to open the variables editor.
2

Add a Variable

Click Add Variable and enter the key (template path) and its value. Values are JSON — objects, arrays, strings, or numbers.
3

Edit a Variable Value

Click the pencil icon next to any variable to open an edit dialog. Enter valid JSON and click Save.
4

Remove a Variable

Click the trash icon next to any variable to delete it.
Variable keys must match the {{template}} paths used in your prompt exactly. For example, a variable with key trigger.contract.name will fill in {{trigger.contract.name}} in the prompt text.

​
Expected Output

Each test case has an Expected Output field containing the JSON value you expect the library item to return. Click the pencil icon next to Expected Output (or the Set expected output button if none is set) to open an edit dialog. Enter valid JSON and click Save.

​
Seeding a Test from a Run

Rather than building a test case from scratch, you can seed a complete test — its name, variables, and expected output — from an AI block that actually ran in one of your past executions. The seeded variables and output are copied straight from the run, so the test reflects exactly what your prompt received and produced.
1

Open the Dialog

In the Test Data tab, click Add from Run. The Add Test from Run dialog opens.
2

Pick a Run

Choose a past run from the Run dropdown. Each option shows the run’s label, mode, when it started, and how many AI blocks it captured. Only runs in your current execution mode (Test or Live) that have captured AI block inputs are listed.
3

Pick an AI Block

Choose one of the run’s AI (Transformer) blocks from the LLM block dropdown. The label shows the step name, its automation, and — for blocks inside a loop or a paginated request — the iteration or page index. Blocks without a captured snapshot are shown but cannot be selected.
4

Review the Captured Data

The dialog loads the block’s captured Variables, its Resolved prompt, and the Expected output it produced during that run. Review them to confirm you picked the right block.
5

Name and Add the Test

Optionally rename the test in the Test name field — it defaults to a name based on the block and run date. Click Add Test to append it to the Test Data list, then run or edit it like any other test case.
Runs only capture AI block inputs from the point this feature shipped onward. If the Run dropdown is empty, run one of your automations and come back — the new run will be available to seed from.
Seeding from a run is the fastest way to build a realistic regression test: capture a prompt’s real inputs and output once, then keep re-running that test as you iterate on the prompt to make sure the output stays correct.

​
Adding Variables from a Run

Instead of typing test variables by hand, you can save a step’s real output directly from the execution log:
  1. Go to the Runs page and expand a run that has already executed.
  2. Expand a step that produced output.
  3. Click Save Output to Library, select this library prompt and the target test case, and confirm the variable path.
  4. The output is added as a variable on that test and is available immediately.
This is the fastest way to build realistic test data — capture actual outputs from real or test runs and feed them straight into your library tests.

​
Test Runs

The Test Runs tab executes the library item’s test cases and scores each one against its expected output. Every click of Run produces a run — all of the item’s test cases executed together and grouped under one numbered heading such as Run #3 · 4/5 passed · 0.94.

​
Running Tests

1

Open a Library Item

Click the Open button on a library card to open the detail view.
2

Add Test Data

Switch to the Test Data tab and add at least one test case. Define any variables needed to resolve your prompt’s template placeholders, and set an expected output — the score is measured against it.
3

Switch to Test Runs

Click the Test Runs tab in the right panel.
4

Run the Tests

Click the Run button. All of the item’s test cases execute together — in parallel — as a single run. While it is in progress the run header shows a progress bar and a live done / total count.
5

Review the Run

When the run finishes its header settles to Run #N · passed/total passed · score. Expand the run to see each test case, then expand a test case to see its field-by-field comparison.
The Run button is disabled until you have added test data. Add test cases in the Test Data tab first.

​
Runs

Each run is one execution of the whole test set, kept as a numbered entry so you can compare results as you iterate on the prompt: The newest run is expanded by default; older runs stay collapsed beneath it, giving you a history of how the prompt performed across edits.

​
Scoring

Each test case is scored by comparing its actual output to its expected output, field by field:
  • Every field in the expected output is compared to the same field in the actual output. A matching value scores 1; a text value scores by similarity (how close the two strings are); a field that is wrong or missing scores 0.
  • Fields that appear in the actual output but not in the expected output are ignored — the score only measures what you asked for.
  • The test score is the average across all expected fields, shown from 0.00 to 1.00. A test passes at 0.80 or higher.
The score appears as a coloured badge next to each test:
Scoring is deterministic — the same expected and actual output always produce the same score. It is a direct field comparison, not a second AI judging the result.
A test’s score is only as meaningful as its expected output. If you rename or restructure a prompt’s output fields, update the expected output on each test case too — otherwise those fields count as missing and the score drops.

​
Test Result Details

Expand a test case to see how it scored. A failed or errored test opens automatically so you land straight on the problem. Shown directly: Click Show details to reveal the run’s inputs — the Variables used and the Resolved Prompt (your prompt text after all {{template}} placeholders were filled).

​
Statuses and results

A test carries two separate signals — whether it ran, and whether it matched: An Error test has no score — there was no output to compare against. Fix the cause (often a missing template variable or invalid JSON) and run again.

​
Downloading a Run

You can download a completed run as a research zip to analyse it offline or share it. Two levels are available:
  • A whole run — hover over a run’s header to reveal the download icon, then click it to download every test in that run.
  • A single test — hover over an individual test row and click its download icon to download just that test.
Downloading is read-only — it is available to every member who can view the library, including viewers, unlike deleting. The download icon is disabled while a run is still executing, because its results are not yet complete.
Each zip contains the source material and the full results: Every test then gets its own folder containing:
When you download a whole run, its tests are sorted into high-score/, medium-score/, and bad-score/ folders so you can jump straight to the tests that need attention. Unscored and errored tests are left at the top level.

​
Deleting Results

You can remove runs in two ways:
  • Delete a run — Hover over a run’s header to reveal a trash icon, then click it to delete that entire run and its results.
  • Clear All — Click the Clear All button in the header to remove every run for this item at once.
Neither action affects the library content or test data.

Variables

Store reusable JSON values for use in automation steps.

Automations Overview

An introduction to automations and how they work.