Test case format

Who this is for: people and AI assistants who write or review workflow and action tests.

Before you start: read Read and write workflow tests for the task itself. This page is the exact format.

Every workflow version carries its own test cases, and so does every version of an action in your library. A test case says what goes in (the trigger, mocked answers from other systems, dummy secrets) and what must come out. Cases run on every save, against mocked HTTP only. A version can be activated only when all its cases pass on that exact version and the required minimum is met.

A version’s tests are a JSON list of cases. The JSON Schema is available to AI assistants as the MCP resource liitora://schemas/test-case.json.

A complete example

These three cases belong to a workflow that fetches open orders every Monday, builds a CSV and emails it (the Liitora template “Weekly open orders report”). They pass and meet the required minimum.

[
  {
    "name": "Emails the sales team a CSV of the open orders",
    "description": "Two open orders lead to one email with a two-row CSV attached.",
    "kind": "happy",
    "given": {
      "now": "2026-10-05T07:00:00+03:00",
      "http": [
        {
          "match": { "method": "GET", "path": "/api/v1/orders" },
          "respond": {
            "status": 200,
            "json": {
              "orders": [
                { "id": "SO-1001", "customer": "Example Oy", "total": 120.5 },
                { "id": "SO-1002", "customer": "Sample Ab", "total": 80 }
              ]
            }
          }
        }
      ]
    },
    "expect": {
      "outcome": "success",
      "recipients": ["sales@example.com"],
      "payload": [
        { "path": "$.steps.report.outputs.rows", "equals": 2 },
        { "path": "$.emails[0].subject", "equals": "Open orders: 2" },
        { "path": "$.emails[0].attachments[0].name", "equals": "open_orders.csv" }
      ],
      "http_calls": [{ "method": "GET", "path": "/api/v1/orders", "url_contains": "status=open", "count": 1 }]
    }
  },
  {
    "name": "Fails cleanly when the order system is down",
    "description": "The order system answers 503: the run fails at the fetch step and nobody gets an empty report.",
    "kind": "upstream_failure",
    "given": {
      "http": [
        { "match": { "method": "GET", "path": "/api/v1/orders" }, "respond": { "status": 503, "json": { "error": "maintenance" } } }
      ]
    },
    "expect": { "outcome": "error", "failed_step": "fetch", "recipients": [] }
  },
  {
    "name": "Sends nothing when there are no open orders",
    "description": "An empty list skips the report and the email.",
    "kind": "empty",
    "given": {
      "http": [
        { "match": { "method": "GET", "path": "/api/v1/orders" }, "respond": { "status": 200, "json": { "orders": [] } } }
      ]
    },
    "expect": {
      "outcome": "success",
      "recipients": [],
      "step_outputs": [
        { "step": "report", "result": "skipped" },
        { "step": "email", "result": "skipped" }
      ]
    }
  }
]

For a workflow started by a webhook, add a case with a bad payload:

{
  "name": "Refuses a ticket without an email",
  "description": "A payload without an email fails the check and nothing is sent to the service desk.",
  "kind": "malformed_payload",
  "given": {
    "trigger": "webhook",
    "event": { "body": { "subject": "No email here" } }
  },
  "expect": {
    "outcome": "error",
    "failed_step": "check",
    "http_calls": [{ "path": "/api/tickets", "count": 0 }]
  }
}

Case

Field	Type	Required	What it is
name	text, 1 to 200 characters	yes	A plain-language name, shown to the people who approve the workflow.
description	text	no	What the case proves, in plain language.
kind	happy, upstream_failure, empty, malformed_payload or edge	no	Which part of the required minimum this case covers. Liitora also recognizes cases by their shape.
given	object	no	What goes in. See Given.
expect	object	yes	What must come out. See Expect.

Unknown fields are errors, with their path, for example given.http[0].match.metod.

Given

Field	Type	Default	What it is
event	any	none	The trigger payload, read as trigger.* in expressions and ctx.event in scripts. For a webhook: { "body": ..., "headers": ... }.
trigger	manual, schedule, webhook, api or workflow_run	webhook if the workflow has a webhook trigger, else schedule if it has a schedule, else manual	Which kind of trigger started the run.
now	RFC 3339 time	2026-01-05T08:00:00Z	The frozen clock for now(), ctx.now() and dates.
secrets	object	generated	Dummy values by secret name: a text, or an object of fields such as { "username": "...", "password": "..." }. Secrets the version uses but the case doesn’t list get the dummy value test-dummy-<name>.
http	list of mocks	none	The answers other systems give.
models	list of model mocks	none	The answers AI models give to platform/ai.generate steps.
stub_steps	object	none	Steps that don’t run. See Stubbed steps.
state	object	none	The workflow’s saved state before the run, as read by state.get and ctx.state.get.
vars	object	none	Company variables for the run, as read by vars.<name>. Tests never read your company’s real variables.
lookups	object	none	Lookup tables for the run, by name, as read by lookup('<name>', key). Each is {"columns": [...], "rows": [[...]]}; the first column is the key. Tests never read your company’s real tables.
snapshots	object	none	What sync.diff steps with snapshot: true remember before the run, by step id. Each is {"records": [...]}: the records the step’s source held at an earlier successful run. Liitora hashes them with the step’s own key, compare, ignore and trim_strings, so write those as plain values. Without an entry the step starts with no snapshot. Tests never read or change your company’s stored snapshots.
input	any	none	Action tests only: the action’s input.
artifacts	object	none	Files the run already holds when it starts, by file name, for steps and actions that read a file. Each file has text (its content), an optional repeat (how many times text repeats, so a large file stays short in the test) and an optional content_type (default application/octet-stream). A name starts with a letter or digit and has only letters, digits, dots, dashes and underscores. At most 20 files, 16 MiB each and 32 MiB together. Use made-up content only.
connections	object	your real connection settings	Per connection name (workflow tests) or slot (action tests): base_url (must start with https://), auth (none, api_key, bearer or basic), header, and for an AI connection you don’t have yet provider (openai, azure_openai, anthropic, mistral or openai_compatible). Workflow tests can change a connection’s base_url this way.

Never put real secrets or personal data in a test. Liitora scans test data for anything that looks like a password or key before any case runs. A finding stops the tests and is shown as a problem.

Mocks

A mock answers requests that match it.

Field	Type	What it is
match.method	text	The HTTP method, in any case.
match.url	text	The full address, query included.
match.url_contains	text	A part of the address.
match.path	text	The path of the address. With a ?, also the query.
match.body_contains	text	A part of the request body.
respond.status	number, 100 to 599	The status. Default 200.
respond.json	any	A JSON body.
respond.text	text	A text body. Use json or text, not both.
respond.headers	object	Response headers.
respond.delay_ms	number, 0 to 120000	Wait this long before answering, to test timeouts.
times	number	How many requests the mock answers. Default: any number.

match needs at least one of url, url_contains and path. Every field that is set must match.

When several mocks match, the most specific one wins: url counts 4, path 2, url_contains 1, and method and body_contains 1 each. On a tie, the mock listed first wins. A mock with times stops answering after that many requests, so the next matching mock answers.

A request without a matching mock is refused (http_unmocked), and the step fails. The case fails too, unless it sets expect.no_unmocked_http: false. Nothing in a test can reach the real network.

Model mocks

A model mock gives the answer an AI model returns to a platform/ai.generate step. You write the answer itself; Liitora turns it into the provider’s own reply, so the step reads it exactly as it reads a real one. Nothing reaches a real model, and no tokens are spent.

"models": [
  { "connection": "brain", "output": { "label": "bug" }, "usage": { "input_tokens": 40, "output_tokens": 5 } }
]

Field	Type	What it is
connection	text	The AI connection’s name, as the step binds it in connections:. Required.
output	any	A complete answer with this JSON. It is checked against the step’s output_schema, so an answer that doesn’t fit fails the step as it would in a real run.
text	text	A complete answer with this raw text, to test an answer that isn’t JSON.
refusal	text	The model refuses to answer, with this message.
truncated	true	The answer is cut off at the token limit.
error	object	The provider answers with an error: status (400 to 599) and an optional message.
usage	object	The tokens the answer reports: input_tokens and output_tokens. Default 10 and 5.
prompt_contains	text	Only answer requests whose body contains this text.
times	number	How many requests the mock answers. Default: any number.
delay_ms	number, 0 to 120000	Wait this long before answering.

Use exactly one of output, text, refusal, truncated and error. A model mock answers at the connection’s own chat address: your real connection’s address, or https://<name>.connection.test for a connection you don’t have yet. An AI connection you don’t have yet stands in for an OpenAI-compatible server unless you set given.connections.<name>.provider. A mock for a connection no step uses, or for a connection that isn’t an AI connection, stops the case with a clear message.

An error of 429 or 5xx is retried by the step, as in a real run, but without waiting. List the failing answer with times: 1 before a good answer to test a retry that works.

Stubbed steps

"stub_steps": {
  "tickets": { "outcome": "failure", "error_code": "http_503", "message": "The service desk is down." }
}

A stubbed step does not run. Its result is recorded as given.

Field	Type	Default	What it is
outcome	success, failure or skipped	success	The step’s result.
outputs	any	none	The step’s outputs, for the steps after it.
error_code	text	none	The error code for a failure.
message	text	none	The error message for a failure.

Steps that call another workflow (uses: workflow/...) must be stubbed. Otherwise the case fails with call_not_stubbed.

Expect

Field	Type	Default	What it is
outcome	success, skip or error	none (required)	How the run ends. skip means every step was skipped.
failed_step	step id	none	The step the run fails at. Only with outcome: error.
recipients	list of addresses	none	Exactly these addresses get email (to, cc and bcc together). [] means no email at all.
payload	one value check or a list	none	Checks on the result document.
http_calls	list	none	Checks on the requests the run made. See HTTP call checks.
step_outputs	list	none	Checks on one step’s result and outputs. See Step checks.
no_unmocked_http	true or false	true	Fail the case when a request had no mock.

A case that expects success must check something concrete: an email, a value, an HTTP call or a step. A bare outcome is not enough.

Value checks

Field	What it checks
path	Required. A JSONPath into the result document, such as $.steps.report.outputs.rows. A trailing .length counts items.
equals	The value is equal. Numbers are compared by value, so 2 equals 2.0.
contains	Text contains the text, a list contains the item, or an object contains these keys and values.
matches	Text matches this regular expression.
exists	true: there is a value. false: there is none.
length	A list or text has exactly this length.

Each check needs path and at least one of the others.

HTTP call checks

Field	What it checks
method, url, url_contains, path, body_contains	Which requests count, the same way as in mocks.
header_present	The request had this header, for example X-API-Key. Its value is never shown.
header_equals	The request had these headers with exactly these values, for example { "Content-Range": "bytes 0-9/10" }. Header names match in any case. Use it for headers that carry no secret.
count	Exactly this many requests matched. 0 means never. Without count: at least one.

Step checks

Field	What it checks
step	Required. The step id.
result	success, failure or skipped.
path	A JSONPath into the step’s outputs. Default $.
equals, contains, matches, exists, length	As in value checks.

The result document

payload checks read this document:

{
  "outcome": "success",
  "steps": {
    "<step id>": { "result": "success", "outputs": { }, "error": null }
  },
  "emails": [
    {
      "to": [], "cc": [], "bcc": [], "subject": "", "text": "",
      "attachments": [{ "name": "", "content_type": "", "size_bytes": 0 }]
    }
  ],
  "state": { },
  "artifacts": [{ "name": "", "content_type": "", "size_bytes": 0, "step": "" }],
  "error": null
}

Action tests also have output: the value the action returned.

The required minimum

Which cases a version must have depends on what it does:

Case	Needed when	A case counts when
Happy path	always	it expects outcome: success and checks something concrete
Upstream failure	a step binds connections:, uses platform/http.request, platform/webhook.send, platform/ai.generate, a library action or a workflow call, or a script uses ctx.connection or ctx.http	it has kind: upstream_failure, or a mock or model mock answers 5xx or 429 or has delay_ms, or a stubbed step fails
Empty or nothing to do	any step has if:, or a script calls ctx.skip(	it has kind: empty, expects outcome: skip, expects a step to be skipped, or expects recipients: []
Malformed payload	the workflow has a webhook trigger	it has kind: malformed_payload, or it has an event, no upstream failure, and expects error or skip

For an action, an upstream failure case is needed when the action has connection slots or its script uses ctx.connection or ctx.http, and an empty case when it calls ctx.skip(.

When something is missing, the report says which case to add and how.

How tests run

Each case runs in a fresh, closed world, through the same engine and sandbox as a real run:

• Every case is a test run in the Test environment, so trigger.environment is test and trigger.is_test is true.
• HTTP requests are answered only by the mocks. Connections use your real connection settings, never secret values: the Test setup of a connection when it has one, otherwise its Production setup. A connection that doesn’t exist yet gets a placeholder address, https://<name>.connection.test.
• Secrets are dummy values. Email goes to a test mailbox, files to test storage, state to a test copy.
• The clock is frozen at given.now.
• Your plan’s sandbox limits apply, as in production.

Differences from a real run, on purpose:

• One timeout_minutes unit is one second, and waits between retries are a few milliseconds. A platform/ai.generate step tries again without waiting.
• Connection-level retries and OAuth2 token requests don’t run. OAuth2 connections get a dummy token.
• Workflow calls must be stubbed.

Time budget: each case gets 20 seconds, and all cases together 60 seconds. Cases that don’t fit in the budget are reported as not run.

The report

The result is stored on the version and shown on the Tests tab.

Field	What it is
passed	true only when every case passed, the minimum is met and there are no problems. This decides whether the version can be activated.
summary	total, passed, failed and not_run.
minimum	met, what is needed, and what is missing with a plain message and a fix.
problems	Problems with the tests themselves, such as a credential found in test data.
cases	One entry per case: name, description, kind, status, duration_ms, the run’s outcome and error, and its checks.
ran_at, duration_ms, version_id	When and on which version the tests ran.

A case’s status is passed, failed, invalid (the case itself is wrong, for example a bad regular expression) or not_run.

Every check is written as a sentence, so people can review what the tests prove without reading code. For example: “Sends email to it@example.com”, “Calls POST /api/v1/devices/search with the X-API-Key header exactly once” or “Step report is skipped”. A failed check shows what was expected and what came out, with dummy secrets and sign-in headers hidden.

Next steps

• Read and write workflow tests
• Workflow file format
• Approve a workflow