Skip to main content

Data-Driven Testing

Data-driven testing runs the same test case repeatedly with different input data. Each row of a data source becomes one test iteration, and the column values are exposed as ${VariableName} tokens you can reference in your steps.

When to Use​

  • Testing an endpoint with many valid/invalid inputs
  • Verifying behavior across boundary values
  • Running the same scenario against different user profiles
  • Validating API responses with varied request payloads

Creating a Data-Driven Test Case​

  1. Right-click a test suite and select Add Data-Driven Test Case. (You can also upgrade an existing regular case later with the Convert to Data-Driven automation tool.)
  2. In the Test Case tab, configure the execution settings and attach a data source.
  3. Map data source columns to variables via mapping rules on your Send Request steps.
  4. Reference the variables in steps with the ${ColumnName} syntax.
  5. Set the Loop Start Step / Loop End Step to define which steps repeat per data row.

The Data-Driven Test Case Panel​

The editor has three tabs:

TabPurpose
Test CaseExecution settings, data source configuration, and data preview
Test InstancesOne row per generated iteration, each runnable and reportable on its own
ParametersPer-case variables (case scope)

Execution Settings (Test Case tab)​

SettingDescription
On Error BehaviorWhat happens when an iteration fails. Default is Resume Next — a data-driven case runs all rows and collects every failure, because rows are independent parameterized cases (one failing row does not invalidate the others). Switch to Stop to halt on the first failure.
Log LevelVerbosity of the run log
Parallel ThreadsNumber of iterations executed concurrently (1–8)
Loop Start Step / Loop End StepBounds of the steps that repeat per data row. Both may be left unset: a missing start means "iterate from the first step" and a missing end means "iterate through the last step", so an unset range iterates the whole case per data row.
Max RowsMaximum number of data rows to process (1–500)
DescriptionOptional notes about the test case

Steps outside the loop range run once — useful for authentication setup or teardown.

tip

The loop hint in the UI reminds you: "Steps outside the loop range run once — useful for auth setup or teardown."

Smart-Stop Guard​

A smart-stop guard aborts a data-driven run when the environment or credentials are uniformly broken, to avoid wasting time on rows that add no information:

  • It triggers after N consecutive same-kind errors of one class — environment (HTTP 502/503/504, timeout, connection refused, DNS, proxy) or authorization (HTTP 401/403, token/credential errors).
  • Business failures never trigger it — assertion rejects, HTTP 500s, and SOAP faults are the results the matrix exists to collect.
  • A different kind of error, a success, or an unclassifiable failure resets the streak.
  • When it fires, the run stops and the case records an explanatory message ("Smart-stop: aborted after N consecutive … errors") in its information and diagnostics, so reports and the AI can explain why it stopped.

Configure the threshold in Settings → General → Test Automation Behavior → Smart Stop (0 disables it), or override it per run on the CLI with aeth run … --smart-stop <n>. See General Settings and the aeth run reference.

Data Sources​

Attach a data source from the Data Source Configuration section of the Test Case tab. Supported types:

SourceMenu LabelBest For
CSVCSV Data SourceSimple tabular data, easy to edit, version-control friendly
ExcelExcel Data SourceComplex data with formatting, multiple sheets
SQLSQL Data SourceDynamic data from a database, refreshed per run
NoSQLNoSQL Data SourceDocument/NoSQL stores
TableTable Data SourceSmall inline datasets defined directly in the test case

CSV/Excel sources can declare a header row (Has Header). Use Load Data to preview the rows, or New Data Source to create one. A preview grid shows the parsed data so you can verify columns before running.

How It Works​

Given a CSV file users.csv with a header row:

Username,Password,ExpectedStatus
admin,secret123,200
guest,guest,200
hacker,wrong,401

The test case runs once per row. Each row's values are injected as ${Username}, ${Password}, and ${ExpectedStatus}. You bind them to request fields and assertions using mapping rules (ObjectPath-based) on the Send Request step.

Writing the ObjectPath​

A path is dot-separated navigation over the service's own fields, not XPath or JSONPath. Each segment is written the way the request body and the object tree show it — Body.photo_urls, Body.If-Match — and it is case-sensitive. When in doubt, right-click the field and choose Copy Object Path instead of typing it. The full guide, with examples and how to read a path error, is Object Paths.

Four things to know when a path does not resolve:

  • Case matters. Body.tags[1].NAME is not Body.tags[1].name. The error names the segment it could not find and lists the names that do exist on that type.
  • An index selects one element. Body.photos[0] is the first photo, and it works on every protocol's collection. Putting an index on a field that holds no elements is refused rather than quietly writing the whole field. The reverse mistake is refused too: a path that walks through a collection without saying which element (Body.photos.name) is an error — it used to hand back the whole collection and drop the rest of the path, so an extraction or a mapping rule reported success on the wrong value. Naming the collection and stopping there (Body.photos) still selects it whole.
  • Where the two spellings differ, the wire name wins. A schema that declares both my-field and my_field generates two members, one renamed; a path written my_field binds the field that actually travels as my_field. If two fields claim the same name, the path says so instead of choosing.
  • A polymorphic array is addressed by the tag its items carry. When the items of an array are different kinds chosen by a tag field — a bag that is either checked or carryOn — write Body.bags[type:checked].weightKg, where checked is exactly the value you read in the request body. Ask that array for a tag it never declares and the path is refused, listing the tags it does; ask it for a tag it declares and no element carries, and the write creates that element. Arrays whose items are all one kind are addressed by position (Body.tags[0]); on a SOAP service whose array is a choice group, the tag is the element name the schema gives each kind, which is again what the tree shows you (bags[type:hold]).

The full reference — how to copy a path out of the object tree, picking an item by position or by tag, worked examples and how to read every path error: Object Paths.

Test Instances​

The Test Instances tab lists every iteration generated from the data source. Each instance shows its name, step count, progress, information, and status, and can be run or reported individually. Disabled instances are skipped.

At run time the case generates one instance per data row (up to Max Rows). Each instance is relatively independent — it runs its own copy of the loop steps with its own row data — and the Parallel Threads setting decides how many of them execute at the same time.

Variable scope inside an instance​

An instance can read everything the case can: project, suite and test case variables, plus its own row values (a row value shadows a same-named variable from the upper scopes). What an instance writes depends on the level the writing step uses:

  • TestCase (the default for Data Transfer steps and extractions) — the value lands in that instance only. Other instances never see it.
  • TestSuite or Project — the value reaches the shared collection. Keep instances read-only at these two levels: with Parallel Threads above 1, several instances can write the same variable at the same time and the result is unpredictable (the surviving value is whichever thread wrote last). If a shared value has to be produced from data, write it in the before-loop steps — they run once, single-threaded, before any instance starts.

Report order​

The report lists instances in instance-number order — #2 after #1 — no matter how many threads ran or in which order the instances finished. When a run is stopped early, instances that never started carry no report entry.

Reports​

Data-driven reports show per-iteration results, including per-row status and timing, so you can see exactly which data combination failed. Instances appear in instance-number order regardless of finish order (see Report order).

See Also​

Last updated on