Tests

Test config: tests.yaml

A tests.yaml file defines the testbenches and tests for a verification suite. Each suite has its own tests.yaml.

rtl_buddy looks for tests.yaml in the current directory, or you can specify a file with --test-config.

Structure

rtl-buddy-filetype: test_config

testbenches:
  - name: "tb_top"
    filelist:
      - "+incdir+../../../verif/tb"
      - "tb_top.sv"

tests:
  - name: "smoke"
    desc: "sanity test"
    reglvl: 0
    model: "my_design"
    model_path: "../src/models.yaml"
    testbench: "tb_top"
    plusargs:
      test_cycles: "50"
    plusdefines:
      FEATURE_X: "1"
    sim_timeout: 120

Test fields

Field	Description
name	Test identifier used on the command line and in log file names
desc	Human-readable description
reglvl	Regression level (int or per-builder dict)
model	Model name from models.yaml
model_path	Path to models.yaml, resolved relative to the suite directory
testbench	Testbench name from testbenches list
plusargs	Key-value pairs passed as +KEY=VALUE at sim runtime
plusdefines	Key-value pairs passed as +define+KEY=VALUE at compile time
sim_timeout	Timeout in seconds (default: 60)
uvm	UVM report thresholds (see below)
sweep	Sweep expansion script (see Plugins)
preproc	Pre-processing script (see Plugins)

Regression levels

reglvl controls which tests run during a regression:

# Same level for all builders
reglvl: 1500

# Builder-specific, with a fallback
reglvl:
  default: 2500
  vcs: 3500

Use --reg-level and --start-level on the regression subcommand to select a level range. See Regressions.

Default transcript parsing

When uvm is not set, rtl_buddy determines the result by parsing logs/{test_name}.log after simulation. Your testbench must print a result marker to stdout at the start of a line:

• PASS <optional detail>
• FAIL <optional detail>

When emitting FAIL, also print an ERR: or FAT: line. The default failure parser expects one:

if (test_passed) begin
  $display("PASS smoke completed");
end else begin
  $display("FAIL smoke completed");
  $display("ERR: expected done=1 before timeout");
end

Rules to follow:

• Emit exactly one terminal result marker.
• Start the line with PASS or FAIL; other wording will not be detected.
• Write the marker to stdout, not stderr.
• When using FAIL, follow it with an ERR: or FAT: line.
• If no PASS or FAIL marker is found, rtl_buddy records the test as NA with description test result unknown.
• Do not rely on the simulator exit code alone to communicate pass/fail in non-UVM tests.

UVM report parsing

When uvm is set, rtl_buddy parses the UVM summary at the end of simulation output and fails the test if thresholds are exceeded:

uvm:
  max_warns: 0
  max_errors: 0

With uvm enabled, rtl_buddy uses the UVM Report Summary instead of PASS / FAIL transcript markers. Missing or malformed UVM summaries are treated as test failures.

Other failure modes

The transcript parser is not the only source of failures. rtl_buddy also marks a test as FAIL when:

• a sweep or pre-processing script fails during setup
• filelist validation fails before compile
• compilation fails
• simulation times out

Running tests

Run a named test:

rtl-buddy test smoke

Run all tests in a config:

rtl-buddy test

List tests without running:

rtl-buddy test --list

Randomization

Two seed options are available with the test subcommand:

• --rnd-new: use a randomly generated seed instead of the root config seed. The seed is saved to logs/{test_name}.randseed.
• --rnd-last: repeat the test with the seed from the last --rnd-new run.

For running a test many times with different seeds, use randtest. See the CLI reference.

Logging

rtl_buddy writes orchestration output to rtl_buddy.log in the directory where it is invoked.

Per-test simulation output goes to:

• logs/{test_name}.log — full simulation output
• logs/{test_name}.err — stderr
• logs/{test_name}.randseed — the seed used

The symlinks test.log, test.err, and test.randseed in the current directory always point to the most recent test run.

For machine-readable logs (JSON Lines), use --machine. See For Agents.

Path and working directory

test and randtest do not automatically change directory to the suite directory. Run them from the directory containing tests.yaml, or pass an explicit --test-config path.

Paths in tests.yaml (such as model_path) are resolved relative to the suite file's directory, not the invocation directory.

Full schema

See YAML Formats: tests.yaml for the complete field reference.