Skip to content

Tests Functions

Test Subsystem

{testTools}

Tag filters

Prefix a tag with + for --tag or --skip-tag queries to add the meaning "previous AND".

  • --tag foo --tag bar means tests must have foo tag OR must have bar tag
  • --tag foo --tag +bar means tests must have foo tag AND must have bar tag (must have both)
  • --skip-tag foo --skip-tag bar means skip any test with foo tag OR with any test with bar tag (either)
  • --skip-tag foo --skip-tag +bar means skip any test with foo tag AND with the bar tag (must have both)
  • --tag a --tag +b --tag c --tag +d --tag +e is (a and b) or (c and d and e)

Your test functions can contain tags as follows:

# Tag: tagA tagB
# Tag: tagC
# Test-Platform: !alpine alpine !linux linux !darwin darwin
# Test-Skip: true
# Test-Housekeeper: false
# Test-Plumber: true
# Test-TAP-Directive: Something
# Test-After: testWhichShouldGoBefore
# Test-Before: testWhichShouldGoAfter
# Test-Fail: true
testThingy() {
   ...
}

Environment variables:

  • BUILD_TEST_FLAGS - SemicolonDelimitedList. Add flags like 'Plumber:false' to disable settings across tests.
  • BUILD_DEBUG - Many settings to debug different systems, comma-delimited.

Filters (--tag and --skip-tag) are applied in order after the function pattern or suite filter.

Location: bin/build/tools/test.sh

Arguments

  • --help - Flag. Optional. Display this help.
  • --clean - Flag. Optional. Delete test artifact files and exit. (No tests run) -l |- --list - Flag. Optional. List all test names (which match if applicable).
  • --env-file environmentFile - EnvironmentFile. Optional. Load one ore more environment files prior to running tests
  • --index-file indexFile - RealDirectoryFile. Optional. If supplied and exists, uses the index file for tests.
  • --cache-path cachePath - Directory. Optional.
  • --make-index - Flag. Optional. Generate the index file if supplied and quit.
  • --continue - Flag. Optional. Continue from last successful test.
  • -c - Flag. Optional. Continue from last successful test.
  • --delete directoryOrFile - FileDirectory. Optional. A file or directory to delete when the test suite terminates.
  • --delete-common - Flag. Delete ./vendor and ./node_modules (and other temporary build directories) by default.
  • --debug - Flag. Optional. Enable debugging for --junit (saves caches).
  • --verbose - Flag. Optional. Be verbose.
  • --stop - Flag. Optional. Stop after a failure instead of attempting to continue.
  • --coverage - Flag. Optional. Feature in progress - generate a coverage file for tests.
  • --no-stats - Flag. Optional. Do not generate a test.stats file showing test timings when completed.
  • --messy - Flag. Optional. Do not delete test artifact files afterwards.
  • --fail executor - Callable. Optional. One or more programs to run on the failed test files. Takes arguments: testName testFile testLine
  • --cd-away - Flag. Optional. Change directories to a temporary directory before each test.
  • --tap tapFile - FileDirectory. Optional. Output test results in TAP format to tapFile.
  • --junit junitFile - FileDirectory. Optional. Output test results in junit format to junitFile. If a directory is specified the output is to junit.xml. --show |- --suites - Flag. Optional. List all test suites. -1 | --suite |- --one testSuite - String. Optional. Add one test suite to run.
  • --tag tagName - String. Optional. Include tests (only) tagged with this name. --list-tags | --tags |- --show-tags - Flag. Optional. Of the matched tests, display the tags that they have, if any. Unique list.
  • --skip-tag tagName - String. Optional. Skip tests tagged with this name.
  • testFunctionPattern ... - String. Optional. Test function (or substring of function name) to run.

Debugging settings

Append to the value of BUILD_DEBUG (a comma-delimited (,) list) and add these tokens to enable debugging:

  • test-dump-environment - When set tests will dump the environment at the end.

Return codes

  • 0 - Success
  • 1 - Environment error
  • 2 - Argument error

Environment

Requires

testSuiteOrdering

undocumented

Usage

testSuiteOrdering [ --cache cacheDirectory ] finderFile

No documentation for testSuiteOrdering.

Location: bin/build/tools/test.sh

Arguments

  • --cache cacheDirectory - Directory. Optional. Cache directory to use for ordering work.
  • finderFile - File. Required. File to reorder.

Writes to standard output

Reordered file.

Return codes

  • 0 - Success
  • 1 - Environment error
  • 2 - Argument error

testSuiteFunctionTested

When environment variable TEST_TRACK_ASSERTIONS is true – testSuite and assertion

Usage

testSuiteFunctionTested [ --help ] [ --help ] [ --verbose ] [ functionName ... ]

When environment variable TEST_TRACK_ASSERTIONS is true – testSuite and assertion functions track which functions take a function value (for example, assertExitCode) and track functions which are run, and stores them in the testing cache directory which accumulate after each test run unless the cache is cleared.

Location: bin/build/tools/test.sh

Arguments

  • --help - Flag. Optional. Display this help.
  • --help - Flag. Optional. Display this help.
  • --verbose - Flag. Optional. Show list of true results when all arguments pass.
  • functionName ... - String. Function to look up to see if it has been tested. One or more.

Return codes

  • 0 - All functions were tested by the test suite at least once.
  • 1 - At least one function was not tested by the test suite at least once.

Environment

assertStatistics

Output assertion counts

Usage

assertStatistics [ --reset ] [ --total ] [ --help ]

Output the total number of assertion failures and assertion successes, separated by a space and terminated with a newline

Location: bin/build/tools/test.sh

Arguments

  • --reset - Flag. Optional. Reset statistics to zero.
  • --total - Flag. Optional. Just output the total.
  • --help - Flag. Optional. Display this help.

Writes to standard output

UnsignedInteger. 2 lines.

Examples

read -r failures successes < <(assertStatistics) || return $?

Return codes

  • 0 - Success
  • 1 - Environment error
  • 2 - Argument error

Code quality tools

bashFindUncaughtAssertions

Search bash files for assertions which do not terminate a

Usage

bashFindUncaughtAssertions [ --help ] [ --handler handler ] [ --exclude path ] [ --exec binary ] [ directory ] [ --list ]

Search bash files for assertions which do not terminate a function and are likely an error

Location: bin/build/tools/lint.sh

Arguments

  • --help - Flag. Optional. Display this help.
  • --handler handler - Function. Optional. Use this error handler instead of the default error handler.
  • --exclude path - String. Optional. Exclude paths which contain this string
  • --exec binary - Executable. Optional. For each failed file run this command.
  • directory - Directory. Optional. Where to search for files to check.
  • --list - Flag. Optional. List files which fail. (Default is simply to exit silently.)

Return codes

  • 0 - Success
  • 1 - Environment error
  • 2 - Argument error

evalCheck

Check files to ensure evals in code have been checked

Usage

evalCheck

Check files to ensure evals in code have been checked

Location: bin/build/tools/security.sh

Arguments

  • none

Return codes

  • 0 - Success
  • 1 - Environment error
  • 2 - Argument error

🛠️ Tools · ⬅ Top