Bitbucket Pipelines marked as success even if one test failed

[Andreas]

I have a test project that runs tSQLt tests on a separate service running the MSSQL 2017 server image.

The tests run correctly, and the output is in a correct XML format.

But even though one test failed:

The pipeline is still marked successful:

What am I missing here?

Thanks in advance for the help!

[Philip Hodder · Accepted solution]

Hi,

A Pipeline is marked as success or failed based on the exit code of all the commands set. So if your command has the exit code 0, we will still mark that as successful. Most tools will return a non-zero exit code when a test fails. Perhaps you can configure your tool to have a different exit code.

The test output shown is read straight from the xUnit XML output. It doesn't influence the final result of the pipeline.

Thanks,

Phil

[Andreas]

Hi Phil,

Thanks for the answer! I am now able to have the Pipeline fail when a test fail. And if you do not mind, I have a follow-up question.

Currently my (simplified for the example) steps are:

# Run all tests
sqlcmd -b -Q "EXEC tSQLt.RunAll"
# Generate XML reports
sqlcmd -Q "EXEC tSQLt.XmlResultFormatter" -o test-reports/tsqlt.xml

Because I use the sqlcmd utility with the "-b" option, the first line returns an exit code different than 0.

But then I cannot execute the next steps (generate the reports). Do you have any idea how to solve this?

Regards

Andreas

[Andreas]

As a workaround, I am using the following steps:

# Run all tests without the "_b" option, to not stop on errors
sqlcmd -Q "EXEC tSQLt.RunAll"

# Generate XML reports
sqlcmd -Q "EXEC tSQLt.XmlResultFormatter" -o test-reports/tsqlt.xml

# And finally, run all tests again, this time with the "-b" option, to make Pipelines fail if there is any error
sqlcmd -b -Q "EXEC tSQLt.RunAll