Prevent Bamboo from failing job when failing tests are moved to quarantine

[pbaranski]

On Bamboo I have MSBuild job for building and running tests.

It works fine - but I want to use test quarantining option so when I have some test moved to quarantine (and this moved tests are only what fail) job will be marked as successful

I moved one failing test to quarantine but job is marked as failed

Configuration:

Job in Tasks I have Command named Test runner:

Executable = VSTest.Console

Argument = "WebServiceTestClient.dll" /Logger:trx

And after running job with failing test quarantined I get log

01-Apr-201410:59:44Total tests:62.Passed:61.Failed:1.Skipped:0.
01-Apr-201410:59:44TestRunFailed.01-Apr-201410:59:44Test execution time:10.1296Minutes
01-Apr-201410:59:44Failing task since return code of [C:\Program Files(x86)\Microsoft VisualStudio11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\VSTest.Console.exe "WebServiceTestClient.dll"/Logger:trx] was 1while expected 0
01-Apr-201410:59:44Finished task 'Test runner'
01-Apr-201410:59:44Starting task 'Parse test logs' of type 'com.atlassian.bamboo.plugin.dotnet:mstest'
01-Apr-201410:59:44Parsing test results...
01-Apr-201410:59:45Failing task since 1 failing test cases were found

Job is marked failed but then:

01-Apr-201410:59:45ChangingTaskResult to SUCCESS as all failed tests were quarantined.01-Apr-201410:59:45Finished task 'Parse test logs'01-Apr-201410:59:45Running post build plugin 'NCover Results Collector'01-Apr-201410:59:45Running post build plugin 'Clover Results Collector'01-Apr-201410:59:45Running post build plugin 'Artifact Copier'01-Apr-201410:59:45Finalising the build...

Unfortunately the output is:

This question was asked here http://stackoverflow.com/questions/22803557/prevent-bamboo-from-failing-job-when-failing-tests-are-moved-to-quarantinebut hopefully somebody here give me right answer.

[Przemek Bruski]

A fixed plugin is available here:

https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.2/atlassian-bamboo-plugin-dotnet-5.5.2.jar

You can install it on older Bamboo versions, but first, remove the old plugin from bundled-plugins.zip

[Pawel Skierczynski]

There is newer version fixing additional problem when validadtion error occur. Not related to this but you may want to install the newest one.

https://maven.atlassian.com/public/com/atlassian/bamboo/plugins/dotnet/atlassian-bamboo-plugin-dotnet/5.5.3/atlassian-bamboo-plugin-dotnet-5.5.3.jar

[Jason Monsorno]

We had this problem 2 weeks ago, I just had it run through a script then used the MSTest Parser instead. Yes, admittedingly it is a hack for functionality that should be builtin but it's not a show stopper anymore.

[Matt Block]

Would you mind providing more details about this script and configuration? We are admittedly a *nix shop but maintaining a Windows app during a migration process, so we defintely aren't windows scripting experts. How do you force the script to return 0?

[Jason Monsorno]

The script always return 0 by default, you have to do extra work to make it fail a build. It just runs MSTest with testContainer, resultsFile, & category command line option. Then I use the MSTest Parser task to pickup the .trx file I just generated.

[Matthew Block]

I'm going to need a little more detail. I tried doing this and it is definitely still returning an error code and it doesn't even make it to the MSTest Parser task. Here is the inline script I was using...

echo off
echo Running Tests writing results to testresults-%BAMBOO_BUILD_NUMBER%.trx
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /testcontainer:./Source/UnitTest.DAL/bin/Debug/DSC.Dal.UnitTest.dll /resultsfile:testresults-%BAMBOO_BUILD_NUMBER%.trx

[Matt Block]

It has to do with the fact that the MSTest Runner script is still returning non-zero result because there is a failing test. Bamboo sees this non-zero result from the script and assumes there is an issue. I'm surprised this doesn't work since Atlassian provides the MSTest Runner script. In my opinion this is a bug so I'm creating a support request with Atlassian.