Is there a way compare two Test Executions (xray)?

[Walter White]

Hello,

we would like to compare different xray Test Executions e.g. the current one for a dev branch and one for some PR to see how that PR affects the results.

Of course, it would be possible to get this result using the API and do some magic.

But is there a better way to do this, e.g. a plugin or some Jira functionality?

[Walter White]

I also asked the xray support and there was not suitable answer.

Our developers must be able to see the impact of each PR on a defined test plan. The baseline is the result of the test plan for e.g. the current development branch.

It's a rather simple thing and I put it together myself using the xray API in a few hours.

The following is now added as a PR comment (with some markdown) once the webhook-triggered tests are done.

[pat.mayer]

Hi can you explain how you got it done?

[Walter White]

There is a Python script bound to a commit or push hook. I am reading the test results using the Xray REST API for the associated test run for this PR and for the main branch. Then I compare the existence and the results for the tests. Afterwards I put together a Markdown table with the results and add that to the PR as a comment using the Bitbucket API.

[Rogério Paiva - Xray Xporter]

Hi @Walter White ,

Yes, you may use the Document Generator as @Damian Wodzinski suggested.
Or you may use the Test Execution report, please check its documentation page here: https://docs.getxray.app/display/XRAY/Test+Executions+Report

Thank you.

Kind regards,
Rogerio Paiva [Xray Support Team]

[Damian Wodzinski]

How to compare them? Which data would be compared?

For simple puproses, you can use xray Document generator to create a template with necessary fields. Then just filter the runs, and generate a document. The data you need can be exported to excel this way, and  there you can compare it.