How to run an Xray test?

[Chris Kemp]

As you can tell by the question, I am 'very' new here.

I am not asking this question without actually attempting/reading articles.  My confusion comes directly from docs.getxray, specifically  https://docs.getxray.app/display/XRAY/Test+Runs

The document categorically states "To execute a Test from a Test issue", and provides steps and screenshots.

Step 1: Open the Test you wish to run.

From a Test Issue (being the test I want to run) there is no Play button as in the screen shot.  My Test Issue looks nothing like the screenshot.  The screenshot clearly shows three Test Runs; CALC-8, CALC-10, CALC-9.  Surely this means that the test has already been run, and is therefore not an instruction how to run a test?

Execute Test Inline states that the option must be enabled in the XRay administration page.  Isn't that what Execute Test is attempting to illustrate prior to this, again, "To execute a Test from a Test issue"??

NOTE: Xray Settings > Miscellaneous > Inline execute tests IS selected within my Team Managed project.

>>> At this stage, I have played with the Test Execution Issue (which is mapped to Test Execution within xray settings).  Having selected an Ad-hoc run and a Test Set, I believe a Test Execution issue is created from which I run the tests.

When I view Test Runs within a test, I believe I am presented with the instances that have been executed from Test Execution.

Although the concept of creating a Test Set AND a Test Execution is alien to me (isn't a Test Execution effectively a Test Set?), I believe this is what I want ...I would never want to run a test from 'itself', as this implies it would record a result against its own instance, which would be applied to any new Test Set/Execution to which the test is added.

I believe Test Instances are only an Xray feature, and nothing to do with native Jira Issues.  If this is correct, it makes sense to me that the instances are separate entities, which I believe are associated with a Test Execution.

Perhaps it is just the way my organisation has chosen to set things up, but If my attained knowledge so far is correct, it appears that the xray doc is very badly written, or at least, badly formatted, resulting in total confusion.

In summary, isn't it the case that a Test (xray) can only be run from a Test Execution (issue, as mapped in settings)?

NB Accepting that a prior Test Run/Execution can be identified (and re-run), within a Test issue.

>>> I suspect my observations are correct, as the document actually starts with a subtle statement "A Test Run (sometimes simply referred to as a "run") occurs when a Test is scheduled to execute within a Test Execution. Therefore, each time you run a Test, you're, in fact, running a Test Run."

If the answer to my question is a simple 'yes', then the documented Actions, Run a Test should be providing instructions to create a Test Execution (Ad-Hoc, Test Set etc).  Once this is done, then you can use a Test isssue to run a test, if you 'open the test' and select Test Runs.

[Stefan Salzl · Accepted solution]

Hi @Chris Kemp 

It seems a bit complex in the beginning. I´ll try to give a short overview as far as possible:

first of all: all xray specific issues are native jira issues and can therefore undergo eg. a workflow and displayed in board

Testcase
as mentioned before it is a native issue therefore (commonly) created in backlog. This is the place where test case design happens, the baseline for a test. This is the collection and description of test case, using test steps.

TestSet
a collection of several test cases. This is just a bundle of already existing tests (eg. RegTest). 

TestExecution
This documents the (scheduled) execution of TestCases for example we get a new build so we create a new TestExecution. Let´s say we have "TestCase01". Now we create a TestExecution out of that TestCase. Within the TestExeuction an instance of the TestCase is created as TestRun --> this is where we go through test steps and validate to pass or fail. 

TestSets can also be the base of TestExecutions. Let´s say we get a new build from development and we want to run RegressionTests. We have a TestSet consisting of 30 TestCases that are relevant for RegTest. We create a TestExecution out of the TestSet for testing the new build. The TestExecution will create TestRuns (instance of TestCase) for each TestCase within the TestSet.

From my point of view those are the most important basic things to start with. I can highly recommend reading through the XRAY documentation from the very beginning, all the test entities and how they go hand in hand. Personally I can say I would have loved to have such a tool and traceability of test management when I was a QA engineer 15 years ago :)

I would also recommend watching this video gives a good introduction on several XRAY topics:
https://www.youtube.com/watch?v=SO2IZefGy7U

Hope my description makes sense for you and could clarify a few things.

Best
Stefan

[Chris Kemp]

Excellent, thank you for confirming my understanding.  Maybe someone from Xray will read this and review their document.

Although I am a little unsure of the usefulness of Test Set, I have come to the conclusion that it 'can' be part of Test Case Development (as opposed to the Test Execution, being a 'project' entity).

I am trying to separate Test and Test Set into one workflow (Design/To-do > Repository/Done), and Test Execution into another (To-do > In Progress > Done), with the intention of having two boards ...annoyingly, I have found that Team Managed only has the ability to show one board(!), and Company Managed is too restrictive to give me the ability to create this structure :(

I have read a suggestion of using a Project (Team managed) to design tests, and then move a Test Execution to the 'actual' project.  Would you mind telling me if this would work, ie the Test Execution containing Test Instances, could be moved without its 'parent test issues'?
(I appreciate that the target project would have to have a Test Execution Issue Type, hopefully specified with the same fields etc)

[Stefan Salzl]

I would not say I completely confirm your understanding.

Although the concept of creating a Test Set AND a Test Execution is alien to me (isn't a Test Execution effectively a Test Set?)

this eg. is a clear no if you read carefully through my post. A test set is a collection of test cases. There is no possibility to to track results (pass/fail) in test set. This functionality is within test execution which generates test runs out of the test cases that will then be started/click play and where results of the runs are documented.

And to answer your internal question:

create Tetst (step-by-step-description of what needs to be done when running the test) 

Now you have a documentation of your test steps. No more, no less. This doesn´t say anything about an execution.

To now execute/run these test steps:
Create TestExecution (several ways, I´ll show just one of them)

• Within your recently created test case change to tab "Test Runs"
• click "Execute in"
• choose "New test execution"

A test execution issue is created. As you can see it contains of a testRun an instance of our recently created test case (see screenshot). This TestRun can now be "run/executed/started with the play button and clicked throug/set to fail/pass"

If you click the play button you will be directed into the TestRun where you can set status of test steps

This is mainly how running tests in XRAY.

Even if I repeat myself: I highly recommend to read through all parts of the xray documentation, the getting started part and the different entities. Everything I know I gained through checking and re-checking things there and also some youtube videos. In my opinion it really explains and documents functionality of XRAY pretty good.

Best
Stefan

[Chris Kemp]

Thank you for illustrating your answer, I expect others will find it informative if they have searched for this subject.

(I would have marked your previous answer as answered, but I didn't know if this forum blocks further submission once an answer has been accepted)

NB my point about a Test Set vs Execution was not in regard to Jira, but the concept of adding individual tests to an 'entity'.  Sorry for the confusion.

I will try and experiment with creating a Text Execution in one project and moving it to another, when I get access to a second project.

I now have a new problem, as it appears xray creates yet another entity called Test Coverage, as opposed to Test, when coverage is configured ... I will look into this, which I suspect will result in more questions :)