Zephyr for Jira: Test execution very tough

[Deepak Bansal]

New columns interface with v2.2 looks better but does not help searching test cases. 'Assignee' and 'Label' would be very handy here.

We have 300+ test cases in a cycle to be executed by multiple testers.

1. Cycles interface is very bad as tester has to search his test cases from all 300 cases by pressing 'Next' several times.

2. Search executions interface is also not helpful as it does not allow query on 'label' or 'assignee'. This is simple requirement and will ease out execution. Tester should be able to see only his test cases filtered by lable or assignee.

[Deepali Patwadkar]

Thanks G3, for the various options.

But these workarounds dont work for me.

a. We use Jira for agile, defects and task tracking etc. The component names are standard and related to the product. Cannot have testers names in there!

b. To have the name of the tester in the "Executed By" field, in the first place the tester needs to find the tests that are assigned to him / her in the execution. Its a cyclic problem.

Need a real solution, this workaround / creating test execution cycles per tester is not a sustainable solution.

[Dennis Ferguson]

It is 2017 and I still do not find a reliable way to search Zephyr for Jira test excutions. I have seen several suggestions to add "ExecutionStatus=UNEXECUTED" or "issue in testsWithoutExecution()" to search query which do not seem to work for us. I am getting

"Field 'ExecutionStatus' does not exist or you do not have permission to view it."

when I use "ExecutionStatus=UNEXECUTED"  or 

"Unable to find JQL function 'testsWithoutExecution()'." when I try to use "issue in testsWithoutExecution()".

When will this be fixed?

[Katja Maliangkay]

I see the same problem. We are trying the Zephyr plugin for JIRA, 15 QA engineers spread over two teams, thousands of test cases. I have to decide if I will migrate all these test cases to the Zephyr plugin and end up not being able to assign a test cycle to an assignee. Adding an assignee field to the test cycle view would be the silver bullet with which Zephyr would be able to kill all other businesses.

Strange days: We will probably abuse the Build field as the assignee per test cycle. (similar work around as stated above where abuse of the component field is suggested. We already use the component field)

Mystery of this century: why don't they add it?

[Deepak V]

All I see above are workarounds.
The issue would be solved only if we had a "assignee" field access through ZQL. 
(which is obviously not present as per, https://zephyrdocs.atlassian.net/wiki/display/ZTD/ZQL+Reference) 

[Deepali Patwadkar]

EUREKA!!!

Here is how I solved this problem ...

Say you have 300 tests to be executed.

Steps:

1. Create Filter_A to filter out 50 tests (not all 300) that are to be assigned to Tester_A.
2. Add these 50 tests "via Filter_A" to the Test_Execution_Cycle
3. Go to Tester_A desk.
4. Search the Test_Execution_Cycle for all un-executed tests.
5. Bulk Change the status of all 50 tests to "WIP".
6. Now the tests have "ExecutedBy = Tester_A"

Repeat above steps for every tester. Ensuring that the 300 tests are covered in various filters.

At the end of above exercise, you have a test execution cycle, with all tests marked "WIP". Plus every test has the name of the assigned tester in "ExecutedBy"

Now testers just filter out their tests with:

cycleName = Test_Execution_Cycle AND executedBy = <Tester_A>

Tester's issue of seeing assigned tests is solved. And all under ONE execution cycle.

* This works if you have a small number of testers.

[G3]

Hello Deepali,

It would definitely be helpful if both ZQL and JQL could be used to search test executions and/or search issues. But, I understand there are some challenges to creating that capability. Anyway...

1. You probably already know this, but here's the ZQL search criteria and information reference link:
• https://zephyrdocs.atlassian.net/wiki/display/ZTD/ZQL+Reference
2. When you say a test execution is 'assigned' to someone, is their name in the 'Executed By' column of the test execution?
• If so, try searching the test executions using this criteria (based on your screenshot):
• cycleName = "EoR ECCN-2014-RC1" AND executedBy = <name of tester>
• That should display all test executions that have an overall 'Execution Status' (not a test step status)that has been set by a specific tester (thus setting the 'Executed By' with their name). Even if they set the overall 'Execution Status' to "WIP" that would at least capture the tester's specific name in the 'Executed By' column, and therefore would be searchable by both the tester and cycle names.

Finally... here's a way you could absolutely nail the execution search per tester.

• Add each tester's name as a 'Component' in each project (or to each project containing test cases)
• 'Assign' an execution by adding that specific tester's name to the 'Component' field of the test case you would like for them to execute
• Then use this 'test execution' (ZQL) search criteria (based on your screenshot):
  • cycleName = "EoR ECCN-2014-RC1" AND component = <name of tester>

This would only display test executions for a specific tester within a specific cycle.

Give it a try... see if that works for you!

[Deepali Patwadkar]

Thanks Emily! Here is the screen shot.

[Emily Schneider]

Hi Deepali,

Is it possible you can attach some screen shots? We want to see how your ZQL is set up and try the same as you.

Kind Regards,

Emily

[Deepali Patwadkar]

Searching test executions is not working: when I search for "Summary ~ xyz_string" it brings up only few test cases. Not all tests that contain xyz_string.

Adding this to the fact that fields like: Labels, Assignee and custom fields are not available for searching it makes testers life very difficult.

[Jia Gui Fen]

We also have the same requirement.

[Emily Schneider]

Hi Deepak,

Instead of using the test cycles have you tried using the Search Test executions navigator? Here you can do searches of your executions using a ZQL. You can narrow down by searching based on what cycles, versions, compoents and more that the executions belong too. For more information on this check out how to use the search test executions navigator as well as a full ZQL reference page to assist you in crafting your ZQL's:

https://zephyrdocs.atlassian.net/wiki/display/ZTD/Search+Test+Executions

https://zephyrdocs.atlassian.net/wiki/display/ZTD/ZQL+Reference

Additionally, you are saying you want to search based on label. It is important to note that you can not assign executions and you can not place a JIRA label on executions specifically. So this is why you can not search for executions based on labeles or assignee. Becuase you can not link executions to assignee or labeles. You can however assign labels and asignee to the test case itself. In order to do a search you will want to do a JQL in the JIRA issue navigator. Not a ZQL in the search test executions navigator.

By doing a JQL in the JIRA issue navigator you can pull up all the tests cases based on a certain label or assignee.

Kind Regards,

Emily

[Deepak Bansal]

Search execution was well tried and not useful in this case. Looking at your comments, I am sure, problem is still not understood.

It is very difficult for tester to search for test cases assigned to him for execution using any interface. You have 2 interfaces for execution as I mentioned above and both does not solve the purpose.

If I have 5 testers in my team, 300 test cases in a cycle, same component. How my tester1 can see only 30-40 cases he is supposed to execute? I know he can see his test cases with Jira filter interface but he will not be able to execute from there. So we want this on execution interface where he can just see his cases and execute them quickly.

[Jessica Sullivan]

Maybe organizing it this way would work? - Break the 300 test cases up into 5 test cycles. Tell each tester which test cycle they should execute.

This would do the following:

1) Make the test cycle much easier to navigate.

2) Remove the need to assign a test case (we never assign test cases so that each case can be used at the same time on mulitple efforts at once. If you assign a test case how do you reuse it for regression testing on the next effort?)

[Deepak Bansal]

300 was just an example. Multiple cycles can be created per user but that would not help in tracking execution. More cycles more difficult to consolidate status of testing.

[Jia Gui Fen]

We also have the same requirement.

[Jia Gui Fen]

I think Label is the same as Component.

Please just add Label column and search condition like Component in search test executions navigator.

It will be very usefull to filter test executions in test cycle.