Reports based on test steps

[ErikB]

Are there plans to include the option to look at the Zephyr for JIRA gadgets based on test steps, as opposed to just tests?

We'd very much like to see a burndown chart that included all test steps in a cycle, for example.

[Daniel · Accepted solution]

Hi Erik,

I have added this and your description to our feature request list. Thank you for your detail.

Regards,

Daniel

[ErikB]

Thank you, Daniel.

[Manny P]

Hi Eric,

Is there any update on this request. Like all the other users below , we work on a complex environment with each scenario having more then 400 steps spread across multiple teams. If we can track the execution status at step level we know which team currently owns the execution and report in terms in all over step progress in a day. As a 400 steps scenario may take several days to complete , not having the capability to step level reporting , will show the same status ( In progress) for all these days. 

[Nilgun Oneren]

Hi Daniel

Has this feature been created? We would like to report on our test steps as well? 

Thanks

[Teressa Reed]

Same situation here.  We want to report metrics at the test step level.  Will this be new feature or is there a workaround using ad-ons?

[swathi]

Hi,

Are there any updates on this ticket?

[Chris Miller]

Hi Erik,

Ok, I understand your reasoning. But I am curious yet on what you do with testcase results.

For example, if you have 15 testcases in a cycle and 200 steps, what does your chart show? Are there only 200 step values in the chart and no testcase results in the chart? Or do you also want the testcase results and someone have each step result included in a family of a single testcase's results?

Note, I think no matter which way we are going here, this is a feature request. But understanding your end goal can allow us to accuractly consider the feature request.

Kind Regards,

Chris

[ErikB]

Thanks, Chris- I really appreciate the attention to detail. I suppose the absolute best case scenario would be to be view both - a chart that shows the 200 steps or a chart that shows the 15 tests, or possibly even a chart that shows a combination view (tests and test steps completed vs tests and test steps remaining).

The main goal, using your example, is to be able to take into account situations when you have 15 test issues, 10 of which have 5 test steps each (50 of 200) and 5 of which have 30 steps each (150 of 200). So when you look at test burndown with the current gadget, if the 10 tests with 5 test steps each are complete, you'd think you were 66.66% done with testing. But that is misleading. If you base it on test steps, then you are 25% done with testing - a very significant difference.

The reality so far in our usage of Zephyr for JIRA is that the number of test steps per test case varies quite a bit, so only looking at stats based on testcase completion doesn't tell the whole story.

Hope this makes sense, and again thanks for being so responsive and eager to improve the product for the users!

Erik

[ErikB]

What do you think, Chris?

[Troy Wang]

It's been several years. Has this problem not been solved yet?

Zephyr can't calculate and report conveniently at the test steps level ?? So disappointing! I want to give Zephyr a negative score. 

[Radoslaw Cichocki {Deviniti}]

Hi Troy,

We hear you here at Deviniti. We've had numerous requests to add this to our tool too.

In case you are open to considering alternatives, we have just released TestFLO 8.6.0 - Test Management for Jira. We have a Test Plan Execution report where you can now choose if it's precision should be based on test case statuses, or step statuses:

Here's the documentation regarding this report:
https://deviniti.com/support/addon/server/testflo-86/latest/test-plan-execution-report/

Just be aware TestFLO is only available for Jira Server and Data Center.

Regards,
Radek Cichocki
Product Manager @ Deviniti

[Jun Li]

@Radoslaw Cichocki {Deviniti} 

it is great that testFlo handles step report. is there a way to migrate existing zephyr test cases, cycles to testFlo?

[Radoslaw Cichocki {Deviniti}]

Hi @jun

Just two words of introduction. In TestFLO we have two concepts:

• Test Case Templates [TCTs] (non-executable test specifications)
• Test Cases [TCs] (executable copies created from Test Case Templates)

You can create TCs from TCTs, so TCs are executable copies of TCTs. TCs are groupped in Test Plans (technically TCs are sub-tasks of Test Plans).

At this moment, you can import Zephyr's test cases as our test case templates using our CSV importer. You would need to export your test cases from Zephyr, and trtansform the resulting file into the format acceptable by our CSV TCT importer.

If importing just the test specifications is enough for you, most likely you will be able to do it on your own (or we can help).

Regarding Zephyr's test executions, you will need our help if you need to migrate these too. For some customers, migrating just the test specifications, without all the execution history is sufficient.

We are considering creating a dedicated importer from Zephyr to automate the whole process, including test executions.

I would like to encourage you to:

1. Consider if importing just the test specifications is enough for you - you can have both tools for a moment, so you would start all your new test efforts in TestFLO while finishing up the old efforts in Zephyr. If so, you may be able to comlpete the process on your own (CSV TCT importer).
2. Try out TestFLO as a trial version and see how it compares
3. Contact our support team with any requests for help or with any questions you may have, they're a bunch of energetic people who love helping people like you. You can also ask them for a product demo, so they'll walk you through.

Regards,
Radek Cichocki

[Radoslaw Cichocki {Deviniti}]

Hello everybody,

If you'd like to consider an alternative, then TestFLO - Test Management for Jira, has a progress bar that shows the test case progress based on the status of individual steps.

You can see it on the Test Plan for all Test Cases within it:

You can also see it on the test case itself.

Regards,
Radek Cichocki

Disclaimer: I am the product manager of TestFLO.

[Vandana Pillai]

Hi all,

I am also looking to report based on test steps, I think it is a necessary and common sensical requirement , reporting at a scenario level alone would not capture the effort the testers have to put in for testing. It will be hard to substantiate that not a lot of testing was done, as the scenario movement would be minimal, but a lot of testing would have been done considering each scenario has many test steps. Is there an update to this ticket?

[Jennifer Taylor]

We are having the same issue.  Would be nice if there was a solution other than "here's a totally different product we'd like to sell you on"

[sebastien legere]

Any updates on this ticket please?

[sebastien legere]

For those of you interested, TestFlo do provide some sort of information about test steps (number of steps, progression based on test steps in Test Plan). That's something we will investigate in our team.

[Radoslaw Cichocki {Deviniti}]

Hi @sebastien legere ,

I'm TestFLO's product manager, and thank you for bringing this up.
Yes, we do have steps-based information on test cases, you can read more about it here (link).

If you'd like to play around with this, here's our live demo environment.
Once you click that link and go there, you can open any test case, e.g. OSA-1465, and find out how it behaves.

Regards,
Radek Cichocki

[Radoslaw Cichocki {Deviniti}]

Hi @sebastien legere ,

Just so you know, yesterday we have released TestFLO 8.6.0 allowing our Test Plan Execution report to based on test case steps. The reporting precision is now as accurate as it can get.

(Documentation link)

Regards,
Radek Cichocki

[Neeti Grover]

Any update on this item? I'm looking for a way to get a report at a test step level in Zephyr.

[arihant jain]

Any update on this item. This is very important to integrate Zephyr with CI/CD and to create a test execution burn down chart basis on test step execution. Please share the update on this as soon as possible. 

Or Can anyone suggest any alternatives to this?

[Andrea Benavides Vargas]

Same situation here... I'm looking for a way to get a report at a test step level. 

Any idea how to get this info?

[giovanni quilotte]

any updates from Jira on this?

[Priyanka Rangi]

Hello,

Any progress on this request please?

I have similar needs. For example, we are doing end to end testing involving three check points to verify data flow in different systems. So, if my test case contain three steps for these three check points, it is important for us know any issues at a particular step level.

This report will help us see overall picture at different check points.

Regards,

Priyanka.

[Jay Paquette]

Hey Folks,

Thanks for all the input – is there any update or possible workaround for this…”Reporting by Test Steps”?

My request/requirement is similar if not the exactly same, we need the overall flexibility to report by Test Cycle the execution status by either “Test” (which we have currently) or “Test Steps” which doesn’t appear to be possible.

Any thoughts/suggestions would be appreciated.

Thanks,

Jay

[Paulina Wegrzyn]

Hi,

Do you have any updates regarding this case? It would be great to have burndown/report with test steps information.

Br,

Paulina

[David Nash (inactive)]

I would also like to see burndown/report information using test steps (as well as test cases) for the same reason as above - a test case with 300 steps is counted in exactly the same way as a test case with 2 steps in a burndown chart.  Using steps would reflect velocity more accuratly.

[Chris Miller]

Hi Eric,

How would that chart look? Can you explain it so we could understand your use case and desired result?

Kind Regards,
Chris

[ErikB]

Hi Chris-

Thanks for the quick response! Essentially we just want a look at progress in terms of total test steps in a cycle instead of tests in a cycle. It would be great to be able to view the burndown chart based on test steps or an execution chart based on test steps. Basically just the option to be able to look at the entire testing effort in terms of all test steps in all tests in a cycle, instead of just all tests in a cycle.

The reason we want this is because looking at only tests can be a bit misleading as tests can vary widely in the number of test steps each one contains. So the ability to look at things at a test step level would give a better idea of how things are progressing.

Thanks,

Erik