Acceptance Test Driven Development with Jira

[Maxime Lemanissier]

Hi there,

We would like to do Acceptance Test Driven Development (ATDD), that is to say, write user stories in a precise format (given / when / then, or equivalent), parseable by a tool to run automated tests (against source code through "fixtures", or against GUI through GUI driver tools like Selenium).

We already tried Fitnesse, Cucumber, etc. but as we're using Jira, we would prefer to have a unique repository / tool for managing our stories.

Is there any Jira plugin that allow to do ATDD, or to integrate with third party ATDD tool?

Thanks,

Maxime

[Bruno Conde]

Hi Maxime,

I suggest Xray for JIRA: https://marketplace.atlassian.com/plugins/com.xpandit.plugins.xray

It's a test management tool supporting both manual and automated tests (using Cucumber) that you can link to requirements (in this case user Stories). You can create automated cucumber tests and generate .feature files to execute externally to JIRA. We are working on a REST API for exporting Cucumber .feature files and importing Test Execution results into JIRA. This will allow you to use CI platforms like Jenkins to execute your tests and have the report and requirement coverage in JIRA.

Best Regards,

Bruno Conde.

[edubme]

@Bruno Conde did your team figure out an API to import test execution results into JIRA?

[kamalpreet kaur]

@Bruno conde Hi Bruno, We already have tests BDD test running in Cucumber framework, any thing on the API for importing test results back in JIRA? Went through a lot of documentation online but no luck 

[Nick Iles _AssertThat_]

Hi, 

We've built a couple of frameworks over the last few years and found that all the solutions had a few gaps in functionality so we decided to build our own.  We released it a couple of weeks ago and are building up a good client base.   The plugin includes a cucumber.json import so your test results are automatically imported back into Jira.  This allows you to see your test results alongside your user storied.

We also built in some functionality to analyse your test results so you can assign defects etc all in Jira.

Please take a look and we'd really appreciate your feedback.

https://marketplace.atlassian.com/apps/1219033/assertthat-bdd-test-management-in-jira?hosting=cloud&tab=overview

Thanks

Nick

[Nick]

Maxime - We're going to try this as well and we've looked at Behave but aren't sure we like it. The biggest limitation was being able to group and categorise stories/acceptance critiera.

Instead we're going to try using Bob Swifts CLI tool to export BDD/ATDD scripts from the description fields of JIRA User Stories to text files. We will then try and pump them through Cucumber bolted onto a Selenium driver and into our application.

Wish us luck!

[Emidio Stani]

@Nick

how did it go with the bob swift cli tool ? I might think to implement the same thing, for example launching jenkins, which retrieves the user stories from jira and then execute cucumber to launch selenium driver.

[Nikhil]

Hi @Maxime Lemaniss1@Bruno Conde @kamalpreet kaur@edubme@Emidio Stani

Actually I am totally new for XRAY- Cucumber integration. Basically I have one cucumber bdd .feature file written in Intellij IDE. At the same time I have corresponding step definition too which is working fine locally. Now as per my requirement, I need to trigger this feature from Xray-Jira but don't know how to do that, I mean which api to call, how to call that api, where to call that api etc etc.

I tried to find some good document on google but could not find anything fruitful which can help me step by step setup.

Can anyone please help me in this ?

Thanks.

Regards,

Nikhil Agrawal

[Bruno Conde]

Hi Nikhil,

You can define cucumber tests directly in Xray.  For this you can just create a new Test issue and choose the Cucumber type. 

A cucumber test in xray only contains the scenario definition. You should create a Test issue per scenario.

The implementation part, or the code that will parse the scenario and execute it's steps is not meant to be stored in Jira. You can use your favorite code repository. 

Xray also provides an API that allow you to import your cucumber feature files. This will automatically create Tests, and other entities in Jira:

https://confluence.xpand-addons.com/display/XRAY/Importing+Cucumber+Tests+-+REST

This is just for defining/importing ccuumber tests in Xray. Regarding the import of execution results Xray also provides a REST API that allows you to simply take your cucumber results json file and import it directly to Jira. This will update/create new Test Executions in Xray with the results. Remember that it is not the purpose of Xray/Jira to execute cucumber tests. Xray just stores the execution results and outputs nice reports.

It is possible to configure builds in a CI tool that will:

1. Export the cucumber definitions from Xray issues:

2. Checkout your implementation from the repository and merge it with the feature files exported in step 1.

3. Execute the tests with cucumber

4. Import the execution results into Xray.

Xray provides plugins for Jenkins and Bamboo to easy the configuration of these tasks.

It is also possible to import the cucumber execution results right from the UI (from a Test Execution or a Test Run).

Please check the following links for importing execution results into Xray:

https://confluence.xpand-addons.com/display/XRAY/Import+Execution+Results+-+REST

 https://confluence.xpand-addons.com/display/XRAY/Continuous+Integration+Tools

https://confluence.xpand-addons.com/display/XRAY/Integration+with+Bamboo

https://confluence.xpand-addons.com/display/XRAY/Integration+with+Jenkins

https://confluence.xpand-addons.com/display/XRAY/Exporting+Cucumber+Tests+-+REST

Hope this helps,

Regards,

Bruno Conde

[Nikhil]

Thanks a lot @Bruno Conde. This information really helped me a lot. Just want to confirm one thing, do we have any plugin of Jira-XRAY with Bitbucket code repository or it would be handled other way round, I mean plugins needs to be installed in Jenkins (CI tool).

What I understood is, we do not have any bitbucket or jenkins plugins that can be configured in XRay rather we can install these XRay and Bitbucket plugins in Jenkins. Correct me if I am wrong ?

[Bruno Conde]

Hi @Nikhil,

Sorry for the late response but somehow I have missed the notification for this answer.

The import execution results process is driven by the CI platform. Xray provides plugins for the most common CI platforms like Jenkins, Bamboo, TeamCity.

These plugins will just help you call the Xray rest API to import execution results.

Hence, you configure the integration between Xray and CI on the CI side.

Regards,

Bruno Conde

[ILYUNG]

I have tried "Behave for JIRA" but there is some limitation in the features. For example, users can not search the created features. And it seems to be hard to organize the created features as well. And what are the 'tags' for?

@Alan Parkinson, do you have some guide or solution for those?

[Jeremy Largman]

Our documentation is a bit outdated, but there's Using JIRA for Test Case Management. Check that out and its sub-pages. In general, search Google (and our resources, like confluence.atlassian.com) for 'test case management'. There's also a good thread over on Stack Overflow.

[Maxime Lemanissier]

ATDD is about test automation. You define user stories as test scenario, and you make them runnable on your system. It's like unit tests understandable and writable by business analysts.

Greenpepper, which is based on Confluence is a tool of that kind. As confluence and jira are integrated, this could be a solution, but a complicated and expensive one (2 new tools and licenses), considering a jira plugin could do the work.

[Alan Parkinson [Hindsight Software]]

We have built such a JIRA Plugin (Behave for JIRA) and its now available in the marketplace https://marketplace.atlassian.com/plugins/com.hindsighttesting.behave.jira

It allows "Scenarios" to be added to user stories or any JIRA issue as Acceptance Tests. The scenarios are written in Cucumber's Gherkin format and a maven plugin transforms the relevant scenarios into features files during the build. Cucumber can then run the generate feature files to execute the tests.

[Renjith Pillai]

I am not sure whether you need a plugin for this as this is more of a practice that needs to be followed in the project. This is what we used to follow. We used to have an issuetype called testcase and with some custom fields like 'pre-condition', 'post-condition' to fully describe the testcase. So for every userstory there will be a set of testcases which are linked to the userstory using the Jira linking function. The link that was used was 'is tested by' even though the type of link has no relevance apart from correcting displaying the relation while viewing the story.

The testcase issue type also was associated with a workflow which moves from states New => Under Testing => Success/Failure

Additionally, our CI system automatically transistions the testcase to success when the testcase passes in the CI and this becomes the closure DONE criteria for the userstory.

Hope this helps.

[Maxime Lemanissier]

how do you link the testcase in jira and the testcase run in ci server ? using jira api?

[Renjith Pillai]

Yes, access jira using SOAP interface.