The journey to finding a Test Management solution in Jira

As described in my previous blog about Quality Assurance (QA), there are many challenges that need to be addressed when it comes to test management in an Agile world. As emphasized in The State of Testing Report 2019, one of those challenges is choosing the right tool(s) to use for test management.

It is easy to get overwhelmed by all the different vendors and the range of apps, especially with the variety of options available. This makes it hard to compare functionalities and unfortunately, there is no silver bullet for finding the best solution. It depends greatly on the nature of your testing activities and the key features needed, such as:

1. Waterfall processes where comprehensive planning and reporting widgets are needed.
2. Agile processes where simple planning and a mix of manual and automated test execution is needed.
3. DevOps processes where tight integration with CI/CD pipelines is needed (using BDD and living documentation).
   
   

There are all kinds of possible variations, but regardless of what your business requires, a Test Management solution can help to increase either test efficiency, control over testing, or the quality of deliverables. This blog will compare the different approaches to finding a test management solution that fits your situation and needs. 

Customization in JIRA

Originally, JIRA Software was not designed to serve as a tool for test management. Though JIRA has several different issue types by default, there are no specific ones for testing. In addition, JIRA doesn't offer direct integration with automated testing frameworks, but you can customize JIRA in certain ways to use it for (manual) testing.

It's common in Agile projects to create a Scrum software development project and customize the configuration to the specific needs of the team. For example, a very basic configuration is to use the issue type 'Story', enter the acceptance criteria to which the user story should be validated and tested, and create subtasks for all work that has to be done within the sprint, including QA. It's good practice to create some new (sub-task) issue types in order to distinguish the specific type of QA work (for reporting and/or alternative workflows): 'Test Case' and 'Bug Task'. The regular Bug issue type is still being used, but only for bugs not related to a specific user story (e.g.: regression testing or non-functional testing). In those cases, bugs can be linked to the user story by using issue links.

1. An example of customizing JIRA for testing

It's also recommended to create a new screen for the Test Case issue type with some custom fields:

• 'Type' (select list) for distinguishing the test type (e.g. system, acceptance, regression, performance)
• 'Actual result' to describe the outcome of the test case

Depending on your team(s) specific way of working, you can choose to make a custom workflow for the Test Case issue type with different statuses (Passed / Failed). 

It seems pretty easy to customize and set up JIRA for testing. While it could work in theory, it presents several challenges in reality:

• Challenge: It becomes quite complicated to reuse the Test Case for regression testing as the User Story has likely been marked as Done.
• Solution: This problem can be solved by some additional configuration and automation.

• Challenge: It's not easy to get detailed test reporting.
• Solution: This is best tackled by using the issue navigator to filter on issue type Test Case. 
  
  

It’s certainly possible to customize JIRA for Test Management. The downside is that it may require a lot of work. If you don’t have experience as a JIRA administrator, you will need to set aside several hours to configure JIRA to work for test case management.

So what are other options available if you want more control over your QA process? Integrations can help!

Integrating JIRA with Test Management apps

There are a lot of Test Management apps available via the Atlassian Marketplace. These apps can extend the functionality of JIRA and make it better equipped for test case management. But how to choose the right one for your business needs? It all depends on the context and the needs of your organization and team(s), like:

• the type of JIRA implementation (Cloud, Server, Data Center), because there might be functional differences in the app
• the amount and size of the teams
• the test and development process
• the application of manual and/or automated testing
• the need to integrate with (automated) testing frameworks
• the way of handling test management activities (planning, specification, execution, tracking, reporting)
• the possibility of importing and exporting test cases
• the intuitiveness of the interface
• the quality of support by the vendor
• the initial and recurring costs

Based on these considerations (and possibly more), you can start the journey to finding the right solution for your needs. From the perspective of manual testing, many apps provide similar features for designing, planning and executing test cases. The biggest differences are in their user interface, the use of issue types, dashboard gadgets, built-in reports and support for automated tests.

As tool selection (and implementation) are important, we can help you with making the right choice. The following table lists a few popular apps that offer great functionality for basic test management (planning, scheduling, estimating, monitoring, reporting)) and more!

	App	Our favorite features
	TM4J - Test Management for Jira	Many reports, gadgets and Confluence Macros available Integration (ScriptRunner, eazyBI, Jenkins, Bamboo) BDD support
	QMetry Test Management for Jira	Exploratory Testing Support for other frameworks (Cucumber, TestNG, Junit, HP UFT,SpecFlow) Cross-project reporting
	Xray Test Management for Jira	Risk-Based Testing Mobile App Integration (Structure, eazyBI, Jenkins, Bamboo)
	Zephyr for Jira - Test Management	Integration (SoapUI Pro, TestComplete, CrossBrowserTesting, LoadNinja) Usage of Zephyr Query Language for advanced searching Zephyr Blueprints for Confluence available

Using apps can help a lot with getting more control over your QA process. But you’ll still be a bit strapped in terms of what you can and can’t do as the app functionality is limited based on what the (JIRA) infrastructure allows it to do. When it comes to scaling, when teams grow from a large to an enterprise level (which demands customization, automation, and different testing types), you may be better off choosing a dedicated Test Management solution.

Dedicated Test Management Solution

Next to (JIRA) apps, there are several other (dedicated) Test Management solutions available that provide more comprehensive functionality around advanced test case management. For example: 

• metrics & analytics
• test data management
• connecting to multiple JIRA projects
• integration with other (testing) frameworks
• more customization around reporting  

As there are many commercial and a few open-source tools available, we have narrowed down the list to three commercial tools that integrate with Jira and are available on both Cloud and On-premise:

	App	Our favorite features
	Zephyr Standalone	DevOps Dashboard Flexibility in choosing mode of deployment, OS or databases Built-in application walkthrough training and help resources
	QMetry Test Management	Reports specific to audit centric industries (eSignature, audit & compliance reports) Quality Analytics (optional) Integration with QMetry Automation Studio
	qTest	qTest Launch (centrally managing frameworks and test automation tools) qTest Pulse (workflow engine/orchestration of triggers) (optional) Integration with Flood (load testing) and Tosca (automated testing)

Summary

Choosing and implementing the right testing tools is important. The success of the implementation depends on both the selection of a tool to address the required improvement and the tool’s implementation process. Once a (thorough) selection and implementation process have been carried out, an adequate test tool (suite) will support the test process.

Ultimately, which solution (JIRA only, JIRA + an app, or JIRA + an external integration) is right for your team depends a lot on the specific context and needs. We can guide you on this journey so that tooling and processes are complementary. 

Get in touch!