There are hundreds of test automation tools that are being offered in the market today and many new entrants are also coming to market every now and then which makes the tool selection process a bit cumbersome.
Selecting the right automation tool will drive the success of your automation journey as the right tool will bring the desired outcomes like the productivity, cost savings and the speed to the market.
Few of you might have already carried out this exercise in the past as well. In the current times of digital transformation, the factors that drive the tools selection process have changed considerably, because of the changes in software development models e.g., Agile, DevOps, shift-left etc. and also due to the emergence of the latest technologies like AI/ML, RPA and so on.
In this article, we will discuss around how to choose the right automation tool for your organization. When we think of selecting a tool for our organization, it’s obvious that we have to consider other programs or projects outside our team. The projects/products outside our team might be using different technologies, different interfaces, services and so on and it is not an easy task to identify one tool to cover all the applications/products in the organization. So, we have to consider and analyse all the applications/products carefully and group them logically so that we can initiate our search for the perfect tool for each of the groups. This will ensure effective utilization of the tool in our organization for better and quicker returns on the investment,
Before we proceed with initiating the selection process of automation tools, we must finish the detailed analysis of the scope of the automation i.e.
- applications/products to be automated,
- team composition/skillset,
- tools/platforms that are already in place in the eco system,
- company’s IT strategy etc.,
Once the automation scope analysis is completed, in order to kick-off the tool selection process we need to work on identifying the selection criteria for the overall evaluation process.
The automation tool selection process generally follows a 4-step method as below:
Stakeholders, roles and responsibilities
Who should be part of the overall evaluation, review and final selection process? Well, for a structured, unbiased and an overall effective tool selection process, we strongly recommend you involve below members as part of the exercise:
- Test engineers
- Tech. Architect
- Test managers
- Project Managers
- BU Leader/QA Head (recommended for selection process for enterprise-wide rollout)
- A representative from your sourcing team (for a better negotiation/deal with the vendor at the later stage of your selection process)
For a smooth coordination, a Test Lead or a QA Manager should own and champion this task under the guidance of a senior leader like QA Head or so. The above structure can be customized as needed depending on several factors like centralized/decentralized evaluation and so on.
Before this task begins, ensure you have roles and responsibilities identified for each stakeholder in the overall evaluation process. Below is a sample RACI matrix for your reference. You can add/remove other stakeholders to customize it as per your Org structure or for any specific evaluation need.
One more crucial point to be considered during the overall process is professional ethics. Senior leaders have a critical responsibility here to set some ground rules around how do we communicate and partner with the representatives of the vendors partners and guide the team as needed. The vendor partners should be dealt with due professionalism and transparency for the best outcomes. Each and every tool have their own advantages as per their designs. Our goal should be primarily focused on identifying our automation need and ultimately selecting the one that fits to most of our requirements and not finding faults with the individual tools.
Identifying the selection criteria for the right automation tool
This is the critical phase of the tool selection process. In order to make our selection process structured and unbiased, we need to evaluate all the potential tool candidates against the same selection criteria so that we have a common rating method to identify the best tool for our purpose. We will discuss about a template later in the article that captures the common selection criteria and the rating for individual tool candidates to find out the best tool.
Below are several aspects or selection criteria of automation tools that you might consider as part of your overall evaluation process. The list might not be an exhaustive one; you have to add/remove/edit criteria based on your company’s automation needs, IT strategy, etc.
- Test coverage/Versatility: In today’s digital age, modern Apps/Products use complex architecture covering Web, Windows, Linux/Unix, Mobile, Databases, Web Services and so on. You have to ensure the tool you are picking up from the market has the ability to support the required end to end UIs/Layers of your app. We also recommend you to check if the tool can also help in doing other types of testing like Unit, Integration, Code reviews/performance checks, security scans, etc.
- Learning curve, scripting speed, maintenance effort and reusability: This criterion will help you evaluating the ease of adoption of the tool for your project/company. The skillset/competency of the current team, ease of developing/debugging the script and the script maintenance effort etc. are few important factors that you have to check to determine the right tool. Script reusability inside or outside your project is also a key factor. Script reusability will help driving the productivity of the QA team. You should also study the abilities of the tools around development of frameworks like Key-word driven, Data-driven or script-less models etc. that best fit to your need.
- Seamless integration with other tools: As we adopt more and more of processes like Agile, DevOps etc. the software development ecosystem becomes more diversified in terms of other tools (open source/commercial) that are used for different other activities like version control, build, testing, deployment, etc. as part of the overall development process. In this scenario, the test automation tool’s capabilities around talking to other integrations in the setup play a critical role for the overall success of the product release to the market. An ideal tool should be compatible with other tools like Test Management tools like ALM, Defect management tool, continuous integration tools like Jenkins and so on.
- Vendor qualifications: This is also an important aspect of the tool. Maturity of the tool, overall rating of the vendor by its customers, market share/financial stability, quality of support, the overall product roadmap of the tool (futuristic, AI/ML capabilities, Hyper automation/RPA etc.), senior management’s commitment around customer success (investments they make for the tool’s future), online training for new users, global presence, recognitions/certifications in the industry, acceptable SLAs around any technical issues/troubleshooting, etc. are several factors that you can consider under this section of your tool evaluation. If you are conducting POCs (Proof of concept) using more than one tools as part of your evaluation phase, then consider few factors around your experience with those vendors during the overall POC phase.
- Installation (On-prem vs Cloud), Security, License fee/License models etc.: In order to maintain an instance of on-prem installation of the tool, factors like infrastructure (servers for DB, License management etc.), admin and maintenance efforts etc. need substantial investment. Cloud offering of the tool/platform might be an option for you because of the lower cost. But then comes the security e.g., firewall config etc. to allow the tool to access your internal staging/testing environments etc. So, you have to evaluate these criteria thoroughly to understand which model works best for you.
There are so many licensing models are offered by the tool vendors e.g., Floating licenses, Named-user licenses, Subscription based licenses, Enterprise-wide licensing and so on, that need to be critically examined before a decision is made. Expensive tools must ensure highest and quickest return on investment (ROI). ROI should be looked at from both short- and long-term perspective.
Pugh Matrix – Putting things together
In the above section, we discussed about several areas that we can consider as part of the overall selection process. In order to make the overall selection process more effective, we would need a template that can help capturing all the required parameters of the participating tools to enable an apple-to-apple comparison. A Pugh Matrix (extensively used in Six-Sigma processes) is a very useful tool for this purpose. We have developed a sample MS Excel based Pugh Matrix that can be readily downloaded from “Here”. You can customise this spreadsheet by adding more rows and columns etc. to capture additional selection criteria, importance, and corresponding scores for a greater number of tools as well. You need to also add additional columns for “Formulae” as well, as shown in the Matrix, incase you are adding more columns. You have to finalise the list of selection criteria that you will use throughout your evaluation process against all the tools that you have considered.
What is Pugh Matrix?
Pugh Matrix (also called as Decision Matrix) is a qualitative technique used to rank several alternatives/options in order to make effective decisions by considering a set of selection criteria. Pugh-matrix uses weighting these selection criteria in order of importance. The more important a particula selection criteria the higher weightage score to be given. Example: IT Teams that primarily focus on ERP Applications will give more weightage for a selection criterion that evaluates the maturity of the tool around optimized ERP support from test automation perspective. There are multiple ways analysts use to baseline the rating for weigtage of the selection criteria. Our Pugh-Matrix uses 3 distinct weightage scores as below:
Extremely desirable: 9
Highly desirable: 6
Good to have: 3
At the right side of the sheet, we have to capture the ratings for our findings from the evlauation of each of the considered alternative tools. We need to ensure different teams participating in the overall evalaution process understand the rating system correctly. This will ensure an error free/unbiased decision making. Our Pugh-matrix uses 3 unique scores for capturing the individual evlaution results as below:
The Pugh-Matrix uses a formula underneath to calcualte the overall score for each alternative tool. It captures the total number of +ves, sames and -ves at the bottom and also shows the overall score that can be compared with the result of other tools.
Make sure the Pugh Matrix has been reviewed by all the stakeholders we identified earlier and their inputs have been captured accordingly. The champion/the owner of this exercise should store the Pugh Matrix in a central repository and grant necessary access privileges like read, edit etc. to different stakeholders.
The Pugh Matrix uses a rating/scoring formula that converts the inputs into a number at the bottom of the sheet. This number actually represents the overall score a tool gets in the evaluation process and can be compared with the scores of the other tools. To make the decision process more effective you can consider the tools that get the top 3 scores and then proceed with the sharp differentiation between them to make the final decision. We hope you got it. Good luck!
(A sample Pugh Matrix for Automation Tool selection process. Download Here)
Selection of the right automation tool is a critical part of an enterprise’s overall automation journey. Procurement of Test Automation tool needs substantial investment and also drives the overall operational cost of the QA organization.
This article discusses the overview of the evaluation process, the stakeholders and their responsibilities, several focus areas for finalizing the right selection criteria, and in order to document the overall selection process we also recommended a Pugh Matrix (a sample can be downloaded using the link above). The Pugh Matrix is a flexible template that can be modified and fine-tuned as needed for an evaluation exercise.
We hope this article was informative and please do let us know your comments/questions. Will appreciate it.