Test Automation Forum

Welcome to TAF - Your favourite Knowledge Base for the latest Quality Engineering updates.

Celebrating our 2nd Anniversary!


(Focused on Functional, Performance, Security and AI/ML Testing)

Brought to you by MOHS10 Technologies


Focused on Functional, Performance, Security and AI/ML Testing

How To Setup Effective Continuous Testing In DevOps?

Continuous_testing-Testautomation Forum

Have you ever wondered how the ongoing COVID19 pandemic has impacted the enterprises to realign their strategies for their survival in these uncertain times? Sudden fall in the customer demand led to huge impact on the financial health of the companies that forced them to be more adaptive than ever. And DevOps comes to rescue in this scenario as this is conceptualized primarily to enable an adaptive software release process while making no compromise on customer experience.

Sources say by 2021, ~55% of world’s population (i.e. more than 4B) will have access to internet. There will be around ~37 billion devices (including smartphones, IOT, wearables, etc.) connected to the Internet. And yes, 5G already started finding its existence in few countries and this will be accelerated in 2021 and beyond. These numbers represent the market situation at a high-level. And thanks to the COVID19 pandemic, the world is connected digitally better than ever as everything goes virtual like online classes, working from home, webinars and so on.

The above data also signifies what would be the impact if software fails due to bad design/coding, improper testing, security vulnerabilities, bad performance and so on. As per the “Software Fail Watch” report published by Tricentis in 2018, the loss due to software failure was estimated to be around USD 1.715 Trillion that impacted around 3.7 billion people worldwide.

Why DevOps?

Software development tasks in DevOps are continuous (coding, building, testing, etc.) because it is designed around adaptiveness and speed which are the 2 most critical factors for success. In order to start with this topic, I would like to discuss a bit around an old depiction of PPP Graph (Process, People and Product), as in the chart below:

As we can see in the chart, your software product’s success is primarily dependent on broadly 2 parameters i.e. process and the people who are developing the product. The organization can achieve a better product quality by continuously improving these 2 parameters. Other competitors will also try their best to bring maturity around these 2 factors to succeed in the market.  And if you examine carefully DevOps also uses this principle at its core and yes of course, Agile plays a complementary role as well in DevOps space. And also, we need to keep in mind that modern techniques like AI/ML are influencing the ‘People’ part of the above chart specifically to bring more intelligence to the process.

Shift-Left, Shift-Right And Continuous Testing

Continuous testing in the build pipeline is an integral part of the overall DevOps setup that executes the testcases in parallel and I’ll explain that shortly.

We are going to cover Functional testing, Performance testing and Security testing under Continuous testing. As you know “Shift-Left” is a buzz-word in DevOps which actually refers to starting QA automation tasks (like Functional testing, Performance testing and Security testing) as early as possible in the Development process. We also call it “in-sprint” automation in Agile language.

Please refer to the above chart that illustrates the different pieces of Continuous Test automation process in DevOps.

As you can see in the illustration, the test automation rigor starts much earlier in the life-cycle (i.e. Shift-left) and in a matured DevOps model the Metrics related to testing is collected at each phase and fed into an Analytics platform (can also be AI/ML based) for better predictability and continuous quality improvement. These tools/platforms can be integrated together so that they can smoothly talk to each other and enable touchless test executions.

Also, please note that some enterprises adopt Shift-Right approach to even test their application/product in the Production environment in order to evaluate the actual behavior/performance in real-time.

As the continuous code delivery matures and Agile teams compress sprint cycle, the window of testing also shrinks due to minimal or zero rework. The idea here is to have all the automation script ready before the build is available so that the test execution can start as soon as a build is deployed. This also enables adequate test executions as the work-product moves forward towards it’s final release. Let’s discuss each of these testing areas one by one.

Continuous Functional Test Automation (CFTA)

What is your favorite automation tool? We will discuss more about several automation tools and their special advantages in upcoming articles but now let’s try to understand how our Functional test automation tool fits in the DevOps build pipeline.

As DevOps focuses on shortening the SDLC in order to release high quality software product, Continuous test automation becomes critical to the overall success for the product. And not only that, over last few years Test Automation tools evolve to become more and more smarter, autonomous and universal. This has enabled superb testing intelligence by infusing AI and ML. AI/ML techniques like Natural Language Processing (NLP) are very helpful in terms of smart recognition of objects which in turn brings additional speed to automation by minimizing the rework due to code/UI change. In this way continuous testing tools can provide quick and timely feedback on business risks through their analytics capabilities. I’ve listed few such popular commercial tools/providers that fit into your continuous test automation space:

  • Eggplant
  • Tricentis
  • Parasoft
  • Micro Focus
  • mabl
  • Broadcom
  • Sauce Labs
  • Perforce Software
  • IBM
  • FrogLogic
  • Ranorex
  • SmartBear Software
  • Cyara
  • Worksoft
  • Experitest

If you are an expert in any of these tools then it will be great to have you sharing you story in this forum. Don’t hesitate to contact me if you think you have a great story to share with our readers.

Continuous Performance Testing

Gone are the days of doing Performance testing much later in the SDLC i.e. after the system integration testing (SIT) and just before going live. It becomes an annoying experience when you have several critical performance issues detected during performance testing and go-live is planned just a few days after. This sometimes puts your go-live at jeopardy and the rework might be very costly as well.

As Application Performance also determines success or failure of your product’s release, it should be a prime design criterion and has to start much early in the development process which is called Shift-Left Performance testing. This is also called as Performance Engineering as there are other tasks involved much beyond the Load Testing.

Shift-left Performance Testing also enables the developers to initiate the Performance testing for a single or very few users to see how their code is actually performing during the Unit Testing phase. This also helps in enabling reusability between Functional Testing and Performance Testing e.g. Selenium and Jmeter (HAR file conversion to JMX file) that can help in saving a lot of time and effort and also encourages better collaboration between Fnctional and Performance testing teams.

Below are few popular Performance Testing tools that can be integrated to your DevOps setup in order to enable Shift-Left Performance Testing:

  • WebLOAD
  • LoadNinja
  • MicroFocus LoadRunner
  • LoadView
  • StresStimulus
  • Apache JMeter
  • SmartMeter.io
  • Rational Performance Tester
  • Testing Anywhere
  • NeoLoad
  • BlazeMeter etc.

Continuous Security Testing (DevSecOps)

Over the last decades, hackers have adopted new methods continuously and breached the defences that led to loss of sensitive customer data, customer frustrations, loss of revenue, brand-value loss and so on.

In 2017, a group of hackers tracing to Finland attempted to acquire around 10GB of data from a North America based casino by using an Internet-connected fish tank via the sensors connected to a PC that regulated the temperature, food and cleanliness of the fish tank. To know more about the incident follow the URL here: https://www.washingtonpost.com/news/innovations/wp/2017/07/21/how-a-fish-tank-helped-hack-a-casino/

Just like Functional and Performance Testing, Security Testing must Shift-Left in order to minimize the rework and ensure a secured app is released to the customer. By integrating Security Tests into the DevOps process, developers can detect the risks/vulnerabilities in the code and can remediate much early.

The OWASP® Foundation (www.owasp.org) works to improve the security of software through its community-led open source software projects and devised the Top 10 web-application Security Risks. And these Top 10 Security risks are probably the most effective areas to consider while planning for a penetration test for web-based applications.

Few 2020 industry popular Security testing tools are listed below:

  • Micro Focus Fortify
  • AcuNetix
  • CheckMarx
  • NetSparker
  • Veracode
  • Coverity
  • SonarQube
  • PyCharm
  • HCL AppScan
  • Resharper


  • DevOps is designed for adaptiveness, speed and quality to deliver a rich customer experience.
  • Shift-Left and Shift-Right techniques must be considered appropriately in order to achieve the desired level of quality (Functional, Performance and Security).
  • Automation tools (Functional, Performance and Security) must be integrated as part of the build pipeline (CI/CD) in order to enable touchless execution e.g. Jenkins based CI.
  • Right Metrics/KPIs etc. should be identified in each software development phase and if possible, this data can be fed into a data analytics system or AI/ML platform in order to have better software health predictability and continuous process improvement.
Total Page Visits: 1063

Submit your article summary today!

[wpforms id="2606"]

Thank you for your interest in authoring an article for this forum. We are very excited about it!

Please provide a high level summary of your topic as in the form below. We will review and reach out to you shortly to take it from here. Once your article is accepted for the forum, we will be glad to offer you some amazing Amazon gift coupons.

You can also reach out to us at info@testautomationforum.com