In DevOps environment, Performance Testing is not about using a commercial or open source performance testing tool to test & certify the system for its Performance & Scalability characteristics. DevOps culture has brought a tremendous improvement in proactive performance measurement & analysis possibilities, but performance testing team always have challenges to adapt to the new needs of the DevOps environment.

As this culture brought Performance as everyone’s responsibility, it significantly removed all tool platform silo’s & open source tools became the popular option, be it for maintaining code & version control, building, deployment, testing, release, production monitoring, etc. Hence, most of the popular commercial performance testing tools & application performance management tool vendors started providing plug-ins for integration with popular CI servers (like Jenkins, Bamboo , TravisCI, etc), to manage performance test execution & server performance monitoring activities from CI server itself (to a good extent). The checked in code (upon code commit), gets automatically build & deployed in various test environments to carry out functional & non-functional testing test & upon test pass criteria being met, gets deployed in production enabling continuous delivery & continuous deployment to production environment.

Continuous Integration practice streamlines code integration using shared repository and has automated build & testing process to validate & find problems quickly to provide rapid feedback to developers. Continuous Delivery practice encompasses develop, build, test followed by making the software release ready, though production deployment is handled manually. Continuous Deployment pushes the code that has been successfully accepted in the CI/CD cycle into the production environment without human intervention.

Key Elements of Performance Testing Framework in DevOps Environment

1. Define Performance Test objectives / SLAs (into user stories)

2. Strictly include Performance in ‘DoD’ & Performance analyst as part of scrum team

3. Define when / what to Test & its KPIs

  • Continuous Integration (CI) Performance tests on nightly builds
  • Sprint level : New Feature Performance tests / Regression Performance Tests
  • Hardening Sprint : System level Performance tests

4. Clearly Define Automated Test Result ( Pass / Fail ) Criteria

  • Transaction Response Time (Average or 90th percentile value)
  • Web page loading / rendering Time
  • Acceptable Error rate (Transaction failures)
  • Server throughput (Hits / sec or Transactions/sec)
  • Server CPU & Memory utilization level

5. Define what to test in various environments (which s/w version? & part of CI pipeline? )

  • QA / Staging / UAT / Pre-Prod

6. Choose the right performance testing tool

  • Integration with CI server (like Jenkins/Bamboo)
  • Integration with reporting tools (like Kibana/ Graphana )
  • Integration with APM Tools
  • Handy & automated Performance Test Report Dashboards
  • Quick & on-demand scaling of virtual users
  • Quick & automated Load Generator Setup (using Docker container)
  • APIs to Read / Share Test data from open source tools

7. Choose the right APM tool

  • To monitor test & production environments (full stack monitoring)
  • To perform Performance Engineering analysis (on Containers / Servers/ Cloud, etc)
  • To bring shift+ right techniques to create early & quick feedback loops

The Key Performance Indicators (KPIs) considered during early performance tests & system level tests can be different. Metrics like Response time, Server Throughput, Server Resource Utilization (CPU & Memory utilization) & Error rate are usually considered in build performance comparison analysis. Don’t over measure & toggle additional metrics only on demand basis. The CI build pass/fail criteria should be automated based on the SLAs set for these KPIs. If the test pass criteria are met, the build is promoted to next stage, else fail the nightly build & the automated performance dashboards are available for the entire team to understand the reason for build failure. Providing detailed root-cause analysis feedbacks on the code performance with the support of APM tool can help developers to nail down the issues quickly to address them.

Happy Performance Testing & Engineering in DevOps Environment