Few months back, on a Saturday dated August 5, 2023, I attended the vodQA conference organized by Thoughtworks. Out of all the talks and presentations, what piqued my curiosity was the keynote address given by Kumar Shankar Iyer (Engineering Director, Thoughtworks). The topic was “Testing Techniques for Modern Challenges”, and the content was interesting. I thought of writing this blog summarizing the points put forward by the speaker (I’ve obtained permission from the speaker to do so).
Lets understand the modern challenges and the possible testing solutions for them.
Challenge 1: Long-winded path to production.
One of the biggest challenges faced by most product teams today is the long time to production. That’s because we try to test our features thoroughly before making them available for actual users to use in production. But still, many critical bugs are leaked to production because of the lack of production-like data and user behavior in testing environments.
Now the question is, can we test our feature in production? For many, the straight answer will be “No”. Simply because of data safety related reasons.
Then how can we test our feature with production-like data patterns and user behaviors? The answer can be “Canary Testing”.
Solution : Canary Testing.
Canary testing refers to testing a new software version or a new feature with real users in a live (production) environment. It is done by routing a small portion of production users (ideally around 5%) requests to the new version of the application without making this small group of users aware that they are using the new version of the application.
Since the new code, or canary, is only distributed to a small group of users, its eventual impact is relatively small. Moreover, the changes can be quickly reversed should the new code prove to be buggy or cause problems for users.
Now, let’s come to the next challenge, which is somewhat related to the first challenge.
Challenge 2: Test Data Management.
If we test in production or try to mimic production data or user behavior, the challenge is the regulatory, monitoring, and financial risks that come along. Canary testing will also be risky, as those 5% of traffic, if faced with a financial or regulatory bug, might cause severe damage to the company in terms of revenue and reputation.
Now the question may arise: if not Canary Testing then what else? The answer can be “Replay Testing”.
Solution : Replay Testing.
Replay testing is somewhat similar to Canary testing, but with an interesting twist. Here, we route the same traffic to both the existing and new application versions, but we will disconnect the response from the new application version to the users. All the users will still be using the existing application version. And we can monitor the new application version’s behavior with the production user data and behavior without impacting any of the users.
In the above architectural diagram, you can see that the same (100%) traffic is being routed to both existing and new application versions. But the response from the new application version is disabled. So the users will always be using the existing application. And we, the product team, can monitor the new application version’s behavior with the same user traffic.
Since none of the users are directly exposed to the new application version, we can avoid regulatory or financial risks in the event of any critical bugs being identified.
Canary Testing vs Replay Testing
Challenge 3: Complex infra architecture.
Most of us can agree that our application or system architecture is more complex than before. That might be because we want to use cloud, container, or Kubernetes infrastructure while at the same time not compromising the security of the application. There might be other reasons for adapting a complex architecture. And in the event of any security-related issues, we don’t know who is responsible!
Now, how do we test this complex infrastructure or architecture to reduce security risks at an early stage? The answer can be “Open Policy Agent (OPA)”.
Solution : Open Policy Agent (OPA).
OPA provides a high-level declarative language that lets you specify policy as code. You can use OPA to enforce policies in microservices, Kubernetes, CI/CD pipelines, API gateways, and more. Using OPA, we don’t necessarily write tests; instead, we write policies to verify the infrastructure.
There are other solutions to this challenge, like Security Testing, Configuration Testing, Reachability Testing etc.
Challenge 4: Quality Assurance in Creative domains (Gen-AI based systems).
We are talking about modern challenges, and to not mention Gen-AI (the hottest topic in the current tech industry) will be simple negligence.
We often talk about using Gen-AI in the testing world. Be it designing test cases or test strategies, writing automation or performance test scripts, or more.
But have we thought of testing a Gen-AI based system or application? This is something that will be trending in the future, with more and more applications starting to use Gen-AI. How to test these kind of applications?
Solution : Fitness assessment frameworks.
We need to transition our testing mindset from correctness to fitness!
Currently, while testing an application or a feature, we know what the exact expected outcome should be. So while testing, we just verify the output to be either this or the test has failed.
But in the case of Gen-AI applications, it’s not that simple. Let’s take the example of ChatGPT. If we ask a question, ChatGPT will prompt an answer. If we ask the same question again, ChatGPT will prompt another answer. And we might get multiple different answers to the same question if we keep asking the same question to ChatGPT. For ChatGPT, all the answers may be correct, but for our context, not all the answers will be correct. So how do you determine which answer(s) are correct?
There are many fitness evaluations and benchmarks available on the market to test Gen-AI based systems. There will be many such things in the future, with more Gen-AI based systems being developed. We should keep evaluating their fitness instead of their correctness.
So to summarize, testing Generative AI-based systems calls for a paradigm shift!
- Embrace probabilistic outcomes
- Shift your Metrics
- Understand Bias and Ethics
- Use benchmarking datasets
- Think in scenarios
- Learn from each release
There are many other modern challenges, which vary from industry to industry and application to application. And to test these challenges, we need to have an open mindset to adapt new techniques and tools. We should be open to exploring or designing new testing methodologies and frameworks to test current and upcoming modern challenges.
Read original blog here.
Thank you for reading this post.