How to use ChatGPT for software test automation?

Test automation is vital in software engineering and is deployed for fast test execution, timely error detection, and to check for consistency and reproducibility. Generative AI in testing is in lockstep with OpenAI’s ChatGPT advancements.

Generative Pre-trained Transformer or ChatGPT is an artificial intelligence (AI) system that can tell stories, poems, essays, and, yes, even code. It can also engage in dialogue and answer questions in natural language.

However, it is worth remembering that ChatGPT is a language model that essentially still acts as a really advanced ‘auto-complete,’ and as such, it may not be the best tool for all test automation cases.

ChatGPT aims to produce responses that sound like a person wrote them when given text input. While it has training in various areas and tasks, people use it for tasks involving natural language processing.

These tasks include creating text, translating languages, answering questions, and summing up information. Testers and quality assurance (QA) specialists can also use ChatGPT's AI abilities to make code testing easier in several ways.

Artificial intelligence testing tools can also help with automated testing. This article will explore some methods of using ChatGPT to speed up and automate the testing process.

Using ChatGPT for QA automation overview

One of the things that makes Artificial intelligence in testing interesting to test automation engineers is its ability to generate relevant and properly formulated code through a simple natural language request. It can generate code in numerous languages and can also employ many built-in packages across those languages.

So, the inevitable question becomes, how can you use ChatGPT for QA automation? First, ChatGPT can generate feature files (Behavior Driven Development (BDD) scenarios) and Selenium-based code in multiple languages; however, writing code is just the first step.

In a perfect world, a test automation engineer would give ChatGPT a description of what test to generate; it would know everything about the website being tested and generate flawless, executable code that requires no further changes. Currently, ChatGPT cannot do this, but what it does is still impressive.

Testers can tap into its strength which is generating natural language using associated source code and is crucial in powering the transition from manual to automation testing.

Let’s explore how automation engineers can use AI to augment software testing by simplifying coding.

Benefits of ChatGPT for automated testing

Let's look at some of the most exciting benefits ChatGPT brings to automated testing:

Writing code from scratch

Starting from a blank slate can be intimidating. With ChatGPT for automated testing, you can start the process of writing code from scratch. It can provide initial guidance and structure, or step-by-step code suggestions tailored to your specific testing requirements and technical stack, ensuring that your testing code is built on solid foundations, saving you time and reducing errors.

Transforming manual test cases to autocode

With ChatGPT testing, you can transform manual test cases into automated scripts. No matter in which format manual test cases are provided: it might be plain text documentation, tabulated formats in spreadsheets, or even manual test cases are structured within a specific Test Management System (TMS) like XRay, Azure DevOps (ADO), or TestRail.

The process can be time consuming without ChatGPT’s help. But with using ChatGPT or any other Generative AI chat, testers can streamline the processes of transforming manual test cases to the programming code.

Continuous learning for smarter testing

Using ChatGPT for automation testing means getting access to new technologies and evolving testing environments as they receive updates. It not only learns from past test runs but also suggests relevant learning resources to keep your skills sharp and updated. This continuous learning mechanism is essential in today's fast-paced software development landscape, allowing testers to stay knowledgeable about emerging tools, frameworks, and best practices.

ChatGPT offers personalized recommendations for online courses, tutorials, articles, and community resources tailored to individual learning needs and preferences. Whether you're a beginner looking to grasp the fundamentals of test automation or an experienced tester wanting to dive into advanced topics like AI-driven testing, the tool curates materials that match your current proficiency and growth goals.

Testers can also benefit from customized learning with AI-assisted tools to keep their skills sharp.

Transforming API contract to autocode

When dealing with API testing, automation can be started from API contract, not manual test cases. Given an API contract, ChatGPT can be leveraged to generate the necessary automated test scripts and client stubs, providing a seamless bridge between design and implementation.

API contracts outline the expected behavior, endpoints, parameters, and response formats of the API. By using ChatGPT, teams can effectively transform these contracts into code that ensures robust testing and integration. By transforming API contracts into autocode, teams can ensure that their testing frameworks align perfectly with the expected behavior of the API. This method not only accelerates the development of testing scripts but also enhances test coverage by ensuring that every defined endpoint and response scenario is adequately addressed.

Maybe it would be good to add examples?

Example: Imagine you have the following API contract in OpenAPI format:

Using ChaGPT can help in creating feature and code, for example, using Cucumber:

Gherkin

UserSteps.java

Will the code be flawless from the very first attempt? For sure, no. But for sure will save the time.

Generating test data for automation run

ChatGPT for QA testing can help generate realistic test data tailored to your specific needs. This ensures that our automated tests are comprehensive and can handle a variety of scenarios, improving the efficiency of your software.

Test data might be formed as JSON or XML, domain format like HL7 or Mismo, SQL insert statements, csv/tsv files or just plain text.

Generating synthetic yet representative data allows teams to simulate real-world usage, capturing typical scenarios such as:

• Valid Inputs: Ensuring that the system behaves as expected with correct data.
• Invalid Inputs: Testing how the system handles errors and exceptions.
• Boundary Values: Checking the limits of input fields to identify potential overflow or underflow issues.
• Large Data Sets: Evaluating the application's performance and scalability under heavy loads.

Assistance for self-review of automated code

ChatGPT can assist in reviewing your automated code by highlighting potential issues and suggesting improvements. This helps maintain high code quality and ensures that your automation scripts are efficient and effective.

Challenges of using ChatGPT for testing automation and ways of solving them

Using ChatGPT help or testing software is great, but it's not flawless. Like any tool, it has its limitations, and understanding these limitations will help you create a more informed roadmap for QA automation testing implementation.

Let's examine the common problems and how you can address them.

Ensuring test cases are accurate

The accuracy of ChatGPT's code generation largely depends on the clarity of your input. If you lack specificity, it might overlook crucial scenarios. For example, if you state, "Prepare a Java Code to test the signup form" ChatGPT might not consider preparing a code for testing password resets or the outcomes of failed validation.

The fix? Provide crystal-clear directions with plenty of specifics. When your inputs are precise and well-outlined, you'll receive granular but correct code. Plus, it's smart to review ChatGPT's output and tweak it as necessary. This approach ensures you don't overlook any crucial elements.

Data privacy and security

When you're testing software that deals with sensitive info—like finance or healthcare apps—data privacy is crucial. It’s essential to ensure that sensitive information is not fed to ChatGPT.

The smartest approach is to use anonymized data instead. This keeps private information like customer profiles or purchase records safe from exposure. Also, adhering to industry standards such as the General Data Protection Regulation (GDPR) or the Health Insurance Portability and Accountability Act (HIPAA) can shield your operations and safeguard your users' privacy.

Getting your team up to speed on ChatGPT

ChatGPT is user-friendly, but using it well for automated testing still takes practice. Teams need to learn how to communicate with the AI, how to prepare inputs, and how to interpret outputs to achieve good results.

Investing in team training is important. Teach your engineers and testers how to use the tool, give them clear guidelines on best practices, and encourage them to experiment. When your team becomes experts with ChatGPT for test automation, they'll extract more value from it.

Handling complex test scenarios

ChatGPT excels at simple tests. However, for more intricate setups— such as workflows with conditional logic or multi-step processes—it might require some additional assistance.

Breaking down complex scenarios into smaller, more manageable parts is a smart strategy. You can also combine ChatGPT's automation with standard testing tools to achieve complete coverage. For the most intricate test cases, human oversight and intervention can help catch any potential oversights.

I asked Tasha Bogushevskaya what the EngX course will cover, with regard to issues of test automation with ChatGPT?

She explains, “This course focuses on typical tasks for manual engineers only. It also covers some aspects of using ChatGPT for API Testing (with Postman), creating test data in JSON/SQL format.”

ChatGPT for software testing: inspiring examples

When learning how to become an automation tester, it is always a good move to learn from the best examples of your chosen career path. Natalia Bogushevskaya shared some highlights and outcomes from pilot projects involving two enterprise platform teams.

These teams leveraged GPT models to enhance testing processes and experienced significant productivity boosts, alongside unique challenges.

Case study 1: Accelerating API test development

One team focused on generating API tests directly from contracts. Using generative AI, they transformed API documentation into robust, documented test cases, and automated test scripts. This approach reduced manual coding efforts and improved coverage.

Results:

• Automated test scripts increased from 57 to 89, a 49% rise.
• The development time for each script dropped by 38.8%, from 3.27 hours to 1.87 hours.

The team noted that the effectiveness of generative AI depended on the experience of the quality engineer. Senior engineers could craft precise prompts and quickly validate suggestions, enabling better outcomes.

Case study 2: Enhancing test coverage and quality

Another team used generative AI to refactor existing tests, adding parameterization and refining assertions. AI-assisted test generation expanded their documented tests and automated test portfolio.

Results:

• Documented tests increased by 32%, while automated tests rose by 29%.
• The team utilized generative AI for auxiliary tasks like generating SQL scripts, which streamlined database testing.

While generative AI significantly reduced time spent on repetitive tasks, team members with less experience in testing or AI found it more challenging to utilize the tools effectively.

These real-world examples illustrate how generative AI like ChatGPT can transform software testing by automating repetitive tasks, improving test coverage, and enhancing productivity.

ChatGPT for testers can be used in formulating how to transition from manual to automated testing. However, remember to keep skilled testers in the loop, as these AI tools can still make mistakes or carry over biases from their training data.

Best practices for using ChatGPT in test automation

Here are some best practices to help you make the best use of ChatGPT as you take the life journey of a QA-automation engineer:

Clearly define your problem

Before even logging into ChatGPT, ensure that you fully understand the problem that you need to solve. Make sure that you can express it as concisely as possible by breaking down the problem into smaller outcomes and defining the functionality or outcomes that you are looking for.

Be specific

ChatGPT will give you general information if you give it general prompts. You need to specify what you need it to do.

Define the rules, preconditions, and outcomes or desired features the same way you would if you were writing code. Don’t forget to specify programming language and framework(s). Provide examples of already created code if possible.

This will increase the chances of getting precise and actionable results from ChatGPT.

Contextualize your prompts

Giving ChatGPT contextual information will enable you to define the coverage of what you need and ask it to come up with a comprehensive automated testing coverage for that branch of the app.

Iterate and refine

Think of ChatGPT as a toolbox whose manual you’re yet to have. You cannot expect to just log in, enter your prompt, and get the ideal solution on the first try.

Learning how to become a QA automation engineer will involve mastering prompt engineering so you can learn the various approaches and refinements that get the best results from AI tools.

Converse with ChatGPT

ChatGPT isn’t just a useful and simple tool at your disposal; it is more like a programming partner. You need to understand how to have a conversation with it.

Viewing it as a collaborator will help you think more critically and discover more creative solutions and also enable the AI to “think” more comprehensively about the problem you’re both solving.

Students considering taking the EngX course may be wondering whether it teaches them how to use ChatGPT for test automation. Bogushevskaya says, “The course doesn't cover typical automation tasks,” adding that there is good news, though. EngX publishes new courses periodically, and its library will soon include a course on automation QA.

In the meantime, you can get some pre-reading done with this list that may contain your next best book for automation testing.

Learn to effectively use ChatGPT for automation testing in the course from EngX

Engineering Excellence (EngX) is a program that began in 2014 to provide software engineers with services, tools, and products to enhance their software development performance.

These tools and services help engineers identify areas for improvement, monitor key metrics, improve processes, drive continuous development, and hone their skills to start working QA jobs with confidence and great results. Many organizations can benefit from using extensive and easily accessible tools to foster a healthy engineering culture.

You can also learn more about the most popular tools for mobile testing.