In this blog, we deep-dive into API Automation Testing and also discuss our in-house API framework.

What is an API? 

An Application Program Interface (API) is a set of commands, functions, protocols, and objects that programmers can use to create or interact with an external system. 

For example, imagine you're at a library looking for a specific book. The library shelves are like the system holding vast amounts of information (or data), but finding what you need quickly is a challenge. Enter the librarian—your API. 

You approach the librarian with your request (the name or type of book you’re looking for). The librarian understands your request, goes to the appropriate shelf (the system), retrieves the exact book (the data), and hands it to you. You don’t need to know where the book is stored or how the librarian found it—your interaction is simple, and your request is fulfilled efficiently. 

An API acts in a similar way: it bridges the gap between a user and a system, making complex data retrieval and interaction effortless. 

What is API Testing? 

Think of a button that shares something from a website to Facebook for instance. To make sure they meet functionality, security, performance and reliability expectations, APIs must undergo API testing. 

API testing implies making requests to an API and getting responses back, that should be thoroughly checked for data accuracy, right status codes, as well as error codes. API testing mostly concentrates on the business logic and is conducted during integration testing. 

QA experts verify data in the output, response time, ability to ride out hacker attacks, API reaction to various failures, unusual inputs, and many others. 

API Automation Testing 

API automation testing involves using automated testing tools or frameworks to test an application's API. The purpose of API testing is to verify the functionality of an API by sending requests to it and validating the responses. 

There are various tools and frameworks available for API testing, including Postman, SoapUI, JMeter etc. Postman is a widely used tool for API testing and development, while SoapUI is specifically designed for testing SOAP and REST APIs. JMeter is typically used for load and performance testing but can also be used for API testing. 

When conducting API testing, it's important to consider factors such as security, performance, and functionality. Automated testing tools can help streamline the API testing process and ensure that APIs are working as intended. 

Evolution of Manual API Testing to CI/CD 

Traditionally, API testing was performed manually by testers or developers. Here, testers would manually send requests to APIs, verify the responses, and check for issues such as incorrect data, missing functionality, or performance problems. While manual testing can be effective, it is prone to human error, time-consuming, and not suitable for frequent testing as part of a CI/CD pipeline. 
 
To improve efficiency and reduce human error, organizations began automating API testing by using tools and frameworks, such as Postman, SoapUI, or custom internal frameworks for enhanced security. These automation tools were used to write scripts that automatically sent requests to APIs, validated responses, and compared them against expected results. Automation allowed for faster and more reliable testing, and test scripts could be run repeatedly. 
 
Since API testing became an integral part of the CI/CD pipeline, organizations adopted continuous integration and deployment practices. APIs are critical components of modern applications, and ensuring their quality and reliability is essential for successful deployments. API tests are executed continuously at various stages of the pipeline, including unit testing, integration testing, and end-to-end testing. 

The automation and integration of API tests into the CI/CD pipeline enable teams to catch issues early, iterate faster, and deliver high-quality API-driven applications. 

Implementation of Quality Gates 

Step 1: Define Quality Criteria 

In this step, we specify the quality standards that our APIs should meet. These criteria encompass functionality, performance, and security requirements. For example, functionality criteria might include the correct implementation of API endpoints and expected responses. Performance criteria could involve response times and scalability requirements. Security criteria may address vulnerabilities such as authentication and authorization. 

Step 2: Develop Automated API Tests 

Create automated tests that validate the APIs against the quality criteria defined in the previous step. This involves writing test scripts using testing frameworks like Postman, Selenium, or scripting languages such as Python. These tests simulate interactions with the API to verify its functionality and performance. 
 
Step 3: Analyze Test Results 

Execute the automated tests against the APIs you're evaluating. Analyse the test results, which include pass/fail outcomes, performance metrics, and any identified security vulnerabilities. The goal is to gather data that will be used to determine whether the API meets the quality criteria. 

Step 4: Set Rules for Passing Quality Gates 

Based on the analysis of the test results, establish specific rules and thresholds for passing the quality gates. For example, if our API must respond within 200 milliseconds, any response time exceeding this threshold would trigger a failure. 

Step 5: Integrate with CI/CD Tools 

Integrate the API testing process into your CI/CD pipeline. CI/CD tools can be configured to automatically run the API tests at specific stages in the deployment pipeline, ensuring that the tests are consistently executed before deployment. 

Step 6: Create a Gate Check Step in the Pipeline 

Insert a gate check step just before the deployment stage in our CI/CD pipeline. This step evaluates the API against the quality criteria and rules established in step 4. If the API passes the checks, the deployment proceeds; if it fails, the deployment is halted. 

Step 7: Notifications and Reporting  

Implement a notification system to alert the development team and stakeholders when a gate failure occurs. Detailed reports, generated as part of the testing process, provide insights into why the API didn't pass the quality gates. Notifications and reports help initiate the issue resolution process. 

Step 8: Issue Resolution 

When a gate failure is detected, the development team must review the issues, identify the root causes, and fix the problems in the API. After making the necessary fixes, retest the API to ensure that it now passes the quality gates. This iterative process continues until the API meets the quality criteria. 

Step 9: Continuous Monitoring in Production 

Even after successful deployment, continue to monitor the APIs in production. This involves performance monitoring, security scanning, and regular testing to ensure that the APIs maintain their quality and adhere to the defined standards over time. 

Step 10: Promote Reliable Deployments 

The goal is to promote reliable deployments by preventing the release of low-quality code. Quality gates ensure that APIs consistently meet their functionality, performance, and security requirements, thus enhancing the reliability of deployments and reducing the risk of issues in production. 

Each step in this process is crucial for ensuring the quality of our APIs throughout the CI/CD pipeline, from development and testing to deployment and ongoing monitoring. It helps us deliver high-quality software with confidence. 

Microservices 

Microservices are a contemporary architectural approach to building applications as a collection of small, independent services that work together to deliver the complete functionality of the application. When it comes to API testing and automation, microservices have a significant impact: 
 

Microservices Architecture 
Microservices architecture is a modern approach to designing software applications. It involves breaking down complex monolithic applications into smaller, self-contained services that interact through APIs. Each of these services, known as microservices, operates independently and can be developed, deployed, and scaled separately. They communicate with each other through APIs, which serve as the interfaces for requesting information and services. Microservices are designed to be autonomous and resilient, meaning they can function independently and gracefully handle failures without disrupting the entire system. This architecture allows for efficient scalability, agility in development, and the use of diverse technologies for different microservices. It's often associated with continuous integration and deployment (CI/CD) practices, making it suitable for rapidly evolving software environments. 

API Testing Challenges in Microservices
Testing Multiple APIs: In a microservices environment, there are numerous individual microservices, each with its own API. These APIs must be tested independently to ensure they function correctly and meet their specific requirements. This means testing a multitude of APIs, which can be time-consuming and complex. 

Consistency: Maintaining consistency across different APIs and microservices can be challenging. This consistency includes ensuring uniform data formats, error handling, authentication mechanisms, and other aspects. Without consistency, the overall system can become fragile and prone to issues. 

Versioning and Compatibility: Microservices can evolve independently, leading to changes in their APIs over time. Testing for version compatibility and backward compatibility becomes essential to ensure that new and existing clients can interact with these services without issues. 

Scalability Testing: Microservices can scale independently. Testing for scalability involves verifying that as services are scaled up or down to handle varying workloads, the system remains performant and reliable. 

 
Role of API Testing in Microservices
Ensuring Individual Microservice Functionality: API testing verifies that each microservice operates as intended. It checks if these individual services correctly execute their specific functions, which is essential to the overall functionality of the application. 

Validating Interactions: Microservices interact with one another through APIs. API testing confirms that these interactions are seamless and error-free. It helps identify potential issues such as miscommunications or data inconsistencies between services. 

Reliability: In a microservices architecture, where each service can evolve independently, maintaining reliability is critical. API testing ensures that changes in one microservice don't negatively impact others, preventing unexpected disruptions and maintaining the overall reliability of the application. 

Fault Isolation: Microservices are designed to be autonomous and resilient. API testing helps isolate faults or issues within individual microservices, preventing them from propagating to other parts of the system. This ensures graceful degradation in the face of failures. 

Version Compatibility: As microservices can undergo version updates independently, API testing is vital to check for version compatibility. This ensures that new and existing versions can communicate effectively, avoiding compatibility issues. 

Scalability: Microservices can be scaled independently to meet varying workloads. API testing verifies that, as services are scaled up or down, the entire system remains performant and reliable. 

Security Assurance: Security is a significant concern in microservices. API testing helps identify and address security vulnerabilities specific to APIs, such as unauthorized access, data leaks, and other security threats. 

In-House API Framework 

At Valorem Reply, we are using an in-house API framework to automate API. The framework follows BDD inspired format (Behavior-Driven framework), which helps by providing a proper structure for defining and executing tests in a human readable format. The “Given-When-Then” format is used to describe the expected outcomes of a software functionality. 
 
Given: This describes the specific condition under which the test is being conducted. It typically includes configuring data, setting up environment variables, or preparing test data. 

When:  This describes any specific action that is being performed. It typically includes interacting with the system, executing a specific function, or simulating user actions. 

Then: This specifies the result of the tests. It describes the expected state of the system after the actions defined in “When” section occurs. It typically includes assertions or conditions that need to be verified to ensure the system is behaving as expected. 
 

API Test Scenarios 

The API test scenarios we commonly cover in-house are as follows: 

Happy Path Testing: This scenario ensures that the functions are correctly behaving under normal conditions by passing valid inputs. It verifies that the API handles requests successfully and returns the expected responses. 

Authentication (AuthN) testing: This scenario tests the authentication mechanisms like username/passwords, tokens, or API keys. It validates that the users are successfully authenticated and can access authorized resources. It also tests how authentication failures and invalid credentials are handled. 

Authentication (AuthZ) testing: This scenario tests the authorization rules and permissions associated with the API endpoints and resources. It also checks if the appropriate error responses are returned for unauthorized access. 

Negative Testing: This scenario makes sure the API accepts unexpected or invalid inputs gracefully, testing with these inputs. It also verifies that the Api provides appropriate error and error messages in response to various problem situations. 

Parameter testing: This scenario verifies the correct handling and processing of input data by passing different combinations of parameters. It also tests optional parameters to ensure that their absence does not affect the functionality. 

Data leakage testing: This scenario verifies that the API does not expose confidential data in payloads, headers, or error messages. 

Cross-customer Validation: This scenario helps in validating the functionality of an API when interacting with data belonging to different customers or users. It also ensures that the API correctly handles data isolation and maintains the integrity and security of customer-specific information. 

CI/CD Integration for API Automation Testing 

Our team set up a nightly pipeline that plays a crucial role in ensuring the reliability and stability of our API testing process. During the nightly run, our API tests are triggered to validate the performance, functionality, and security aspects. With the help of this regular testing cycle, we can quickly spot any problems or regressions and fix them. We may do thorough testing overnight without affecting our APIs' availability or performance during peak usage periods by using the off-peak hours. 

The nightly pipeline includes several phases, such as test planning, execution, analysis of the results, and report creation. To make sure that our APIs adhere to the necessary standards and criteria, test cases are run against various API endpoints, covering a variety of scenarios. To spot any errors or departures from expected behavior, test results and logs are gathered and examined. 

The pipeline produces thorough test results that give us information about the condition of our APIs and make it easier to prioritize bugs and address issues. These reports assist us in tracking the status of problem fixes, keeping an eye on the quality of our APIs over time, and ensuring that our API infrastructure is constantly being improved.  

Our framework is able to automate the generation of reproducible steps within our test logs. This will enhance the efficiency of our debugging and issue resolution process. When a test fails, the framework is able to capture details including the specific actions taken, the test data utilized, and the system status at the time of failure. Developers and testers can readily duplicate the series of actions that caused the failure by attaching these reproducible steps to the test environment. 
 
Additionally, the framework takes relevant screenshots and will attach to log. This will help in identifying UI rendering problems or any unexpected behaviors, that can be test failures. 

Our framework's focus on concise test methods enables us to increase test coverage, enhance code maintainability, and increase reusability. By keeping test methods concise, we are able eliminate unnecessary code and reduce redundancy. This allows us to focus on the essential aspects of each of the test scenarios. 

Conclusion 

In conclusion, our in-house API framework provides a comprehensive and efficient solution for automating API testing. With the power of automation, we can enhance the reliability accuracy, and scalability of our testing process. We can cover a greater range of test scenarios while preserving the modularity and reusability of our code by condensing our test methods. In the end, our API framework enables us to produce high-quality software, verify the security and functioning of our APIs, and guarantee a flawless user experience. This success story demonstrates our capability to implement DevOps Quality Engineering covering UI and API automation for a leading cloud service provider. We have further developed our framework to stay ahead of the curve and to offer enhanced API security test coverage via Azure OpenAI integration.