Auto-Generated DRY API Test Suite - OpenAPI

Maintenance is a challenging problem to solve for an API test suite. With new parameters getting added to existing APIs in a fast-paced development environment, this problem becomes more difficult.

Usually, when negative test cases are written, all combinations of parameters are not covered. Writing all the combinations manually is difficult and gets neglected.

DRY Solution

In our previous blog (API Documentation and Validations While Keeping The Code DRY), we saw how to generate API documentation which is based on openapi.json using the pre-existing API code for parameter and response validations. Now we will re-use the already generated openapi.json file to generate API test cases. The DRY approach taken is described below:

Using the openapi.json file, for each route, we will generate a set of correct values and a set of incorrect values for each parameter.
Now we combine the correct and incorrect value sets to get to a union set for each parameter.
Then, we do a cartesian product of these union sets to get all parameter combinations. We took inspiration from the node.js code to determine the cartesian product from here.
Out of these parameter combinations, we leave out the combination with all correct parameter values, as all correct values generated in an automatic way might not be logically correct too.
For each of the rest of the parameter combinations, we hit the API server and get the response. Each of these API calls is an auto-generated test case.
The obtained response should be an error response with the incorrect parameters specified in it. If not, then the test case fails and we log the failure.

Correct Parameter Values

Values that are of respective types and follow constraints specified in the openapi.json file are considered correct values. These are auto-generated values. That’s why these can be right or wrong when considered from the application logic perspective. But for the purpose of our solution, we call them as correct.

As an example, if in the openapi.json file, the type of a parameter is an integer and the constraint is “min value: 10”, then correct values can be 11, 12, 13, …

Incorrect Parameter Values

Parameter values can be incorrect due to many possible reasons, which are listed below:

The type of value is not as per the type given in openapi.json. For example, if the type of parameter is an integer, then “abc” is an incorrect value.
Value is not as per at least one constraint. For example, if the constraint is “min value: 10”, then 9 is an incorrect value.
The value of null or undefined or missing for a mandatory parameter is incorrect.
Values of parameters that try to do SQL injections OR try to run commands on the server are also incorrect.

Node Package

To make the test suite reusable and open source, we have published it as an NPM package.

For installing the NPM package, run the following in the terminal:

npm install @truesparrow/openapi-test-suite

Following is the node snippet to run the test suite:

const openapiObj = require('path of openapi.json');
    
const ApiTestSuite = require('@truesparrow/openapi-test-suite');
const serverIndex = 0; // Index of the server (to be hit) from the servers block of openapi.json
const apiSuiteObj = new ApiTestSuite(openApiObj, serverIndex);

// To run the test suite, use following command
apiSuiteObj.runTest();

// After the test suite run is over, it is recommended to call cleanup because the openApiObj and serverIndex are in-memory cached.
apiSuiteObj.cleanup();

You can contribute to the OpenAPI test suite here.

Conclusion

As the test cases in our DRY approach are auto-generated, maintaining them is no longer a problem. Re-use of openapi.json file for auto-generation of the test cases makes the approach DRY.
We test with all possible incorrect combinations, which is difficult to achieve in a manually written test suite.