API Automation Tools
Automation tools are a different class of tool from REST clients, online clients, and proxies.
A REST client helps us build and send requests. An online client helps us quickly use or view an API in the browser. A proxy helps us inspect traffic.
Automation tools run checks for us. They can generate requests, vary data, fuzz payloads, compare responses with an OpenAPI file, and create repeatable test code or reports.
I still use REST clients and proxies for exploratory API testing. Automation tools are useful when I want a different kind of pressure on the API, or when I want a tool to find coverage gaps and contract mismatches that I might not notice manually.
Reviewed Automation Tools
EvoMaster
EvoMaster is an open source API test generation and fuzzing tool.
It can read an OpenAPI file, send many generated requests, and write out a test suite plus an HTML report. When we ran it against API Challenges it helped identify OpenAPI issues, response contract mismatches, and a few behaviour issues worth investigating.
Read the longer review:
How Automation Tools Fit Into API Testing
Automation tools are useful because they do not use an API like a human does.
They will try odd values, missing values, unexpected sequences, unusual content types, and combinations that we might not think to try during a short exploratory session.
This makes them useful for:
- fuzzing request parameters and payloads
- checking whether responses match OpenAPI
- generating repeatable tests
- finding undocumented status codes
- finding content type mismatches
- exposing weak or incomplete OpenAPI schemas
- creating evidence of what was tested
The reports need human review. A generated "fault" might be a real API bug, a documentation bug, a tool limitation, or a useful warning that the API is not described clearly enough for automation.
OpenAPI Quality
The quality of the OpenAPI file makes a big difference in the output from the Automation tools.
Automation tools work best when the OpenAPI file is explicit:
- path parameters are described on each operation
- schema properties have clear types
- error responses are documented
- response content types match what the API actually sends
- examples are representative
For API Challenges we found that EvoMaster worked better with an OpenAPI file that used operation-level path parameters and stronger schemas.
If a tool struggles to work with an OpenAPI file, that is often useful feedback that you need to expand and improve your OpenAPI file definition. The struggles can help identify documentation improvements that will also have a knock on impact for people using Swagger UI, REST clients, SDK generation, contract checks, and other automation.