Contract Tests

• Contract Tests
  • Why Contract Testing
  • Why Use Qontract
  • IMPORTANT: Using Stub As A Sample Application API
  • Contract Tests Without Examples
  • When The Application Breaks The Contract
  • Contract Tests With Examples
  • Contract Tests With Multiple Examples
  • Contract Tests With JSON
  • When The Example Breaks The Contract
  • When The Application Returns Extra Keys
  • Examples Of Conflicting Keys
  • External Suggestions
  • Examples For Other Parts Of The Request
  • Examples Without Hardcoded Values
  • Examples Of Only Some Values
  • Escaping Pipes In Examples

Read here about contract testing and where Qontract fits in.

Why Contract Testing

Qontract reads the contract spec and generates tests for each scenario in the spec. It then runs these tests on your API end point, which you also provide to Qontract. If your application is built right, it will understand the request sent to each test, and send a response back. Qontract compares the response with the contract spect, and the test passes if they are in sync. These are meant only to test the request and response formats. They do not validate the values in the responses. That is the the role of API tests, which cover many more scenarios in detail.

How then are contract tests different from API tests?

The answer is that the developer alone controls the tests completely. The developer can change the tests for legitimate reasons, without realising that there may be changes to the API format, parameters, etc. If there is any such accidental breakage, the contract tests will fail.

So what prevents the developer from updating the contract to match the API tests, thus breaking downstream consumers?

Well, the contract spec is typically stored in a central repository, which is managed and updated by multiple teams. So at the very least, developers are conscious of this, and will not change the contract without talking to all concerned. To eliminate concerns around integration breakage, make sure that all changes to the contract are back backward compatible, using the tools that Qontract has provided.

Why Use Qontract

The same contract spec that is used for contract testing is also used by the API consumers for service virtualisation. Since the consumer sets expectations on it’s stubs that match the contract, and the provider API is built to adhere to the same contract, the integration between the consumer and provider stays intact.

Additionally, the contract spec is human-readable. So contracts can be circulated around by email, chat, etc while the API design is under discussion.

IMPORTANT: Using Stub As A Sample Application API

We try to provide examples that you can run. Contract tests target a provider API. So the examples in this document need a sample API application to run against. Instead of asking you to download some such sample API code, and setup up a tool chain that you may be unfamiliar with to build and run it, we are going to take a shortcut.

For the purposes of this document, most of our contract test samples will be run against Qontract stubs as the target, and we will provide the needed stub contracts.

Contract Tests Without Examples

Here’s a contract for a simple API that takes a number and returns it’s square. The contract doesn’t validate the value, it just checks that the API takes a number and returns a number.

#filename: numbers.qontract

Feature: Numerical Operations
  Scenario: Square of a number
    When POST /square
    And request-body (number)
    Then status 200
    And response-body (number)

Run this: java -jar /path/to/qontract test numbers.qontract, and you’ll get an error because the application doesn’t exist yet.

Instead of a sample API to hit, as discussed above let’s cheat and setup a stub. The contract test won’t know the difference :-)

#filename: sample_application.qontract

Feature: Sample application
  Scenario: Takes and returns a number
    When POST /square
    And request-body (number)
    Then status 200
    And response-body (number)

And run it: java -jar /path/to/qontract stub sample_application.qontract.

In another tab, run the test command again against the “sample” application :-)

> qontract test numbers.qontract
>> Request Start At Fri Jun 19 12:35:50 IST 2020
-> POST /square
-> Accept-Charset: UTF-8
-> Accept: */*
->
-> 255
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 3
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- 779
<< Response At Fri Jun 19 12:35:51 IST 2020 ==


Scenario: Square of a number POST /square SUCCESSFUL

Tests run: 1, Failures: 0

All’s well, the “application” is in sync with the contract.

When The Application Breaks The Contract

Take the same numbers contract again.

#filename: numbers.qontract

Feature: Numerical Operations
  Scenario: Square of a number
    When POST /square
    And request-body (number)
    Then status 200
    And response-body (number)

This time let’s modify the “sample” application to return a string instead of a number.

#filename: sample_application.qontract

Feature: Sample application
  Scenario: Takes and returns a number
    When POST /square
    And request-body (number)
    Then status 200
    And response-body (string)

As before, run the stub: java -jar /path/to/qontract stub sample_application.qontract.

And in another tab, run the contract test.

> qontract test numbers.qontract
>> Request Start At Fri Jun 19 12:42:49 IST 2020
-> POST /square
-> Accept-Charset: UTF-8
-> Accept: */*
->
-> 454
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 5
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- CPRTP
<< Response At Fri Jun 19 12:42:49 IST 2020 ==


Scenario: Square of a number POST /square FAILED
Reason: Testing scenario "Square of a number"
    >> RESPONSE.BODY

    Expected number, actual was string: "CPRTP"


Tests run: 1, Failures: 1

The contract test generated a request with a random number that matched the contract, and sent it to the “sample” application. The test expected a numerical response, but got a string instead. Since the applications response did not not match the contract spec, Qontract reported the problem.

Contract Tests With Examples

Consider this contract for getting the username for a given id.

Feature: User API
  Scenario: Get User Name
    When GET /user/(id:number)
    Then status 200
    And response-body (string)

By default, the test will randomly generated the user id number in the url on the fly just before running the test. So the application won’t have a user with that id, and hence when it receives this GET request, it will not be able to return a username.

We have to help things along. The application will have to be setup with a user, and the contract test will have to be fed this pre-existing user id, so that it can generate a request that the application can respond to.

Let’s use the Examples feature for this.

We shall first setup the “sample” application, in which querying for a user with id 10 will get back the username jane_doe:

#filename: sample_application.qontract

Feature: Sample applicatioon
  Scenario: Returns username
    When GET /user/10
    Then status 200
    And response-body jane_doe

Then let’s create the contract.

#filename: user.qontract

Feature: User API
  Scenario: Get User Name
    When GET /user/(id:number)
    Then status 200
    And response-body (string)

  Examples:
  | id |
  | 10 |

Note that the id in the URL retains the type. By using types instead of hardcoded values, the contract becomes useful for the consumer as well, for the purpose of service virtualisation.

However, to avoid randomly generated values, we provide concrete examples. Qontract looks up the url parameter name “id” in the example, and it finds a column named “id” as well. It picks up the value 10.

First it checks that the example matches the contract, which it does here. The id must be a number, and the example is a number. It then formulates the request (GET /user/10) and sends it to the “sample” application.

Let’s run this. Run the “sample” using ` stub sample_application.qontract`.

In a new tab, run the test:

> qontract test user.qontract
>> Request Start At Fri Jun 19 13:14:38 IST 2020
-> GET /user/10
-> Accept-Charset: UTF-8
-> Accept: */*
->
->
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 8
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- jane_doe
<< Response At Fri Jun 19 13:14:39 IST 2020 ==


Scenario: Get User Name GET /user/(id:number) SUCCESSFUL

Tests run: 1, Failures: 0

Contract Tests With Multiple Examples

Why stop at 1 example. You can specify multiple examples, if needed.

#filename: user.qontract

Feature: User API
  Scenario: Get User Name
    When GET /user/(id:number)
    Then status 200
    And response-body (string)

  Examples:
  | id |
  | 10 |
  | 20 |
  | 30 |

Let’s setup the “sample” application to handle all the user ids.

#filename: sample_application.qontract

Feature: Sample applicatioon
  Scenario: Returns username
    When GET /user/10
    Then status 200
    And response-body jane_doe

  Scenario: Returns username
    When GET /user/20
    Then status 200
    And response-body joan_doe

  Scenario: Returns username
    When GET /user/30
    Then status 200
    And response-body john_doe

Let’s run the “sample”, as usual, using ` stub sample_application.qontract`.

Now run the tests.

> qontract test user.qontract
>> Request Start At Fri Jun 19 13:32:03 IST 2020
-> GET /user/10
-> Accept-Charset: UTF-8
-> Accept: */*
->
->
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 8
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- jane_doe
<< Response At Fri Jun 19 13:32:04 IST 2020 ==


Scenario: Get User Name GET /user/(id:number) SUCCESSFUL

>> Request Start At Fri Jun 19 13:32:04 IST 2020
-> GET /user/20
-> Accept-Charset: UTF-8
-> Accept: */*
->
->
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 8
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- joan_doe
<< Response At Fri Jun 19 13:32:04 IST 2020 ==


Scenario: Get User Name GET /user/(id:number) SUCCESSFUL

>> Request Start At Fri Jun 19 13:32:04 IST 2020
-> GET /user/30
-> Accept-Charset: UTF-8
-> Accept: */*
->
->
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 8
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- john_doe
<< Response At Fri Jun 19 13:32:04 IST 2020 ==


Scenario: Get User Name GET /user/(id:number) SUCCESSFUL

Tests run: 3, Failures: 0

Contract Tests With JSON

Let’s try a simple example with JSON.

#filename: user.qontract

Feature: User API
  Scenario: Update user
    When POST /users
    And request-body
      | id   | (number) |
      | name | (string) |
    Then status 200
    And response-body success

  Examples:
  | id  | name     |
  | 10 | Jane Doe |

And the “sample” application that accepts information about Jane Doe:

#filename: sample_application.qontract

Feature: User info
  Scenario: Info about Jane
    When POST /users
    And request-body
      | id   | (number) |
      | name | Jane Doe |
    Then status 200
    And response-body success

Run the stub using `java -jar /path/to/qontract stub sample_application.qontract.

Let’s try running user.qontract as a test.

> qontract test user.qontract
>> Request Start At Fri Jun 19 16:23:43 IST 2020
-> POST /users
-> Accept-Charset: UTF-8
-> Accept: */*
->
-> {
->     "name": "Jane Doe",
->     "id": 10
-> }
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 7
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<- success
<< Response At Fri Jun 19 16:23:43 IST 2020 ==


Scenario: Update user POST /users SUCCESSFUL

Tests run: 1, Failures: 0

When The Example Breaks The Contract

#filename: user.qontract

Feature: User API
  Scenario: Update user
    When POST /users
    And request-body
      | id   | (number) |
      | name | (string) |
    Then status 200
    And response-body success

  Examples:
  | id  | name     |
  | ten | Jane Doe |

Look closely:

• The request body json has a key id, whose value must be a number.
• The example provided for id however is “ten”, which is definitely not a number.

Let’s see what happens:

> qontract test user.qontract
In scenario "Update user"
>> REQUEST.BODY.id

Format error in example of "id"
Expected number, actual was "ten"
Tests run: 0, Failures: 0

Qontract didn’t even run the test. Instead, it flagged the error in the example.

When The Application Returns Extra Keys

The contract documents that which the consumer and provider must be in sync about, no more, no less.

Therefore if the application returns keys in a json object that the contract does not know about, it ignores them.

Here’s the “sample” application:

#filename: sample_application.qontract

Feature: User information
  Scenario: Get user info for id 10
    When GET /user/10
    Then status 200
    And response-body
      | id      | (number) |
      | name    | (string) |
      | address | (string) |

Actually, we only care about the id and name. So although the application returns an address, the contract doesn’t care, and looks like this:

#filename: user.qontract

Feature: User information
  Scenario: Get user info
    When GET /user/(id:number)
    Then status 200
    And response-body
      | id      | (number) |
      | name    | (string) |

  Examples:
  | id | name |
  | 10 | John |

Run the stub using `java -jar /path/to/qontract stub sample_application.qontract.

In a new tab, let’s try running user.qontract as a test.

> qontract test user.qontract
>> Request Start At Fri Jun 19 16:30:36 IST 2020
-> GET /user/10
-> Accept-Charset: UTF-8
-> Accept: */*
->
->
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 62
<- Content-Type: application/json
<- Connection: keep-alive
<-
<- {
<-     "id": 119,
<-     "name": "HFIHJ",
<-     "address": "SYMNQ"
<- }
<< Response At Fri Jun 19 16:30:37 IST 2020 ==


Scenario: Get user info GET /user/(id:number) SUCCESSFUL

Tests run: 1, Failures: 0

Examples Of Conflicting Keys

Supposing we have the same key in two parts of the request.

Feature: Order API
  Scenario: Add order for product to a customer
    When POST /customer/(id:number)
    And request-body
      | id  | (number) | #product id
      | qty | (number) |
    Then status 200

  Examples:
  | id | qty |
  | 10 | 10  |
# The id will be picked up by both url and request body

There are 2 id fields we have to give. One in the url, one in the body. How do we provide different examples for both?

You can specify another key in the json body instead.

Feature: Order API
  Scenario: Add order for product to a customer
    When POST /customer/(id:number)
    And request-body
      | id  | (productid:number) |
      | qty | (number)           |
    Then status 200

  Examples:
  | id | productid | qty |
  | 10 | 123       | 10  |

See how productid is used both in the value and the example.

When the value contains it’s own key, the json key is ignored.

External Suggestions

Sometimes, you need different examples in different environments. If so, the examples can’t be part of the contract. Read up on suggestions to learn how to load examples from outside the contract.

Examples For Other Parts Of The Request

So far we have seen how to use examples for json objects and url path elements.

But you can use examples for anything that has a key or a name, including headers (use the header name as the example column name), query parameters (use the query parameter name), form fields (use the form field name), multipart (use the part name).

To illustrate:

Feature: Examples
  Scenario: With query params
    When GET /data?type=(string)
    Then status 200
  Examples:
  | type      |
  | important |

  Scenario: With headers
    When GET /data
    And request-header X-Request-Id (number)
    Then status 200
  Examples:
  | X-Request-Id |
  | 10           |

  Scenario: With form field
    When POST /data
    And form-field name (string)
    Then status 200
  Examples:
  | name |
  | Jill |

  Scenario: With multipart
    When POST /data
    And multipart-formdata name (string)
    Then status 200
  Examples:
  | name |
  | Jack |

Examples Without Hardcoded Values

Sometimes you may want to provide specific values for some examples because you don’t care about the rest.

#filename: user.qontract

Feature: User information
  Scenario: Get user info
    When GET /user/(id:number)
    Then status 200
    And response-body
      | id      | (number) |
      | name    | (string) |

  Examples:
  | id       | name     |
  | 10       | (string) |
  | (number) | John     |

In the first example, id is hardcoded but the name will be autogenerated. And in the second, the id will be auto generated, while the name is hardcoded.

Examples Of Only Some Values

You can provide examples for only the keys you care about. Where there is a type but no example, Qontract will generate it.

#filename: user.qontract

Feature: User information
  Scenario: Get user info
    When POST /user/(id:number)
    And request-body
      | id      | (number) |
      | name    | (string) |
    Then status 200

  Examples:
  | id |
  | 10 |

There is an example for the id but not the name. So when generating a test, qontract will generate the name.

Let’s run this as a test against a stub of the same contract.

To run the stub: ` stub user.qontract`

And then the test.

> qontract test user.qontract
>> Request Start At Fri Jun 19 16:56:02 IST 2020
-> POST /user/10
-> Accept-Charset: UTF-8
-> Accept: */*
->
-> {
->     "name": "UYJAA",
->     "id": 10
-> }
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 0
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<-
<< Response At Fri Jun 19 16:56:03 IST 2020 ==


Scenario: Get user info POST /user/(id:number) SUCCESSFUL

Tests run: 1, Failures: 0

Escaping Pipes In Examples

If we want to put pipes in the examples, we must escape the pipes.

Feature: Data
  Scenario: Add data
    When POST /data
      And request-body (data:string)
    Then status 200
  Examples:
  | data    |
  | 1\|2\|3 |

Let’s run this contract as a test against it’s own stub.

To run the stub: ` stub user.qontract`

And then the test.

> qontract test data.qontract
>> Request Start At Fri Jun 19 16:58:41 IST 2020
-> POST /data
-> Accept-Charset: UTF-8
-> Accept: */*
->
-> 1|2|3
<- 200 OK
<- Vary: Origin
<- X-Qontract-Result: success
<- Content-Length: 0
<- Content-Type: text/plain
<- Connection: keep-alive
<-
<-
<< Response At Fri Jun 19 16:58:41 IST 2020 ==


Scenario: Add data POST /data SUCCESSFUL

Tests run: 1, Failures: 0