How Tyk leverages Tracetest for effective integration testing of OpenTelemetry

We’ve just launched Tyk API Gateway’s support for OpenTelemetry, which means that with Tyk’s native OpenTelemetry integration, developers and API platform teams gain complete visibility into their API traffic, making troubleshooting and issue resolution faster.

In our development process, our team questioned: How do we ensure our instrumentation undergoes thorough testing with every Tyk Gateway update? How do we prevent errors from reporting misleading information and impacting our users’ API monitoring?

In this blog post, I want to share how we have implemented Tracetest for effective integration testing of OpenTelemetry in Tyk’s open source Gateway.

The challenge: Accurate data reporting across diverse protocols

Tyk’s diverse proxying capabilities—encompassing HTTP, HTTPS, gRPC, GraphQL, and our proprietary Tyk internal proxy protocol—introduce complexity to our environment.

This diversity, alongside a dynamic middleware chain that adjusts based on the Tyk API configuration, presents a distinctive challenge: guaranteeing that our OpenTelemetry implementation accurately captures and reports data across these varied protocols.

The solution: Trace-based testing with Tracetest

In our search for a solution, we found our answer in Tracetest. Designed specifically for Observability-Driven Development (ODD), Tracetest became an indispensable tool in our arsenal. It allowed us to build and execute comprehensive end-to-end tests encompassing the full breadth of our API scenarios.

Integrating Tracetest into our workflow

A key aspect of our integration strategy was ensuring a seamless and consistent testing experience locally and on our Continuous Integration (CI) platform. This approach was crucial for improving the Developer Experience (DX), allowing our team to test new features efficiently and fix issues under similar conditions, irrespective of the environment.

We achieved this by crafting a comprehensive Taskfile, which included a series of commands for setting up, executing, and tearing down our e2e OpenTelemetry+Tyk+Tracetest testing environment. These commands – detailed in our Taskfile – are integral to our testing workflow and include:

• opentelemetry:setup: This command is responsible for initialising the Tracetest environment, installing the necessary CLI tools, and building a local image of the Tyk gateway. It ensures that all prerequisites for testing are met, both in local development setups and in our CI environment.
• opentelemetry:test: This command triggers the execution of all our end-to-end test scenarios using Tracetest CLI.
• opentelemetry:teardown: Post-testing, this command is used to dismantle the testing environment cleanly, ensuring that no residual configurations or data impact subsequent tests.

Our GitHub Actions pipeline, which can be viewed in detail here, is configured to trigger each pull request to the master branch, ensuring that every change proposed is thoroughly vetted for compatibility and performance before integration.

Thanks to the verbose option of Tracetest CLI, we can have a clear understanding of what’s tested and what scenarios fail on each PR, for example:

This uniformity between local and CI testing environments not only streamlines the development process but also instils confidence in our team. Developers can replicate the exact testing scenarios locally executed in CI, ensuring that any issues are caught early and reliably.

Diving into our test scenarios

Our approach to testing goes far beyond basic checks; it’s a comprehensive exploration of the varied landscape that an API gateway presents:

• Diverse API protocols: From HTTP to gRPC and GraphQL, each protocol is thoroughly tested to ensure accurate trace collection.
• Internal protocols: Special attention is given to our internal protocols, ensuring they integrate seamlessly with OpenTelemetry.
• Middleware execution: Each middleware’s execution is traced and analysed, ensuring the integrity of our API management.
• Tyk attributes: We ensure that our attributes follow the expected semantic convention and are present on each request.

This is an example of a basic test scenario for a valid Tyk API request:

```

type: Test

spec:
id: 4pnmVurVg
name: HTTP Test API - ok request
trigger:
  type: http
  httpRequest:
    method: GET
    url: 'tyk:8080/test/ip'
    headers:
      - key: Content-Type
        value: application/json
specs:
  - selector: 'span[tracetest.span.type="http" name="GET /test/ip" http.method="GET"]'
    name: Test main span attributes
    assertions:
      - 'attr:http.method         =         "GET"'
      - 'attr:http.status_code         =         200'
      - 'attr:http.user_agent         =         "Go-http-client/1.1"'
      - 'attr:http.wrote_bytes         !=         0'
      - 'attr:tracetest.span.type         =         "http"'
      - 'attr:tyk.api.id       =       "3"'
      - 'attr:tyk.api.name       =       "TestAPI"'
      - 'attr:tyk.api.orgid       =       "default"'
      - 'attr:tyk.api.tags       not-contains      "test"'
      - 'attr:tyk.api.path       =       "/test/"'
  - selector: >-
      span[tracetest.span.type="general" name="Tracetest trigger"]
      span[tracetest.span.type="general"]
    name: Check if it contains Version and Rate middlewares
    assertions:
      - 'attr:tracetest.selected_spans.count  =  2'
      - 'attr:tracetest.span.parent_id != ""'
  - selector: 'span[tracetest.span.type="general" name="RateCheckMW"]'
    name: Check for RateCheckMiddleware
    assertions:
      - 'attr:name  =  "RateCheckMW"'
  - selector: 'span[tracetest.span.type="general" name="VersionCheck"]'
    name: VersionCheck MW attributes
    assertions:
      - 'attr:tyk.api.version = "Non Versioned"'

```

You can find the whole list of tested scenarios here.

This endeavour goes beyond a one-time integration; it’s about continuous improvement and adaptation. As we add more scenarios to our tests, we refine our approach, ensuring that every step in our API management is accurately reflected in our traces.

Transparency and reliability at Tyk

In embracing OpenTelemetry for Tyk API Gateway, we needed to ensure robust integration testing. Tracetest is our solution to accurate data reporting across diverse protocols. Now seamlessly woven into our workflow, Tracetest ensures consistent, reliable testing across local and CI environments and has a permanent spot in the Tyk toolkit to make sure our future releases are rock solid.

Want to connect with the team about Tyk API Gateway’s support for OpenTelemetry? You can ask us anything here.

This blog post was initially published on tyk.io.