skip to content
Deanna Thompson
Table of Contents

Earlier this year, I launched Beyond the Docs, a learning project where I built an ecosystem of developer tools to better understand the systems behind the APIs, CLIs, SDKs, and documentation I work with as a technical writer.

As part of the project, I built a Django REST API for Fstop, a fictional photography platform. I’ve been using the project to experiment with GitHub Actions, CI/CD, and more recently, AI-assisted workflows.

I wanted to incorporate AI into the project, but in a meaningful way. Ideally, I wanted to use it to solve a problem I actually encounter as a technical writer.

The problem

When an OpenAPI specification (OAS) changes, I usually need to figure out what changed before I can determine what documentation needs to be updated.

I’d want to know:

  • What endpoints changed?
  • Did request or response fields change?
  • Was anything removed?
  • Is anything potentially breaking?
  • What documentation should I review?

Sometimes, I have to wait on the developer to give me that information.

And sometimes developers are busy.

I started wondering if I could use AI to help surface that information for me. Rather than waiting on a developer to explain the changes, could I build an agent that could tell me what changed in the OAS?

Enter: the API Change Review workflow

I decided to build an agentic workflow that runs when an OAS file changes in a pull request.

The workflow prepares two inputs:

  1. The OAS from the pull request
  2. The current OAS from main

The agent then analyzes those changes and produces a writer-facing summary covering:

  • New, modified, and removed endpoints
  • Meaningful request or response changes
  • Potentially breaking changes
  • Documentation areas that may need review

The goal of the workflow isn’t to have the agent write the API documentation; it’s to give the writer a useful starting point.

Keeping the workflow deterministic where possible

One design choice I made was to keep the inputs as deterministic as possible.

Rather than asking the agent to figure out which OAS files to use, the workflow explicitly defines where to retrieve each file and prepares them before passing them to the agent. The agent then focuses on interpreting the changes.

So the workflow itself is a healthy blend of deterministic and non-deterministic steps:

Deterministic: explicitly defining where to get the OAS files
Non-deterministic: letting the agent interpret the differences between the files

Building and refining the workflow

I used GitHub Copilot’s Agentic Workflows to generate the initial workflow from my design, then tested and refined it.

Getting the workflow to run successfully was its own debugging exercise. I worked through GitHub Actions logs, workflow configuration, permissions, and model configuration before getting the workflow running successfully.

Once I got the workflow running, I had to refine the output. The first version was a little too technical and provided implementation-level details that weren’t necessary for the workflow. As a technical writer, I may need some of that information eventually, but the goal here was to get the agent to provide a high-level summary of the changes and identify potential documentation impact.

I really wanted a technical writer to be able to look at the agent’s comment and immediately get answers to three questions:

  1. What changed?
  2. Do the changes affect the documentation?
  3. What docs should be updated?

So I refined the prompt to focus on meaningful changes and avoid implementation-level details.

The output was much closer to what I wanted:

API change summary

The OpenAPI contract shows 0 new, 5 modified, and 0 removed endpoints. The changes are limited to the gallery resource schema: the created_at field has been removed from the documented gallery representation used across gallery operations.

Potentially breaking changes

The created_at response field was removed from the Gallery schema. This may affect API consumers that expect the field to be present.

Documentation review

  • Gallery endpoint reference pages
  • Gallery response examples
  • Gallery field/model reference content
  • Release notes or migration guidance

What’s important here is that the agent isn’t making the documentation decision. It’s giving me enough context to decide what I need to investigate.

What I learned

One thing I learned is how much audience, scope, and constraints matter when building agents. My first prompt produced something that was technically accurate but wasn’t very helpful to its audience. I had to be more explicit about who the output was for, what information was meaningful, how much detail I wanted, and what I didn’t want the agent to do.

I also came away with a better understanding of agentic workflows: GitHub Actions, YAML configuration, permissions, safe outputs, model configuration, and debugging.

Technical writers who work in docs-as-code environments may find agentic workflows useful for other documentation tasks, too. For example, I could see them being used to detect drift between code and documentation or surface accessibility issues on a docs site.

I plan to think of some other cool agentic workflows I can use with Beyond the Docs soon!