A few weeks ago, I shared that I was building a tool called SmartDoc, an LLM-powered CLI for analyzing OpenAPI specification (OAS) files. I’ve been working on this project since mid-December.

After releasing v0.4.0, I turned my attention to v0.5.0 and ran into a small problem: I didn’t know what to build next. It felt like I was reaching the end of my development journey. Most of the core functionality was largely in place: the CLI returned JSON, the metrics were defined..so what was left to work on?

Turns out, quite a bit!

Changing the categories

In 0.4.0, the CLI output results for several areas:

• Descriptions
• Description coverage
• Naming consistency
• Examples

I decided that I wanted these categories to be more granular. Now, the tool examines parameters, operations, schemas, request bodies, and responses.

Ultimately, these five categories are the fundamental building blocks of an API. For example, an operation without a description or a request body with missing fields can negatively affect the developer experience. Changing SmartDoc’s focus to be more granular means that it can examine these categories thoroughly and surface any issues.

Removing metric scores

This was one task I had planned to do in 0.5.0 anyway. The scores didn’t really have much value, and the LLM could return whatever score it deemed fit. So for 0.5.0, I decided to remove the scores and use the LLM to report counts instead.

Now, the CLI outputs the total number of elements found, the number of missing fields, and the number of missing descriptions for each category.

"operations": {
    "total": 3,
    "missing_descriptions": 2,
    "missing_fields": 3
},
"parameters": {
    "total": 2,
    "missing_descriptions": 1,
    "missing_fields": 0
},
"schemas": {
    "total": 4,
    "missing_descriptions": 4,
    "missing_fields": 0
},
"responses": {
    "total": 6,
    "missing_descriptions": 1,
    "missing_fields": 0
},
"request_bodies": {
    "total": 1,
    "missing_descriptions": 1,
    "missing_fields": 0
},

I do still question the value in including total. Users may care more about missing fields or failures, especially in a CI context. I may address this in a future release.

CI capabilities

One goal I have is to make SmartDoc CI-ready. For example, someone may want to run SmartDoc on every PR or run if the OAS file changes. So I needed to add some functionality that would work in a CI pipeline. If a condition is violated (i.e., a metric exceeds a defined threshold), the pipeline should fail.

I added a new option to the analyze function called --fail-on that can only be used for full analyses. You pass the condition that you want the CLI to evaluate using the format CATEGORY.METRIC=THRESHOLD. For example, a user may want the pipeline to fail if responses.missing_descriptions is greater than 2:

smartdoc analyze sample_oas.json --fail-on responses.missing_descriptions=2

If the value in the results exceeds the threshold defined in the condition, the fail-on is triggered and the CLI exits with a code 1.

I was able to test this out with my old Brew-Ha-Ha API project. After I added my API key to my repository secrets, I created a new workflow file and created a job that installs v0.5.0 of SmartDoc, and runs the command smartdoc analyze schema.yml --fail-on responses.missing_descriptions=2. The pipeline failed as expected:

Evaluating fail-on: responses.missing_descriptions=2
Fail-on triggered, exiting 1
Error: Process completed with exit code 1.

I may add an optional human-readable summary when a fail-on condition is triggered.

What’s next?

Another good question. I’m starting to feel like I’m going to the tail end of development for SmartDoc, at least for any new features. Now that the core functionality is in place, I might focus on usability and customization. For example, a config file could define fail-on defaults, selected categories, or the model used for analysis. But the days of new feature after new feature are behind me, I think.

And that’s not a bad thing. This entire project has been a great learning experience in thinking about the developer experience wih API documentation and the benefits and limitations of LLMs. I might decide to build a version of the CLI that uses Anthropic’s Claude AI. Who knows.

I did briefly return to the project to add a workflow file that builds the docs on every PR, checks links, and checks the docs for style using Vale. So the latest version is now 0.5.1. I might consider adding a step that tests the Python code, too.