Lately, I’ve been doing some work with OAS specs, and I’ve been spending more time learning about all things AI. It feels like there’s something new to learn about AI everyday. I’ve been itching to work on a new project, so I thought I’d build something that combines AI with two of my interests: APIs and Python.

Why I built it

A well-structured, complete OAS file can significantly enhance the developer experience with your API docs. An incomplete OAS file or an OAS file filled with vague descriptions can cause developers to be frustrated or lead to an increase in support requests.

There are several tools developers can use to check for errors, inconsistencies, or missing information in their OAS files. One example is Spectral, an OAS linter. Users can enforce quality and style rules by creating rulesets.

Since I’ve been working with OAS files lately, and learning a ton about LLMs, I thought it might be fun to build a tool that uses an LLM to analyze OAS files. The LLM evaluates the OAS file, scores the content against several key metrics, and suggest some improvements. I call it SmartDoc.

How I built it

When thinking about building the functionality for SmartDoc v0.1.0, I broke it down into two key steps: I needed to write code that loads an OAS file and sends it to the LLM for analysis.

Loading the OAS file

I used pyyaml to load and parse the OAS file. The load_file() function opens the file and uses yaml.safe_load() to parse the YAML data:

def load_file (path: str) -> dict:
    with open(path) as file:
        yaml_content = yaml.safe_load(file)
    print("OAS document loaded successfully.")
    return yaml_content

Prompting the LLM

The LLM I chose for my project is an OpenAI model accessed through their Response API. I picked the gpt-5-mini model because it’s faster and more cost-efficient.

Since I’m accessing the model through OpenAI’s Response API, I added some code to load my API key, which is stored in a .env file:

load_dotenv(dotenv_path=".env")

API_KEY = os.getenv('OPENAI_API_KEY')

This code creates the OpenAI client:

client = OpenAI() 

def analyze_spec(content: dict):

    serialized_oas = yaml.dump(content, sort_keys=False) # Converts Python object to YAML

    response = client.responses.create( # Creates a model response
        model="gpt-5-mini",
        reasoning={"effort": "low"}, 
        instructions=#Persona and rubric go here
        input=serialized_oas 
    )
    print(response.output_text) # Print the LLM's analysis

I provided the LLM a persona to use for its analysis (an OAS quality assurance specialist…might change this later). Then I provided the components the LLM must analyze, metrics, and rubric. You can view the full prompt on GitHub if you’re interested!

The code in analysis_engine.main kicks off the program:

def main():
    oas_path = "sample_oas.yml"
    content = yaml_loader.load_file(oas_path)
    llm_client.analyze_spec(content)
    
    
if __name__ == "__main__":
    main()

Here is part of the LLM’s response:

Quality metrics and grade
(Score 0–100% where 100% = complete, clear, and 
follows OAS best practices)

1) Description coverage: 30%
- Many descriptions are missing or empty 
(operations, responses, parameters, schema properties).

1) Description clarity: 40%
- Present descriptions are often too terse 
(e.g., "The ID", "Order created"). 

1) Naming consistency: 85%
- Names are consistently formatted and predictable; 
small opportunities to standardize enums and parameter reuse.

The next step was to build a CLI for SmartDoc, and I’ll explain how I did that in my next post!