islammdshariful
QA Engineer. I write about Playwright, automation, and AI-driven QA workflows.

How I Built a Self-Updating Knowledge Base That Keeps My AI QA Agent Always in Sync With the Codebase

An AI QA agent is only as good as the knowledge it has about your codebase. When your REST API adds new endpoints, your database schema gains new columns, or your settings structure changes, your agent’s reference docs go stale — and stale docs produce wrong test cases.

I built a system that extracts live knowledge from the codebase, diffs it against the last extraction, and surgically updates curated reference documents — all triggered by a single command. In this post, I’ll walk through the full architecture: the extraction script, the curated docs, the update command, and how they work together to keep an AI agent permanently up to date.

The Problem: AI Agent Knowledge Decay
Architecture Overview
Component 1: The Knowledge Extraction Script
Component 2: The Curated Reference Documents
Component 3: The Update Command (Orchestrator)
How It All Works Together
Step-by-Step Implementation Guide
Key Design Decisions and Why
Customizing for Your Project
Conclusion

The Problem: AI Agent Knowledge Decay

In a previous post, I showed how I built an AI-powered QA agent that generates comprehensive test plans from pull requests. The agent reads code diffs and produces white-box test cases with full prioritization.

But there’s a problem that becomes obvious after a few weeks: the agent’s domain knowledge goes stale.

Here’s what happens:

A developer adds a new REST API endpoint — the agent’s API reference doesn’t know about it
A migration adds a new database column — the agent’s schema reference is outdated
New settings are introduced — the agent doesn’t know the defaults or validation rules
New hooks and filters are registered — the agent misses cross-feature regression risks

If the agent doesn’t know about a new endpoint, it can’t generate test cases for it. If it doesn’t know about a new database column, it can’t create data integrity tests. The agent’s test plans gradually degrade from comprehensive to incomplete.

Manual updates don’t scale. Reading through every PR to figure out which docs need updating, then editing markdown tables by hand — that’s exactly the kind of tedious, error-prone work that should be automated.

I needed a system that:

Extracts fresh knowledge directly from the source code (not from PRs or changelogs)
Diffs the extraction against the previous one to detect what changed
Updates only the affected curated docs, preserving formatting and structure
Never rewrites docs that haven’t changed

Architecture Overview

The system has three layers:

┌──────────────────────────────────────────────────────────────┐
│                   /update-qa-docs                            │
│                 (Slash Command)                              │
│                                                              │
│  1. Runs extract-knowledge.sh ──► Raw extraction files       │
│  2. Diffs raw files against .prev/ snapshots                 │
│  3. For each changed file, reads the delta                   │
│  4. Surgically updates the corresponding curated doc         │
│  5. Shows summary of what changed                            │
└──────────────────────────────────────────────────────────────┘

File structure:

.claude/
├── commands/
│   └── update-qa-docs.md         # Slash command (orchestrator)
├── scripts/
│   └── extract-knowledge.sh      # Codebase knowledge extractor
└── docs/
    ├── rest-api.md               # Curated: API endpoint catalog
    ├── database-schema.md        # Curated: Table schemas & migrations
    ├── smart-tags.md             # Curated: Template variable reference
    ├── settings-map.md           # Curated: Settings structure & defaults
    ├── feature-interactions.md   # Curated: Cross-feature dependency map
    ├── qa-playbook.md            # Curated: Security payloads & test commands
    └── raw/                      # Machine-generated extractions
        ├── rest-routes.txt
        ├── db-tables.txt
        ├── smart-tags.txt
        ├── settings-defaults.txt
        ├── hooks-filters.txt
        ├── capabilities.txt
        ├── migrations.txt
        └── .prev/                # Previous extraction (for diffing)
            ├── rest-routes.txt
            ├── db-tables.txt
            └── ...

There are two distinct types of documentation:

Raw extractions (raw/*.txt) — Machine-generated, unedited grep/awk output from the codebase. These are disposable and regenerated every run.
Curated docs (docs/*.md) — Human-structured, agent-readable markdown references. These are the docs the QA agent actually uses. They have tables, explanations, cross-references, and domain context that raw grep output lacks.

The key insight: raw extractions are truth, curated docs are interpretation. The system uses the raw data to keep the curated interpretation accurate.

Component 1: The Knowledge Extraction Script

File: .claude/scripts/extract-knowledge.sh

This shell script scans your entire codebase and extracts structured knowledge into text files. Think of it as a specialized static analysis tool that outputs human/AI-readable data rather than lint warnings.

What It Extracts

The script has 7 extraction targets, each producing a separate raw file:

Target	Output File	What It Extracts
`rest-routes`	`rest-routes.txt`	Route registrations, HTTP methods, permission callbacks, API namespaces
`db-tables`	`db-tables.txt`	CREATE TABLE statements, column definitions, model field declarations, option keys, meta keys
`smart-tags`	`smart-tags.txt`	Template variable registrations, context definitions, replacement logic
`settings`	`settings-defaults.txt`	Full options files, default values, dynamic options, internal options, addon options
`capabilities`	`capabilities.txt`	Permission definitions, access control classes, route access declarations
`hooks`	`hooks-filters.txt`	Custom actions/filters, core framework hooks used, conflict prevention patterns
`migrations`	`migrations.txt`	Update/migration classes, version constants, addon update entries

Key Implementation Details

Modular extraction functions:

Each target has its own extraction function. This allows selective updates — you don’t need to re-extract everything when only one area changed.

extract_rest_routes() {
    local outfile="$OUTPUT_DIR/rest-routes.txt"
    {
        echo "# REST API Routes"
        echo "# Extracted: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
        echo ""

        echo "============================================"
        echo "ROUTE REGISTRATION CALLS"
        echo "============================================"

        for dir in "${ALL_DIRS[@]}"; do
            grep -rn "register_rest_route\|register_rest_field" "$dir/app/" \
                --include="*.php" 2>/dev/null || true
        done

        echo ""
        echo "============================================"
        echo "ROUTE DEFINITIONS"
        echo "============================================"

        for dir in "${ALL_DIRS[@]}"; do
            find "$dir/app/" -name "Api.php" -o -name "Routes.php" 2>/dev/null | while read -r file; do
                echo "--- $file ---"
                grep -n "'GET'\|'POST'\|'PUT'\|'DELETE'\|'PATCH'" "$file" 2>/dev/null || true
            done
        done

        # ... permission callbacks, namespaces
    } > "$outfile"
    write_output "$outfile"
}

Built-in diff mode for change detection:

The --diff flag enables the most important feature: comparing current extractions against previous ones.

# Save previous extraction before overwriting
if $DIFF_MODE && [ -d "$OUTPUT_DIR" ]; then
    rm -rf "$OUTPUT_DIR/.prev"
    mkdir -p "$OUTPUT_DIR/.prev"
    cp "$OUTPUT_DIR"/*.txt "$OUTPUT_DIR/.prev/" 2>/dev/null || true
fi

After extraction, it reports what changed:

write_output() {
    local file="$1"
    local old_file="$OUTPUT_DIR/.prev/$(basename "$file")"

    if $DIFF_MODE && [ -f "$old_file" ]; then
        local diff_count
        diff_count=$(diff "$old_file" "$file" 2>/dev/null | grep -c '^[<>]' || true)
        if [ "$diff_count" -gt 0 ]; then
            print_warn "$(basename "$file"): $diff_count lines changed"
        else
            print_info "$(basename "$file"): no changes"
        fi
    fi
}

Multi-directory scanning:

For monorepos or plugin architectures with addons, the script discovers all relevant directories and scans across them:

# Collect all addon/module directories
ADDON_DIRS=()
for dir in "$REPO_ROOT"/my-addon-*/; do
    if [ -d "$dir" ]; then
        ADDON_DIRS+=("$dir")
    fi
done

# All search directories
ALL_DIRS=("$CORE_DIR" "${ADDON_DIRS[@]}")

Every extraction function iterates over ALL_DIRS, so new addons are automatically picked up without changing the script.

Deep extraction, not surface-level grep:

Each extraction goes multiple layers deep. For example, the database extraction doesn’t just find CREATE TABLE — it also extracts:

Table name references across the codebase
Column additions via ALTER TABLE or migration helpers
Model field type declarations ($jsonFields, $booleanFields, etc.)
wp_options keys
Post meta, term meta, and user meta keys

extract_db_tables() {
    # CREATE TABLE statements
    grep -rn "CREATE TABLE\|dbDelta\|getTableName\|tableName" ...

    # Table name references
    grep -rn "aioseo_posts\|aioseo_terms\|..." ...

    # Column additions
    grep -rn "addColumn\|dropColumn\|ALTER TABLE" ...

    # Model field declarations
    grep -rn '\$jsonFields\|\$booleanFields\|\$numericFields' ...

    # Options keys
    grep -rn "get_option\|update_option\|add_option\|delete_option" ...

    # Meta keys
    grep -rn "_aioseo_\|'_aioseo" ...
}

Usage

# Extract everything with diff detection
./extract-knowledge.sh all --diff

# Extract only REST routes
./extract-knowledge.sh rest-routes --verbose

# Extract multiple targets
./extract-knowledge.sh db-tables settings migrations --diff

# Custom output directory
./extract-knowledge.sh all --output-dir=/tmp/extraction

Component 2: The Curated Reference Documents

The raw extractions are comprehensive but noisy — they’re grep output with line numbers, file paths, and duplicates. The curated docs transform this raw data into structured, agent-optimized references.

Here’s what makes curated docs different from raw extractions:

Aspect	Raw Extraction	Curated Doc
Format	grep output with line numbers	Markdown tables, headers, code blocks
Duplicates	Many (same pattern matched multiple times)	Deduplicated, one entry per item
Context	File paths only	Explanations, relationships, behavior notes
Cross-references	None	Links between related features
Usability	For diffing	For AI agent consumption

Document Catalog

Here’s what each curated doc covers and why the QA agent needs it:

REST API Endpoint Catalog (`rest-api.md`)

A complete catalog of every API endpoint with method, route, access level, callback, and purpose:

## System & Options

| Method | Route | Access | Callback | Purpose |
|--------|-------|--------|----------|---------|
| GET | `ping` | any | `Ping::ping` | Health check |
| GET | `options` | any | `Settings::getOptions` | Get all plugin options |
| POST | `options` | options | `Settings::saveChanges` | Save settings |

Also includes:

Architecture overview (namespaces, auth patterns, nonce handling)
Capability access level definitions
Request/response format conventions
Common blocking issues (security plugins, CDN firewalls)

Why the agent needs this: When a PR modifies an API endpoint, the agent needs to know the auth requirements, expected request format, and which other endpoints interact with the same data.

Database Schema Reference (`database-schema.md`)

Every custom table with full column definitions:

### posts_table (Core)

| Column | Type | Default | Notes |
|--------|------|---------|-------|
| id | bigint(20) unsigned | AUTO_INCREMENT | PK |
| post_id | bigint(20) unsigned | NOT NULL | FK (indexed) |
| title | text | NULL | SEO title |
| description | text | NULL | Meta description |
| keywords | mediumtext | NULL | JSON |
| seo_score | int(11) | 0 NOT NULL | 0-100 |
| robots_default | tinyint(1) | 1 NOT NULL | boolean |

Also includes:

Options table entries with purpose and storage class
Meta key definitions
Migration history table
JSON field structures

Why the agent needs this: Database PRs need data integrity test cases. The agent needs to know column types (to test boundaries), JSON fields (to test structure), and migration history (to test upgrade paths).

Template Variable Reference (`smart-tags.md`)

If your application has any kind of template variable or placeholder system, this doc catalogs them all:

## Core Tags

### Site-Level

| Tag | Description | HTML |
|-----|-------------|------|
| `#site_title` | Site title from settings | Yes |
| `#tagline` | Site tagline | No |
| `#separator` | Separator character | No |

### Content-Level

| Tag | Description |
|-----|-------------|
| `#post_title` | Original title of the current post |
| `#post_excerpt` | Post excerpt or auto-generated |
| `#post_date` | Publish date, localized |

Also includes:

Dynamic tag syntax (e.g., #custom_field-name)
Context definitions (which tags are available where)
Replacement logic and key files
Deprecated tags and their replacements

Why the agent needs this: Template variables are cross-cutting — they appear in meta titles, descriptions, schema, social tags, breadcrumbs, and RSS feeds. A bug in tag resolution affects multiple features.

Settings Structure Map (`settings-map.md`)

Complete settings tree with default values, types, and sanitization rules:

## Settings Architecture

### Options Storage

| Option Key | Purpose | Class |
|-----------|---------|-------|
| `app_options` | Main plugin settings | `Options.php` |
| `app_options_dynamic` | Per-content-type settings | `DynamicOptions.php` |
| `app_options_internal` | Internal state & tokens | `InternalOptions.php` |

### Sanitization by Type

| Type | Sanitization |
|------|-------------|
| boolean | `(bool) $value` |
| string | `sanitize_text_field($value)` |
| html | `sanitize_textarea_field($value)` |
| number | `intval($value)` |

Then the full settings tree with every key and its default value.

Why the agent needs this: Settings PRs need save/load tests, default value tests, and validation tests. The agent needs to know what type each setting is (to test boundaries) and what the defaults are (to test reset behavior).

Feature Interaction & Dependency Map (`feature-interactions.md`)

This is the most unique doc — it maps how features depend on each other:

## Meta Output (Title, Description, Keywords)

Meta Output depends on:
  ├── options.searchAppearance.global (siteTitle, metaDescription)
  ├── options.searchAppearance.archives (author, date, search)
  ├── tags.replaceTags() (template variable resolution)
  ├── Post/Term Models (per-content overrides)
  └── Filters: disable_meta, disable_title_rewrites

Affects: Frontend <head>, search engine indexing, social sharing fallbacks

## Schema Output (JSON-LD Structured Data)

Schema Output depends on:
  ├── options.searchAppearance.global.schema (organization, site representation)
  ├── tags.replaceTags() (variable resolution in schema values)
  ├── meta.description (description helper)
  ├── breadcrumbs (BreadcrumbList schema)
  ├── Local Business addon (LocalBusiness schema via filters)
  └── Filters: schema_graphs, schema_output, schema_disable

Affects: Rich results in search, Knowledge Panel, breadcrumbs in SERPs

Why the agent needs this: When a PR touches the template variable engine, the agent needs to know that meta, schema, social, breadcrumbs, and RSS are all affected. This doc tells the agent what to include in “Regression Areas.”

QA Playbook (`qa-playbook.md`)

The baseline reference loaded for every PR analysis. Contains:

Security testing payloads (XSS, SQLi, CSRF) with platform-specific variants
Verification commands (curl-based checks for meta tags, schema, sitemaps, redirects)
Test case templates by change type
Priority classification guidelines
Environment matrix and common gotchas
Integration testing matrix

This doc was covered in detail in the previous blog post.

Component 3: The Update Command (Orchestrator)

File: .claude/commands/update-qa-docs.md

This is the command that ties everything together. When you type /update-qa-docs in Claude Code, it orchestrates the full update pipeline.

Command Definition

---
description: Re-extract codebase knowledge and update curated QA docs
argument-hint: [target]
allowed-tools: Bash, Read, Write, Edit, Glob, Grep, Agent
---

The Update Flow

Step 1: Run the extraction script with diff mode

cd "$(git rev-parse --show-toplevel)"
.claude/scripts/extract-knowledge.sh all --diff

This regenerates all raw extraction files and saves the previous versions to .prev/ for comparison.

Step 2: Identify which curated docs need updating

The command uses a mapping table to know which raw files feed which curated docs:

Raw File(s)	Curated Doc	What to Update
`raw/rest-routes.txt`	`docs/rest-api.md`	New/removed/changed routes, access levels
`raw/db-tables.txt` + `raw/migrations.txt`	`docs/database-schema.md`	New tables, columns, migration entries
`raw/smart-tags.txt`	`docs/smart-tags.md`	New/removed tags, context changes
`raw/settings-defaults.txt`	`docs/settings-map.md`	New settings, changed defaults
`raw/hooks-filters.txt` + `raw/capabilities.txt`	`docs/feature-interactions.md`	New hooks, capabilities, dependency changes

If a raw file has no changes (diffing .prev/ against current shows zero delta), its corresponding curated doc is skipped entirely.

Step 3: Generate the delta for each changed area

diff .claude/docs/raw/.prev/rest-routes.txt .claude/docs/raw/rest-routes.txt | head -200

The command reads the delta output (just the added/removed lines) alongside the current curated doc.

Step 4: Surgically update the curated doc

This is where the AI shines. For each doc that needs updating, the command:

Adds new entries (routes, tables, columns, tags, settings) matching the existing format and style
Removes entries that no longer exist in the raw data
Updates entries where details changed (access levels, defaults, column types)
Preserves all existing structure, formatting, and section organization
Does NOT rewrite sections that haven’t changed

For migration history specifically, new version entries are appended at the bottom of the Migration History table using the same | version | change | format.

Step 5: Show a summary

Updated docs:
- rest-api.md: +3 routes, -1 route
- database-schema.md: +1 column (posts_table.new_col), +1 migration entry
- smart-tags.md: no changes
- settings-map.md: no changes
- feature-interactions.md: +2 hooks

Step 6: Do NOT commit

The command explicitly does not commit. It updates the files and lets you review with git diff before deciding what to keep.

Strict Formatting Rules

The command enforces several rules to maintain doc quality:

Match exact formatting style — If the existing doc uses | col1 | col2 | tables, new entries use the same format
No temporal commentary — Never adds “Added on 2026-04-03” to entries. New entries look like they were always there
Skip unchanged docs — If the delta is empty, don’t touch the curated doc
Handle missing files gracefully — If a raw file or .prev/ file is missing, skip and note in summary

Selective Updates

You can update just one area if you know what changed:

# Update only the REST API reference
/update-qa-docs rest-routes

# Update only the database schema reference
/update-qa-docs db-tables

# Update everything
/update-qa-docs

How It All Works Together

Here’s the complete flow:

Developer ships PRs with new endpoints, schema changes, settings...
        │
        ▼
Maintainer runs: /update-qa-docs
        │
        ▼
┌──────────────────────────────────┐
│  extract-knowledge.sh all --diff │
│                                  │
│  1. Saves current raw/ to .prev/ │
│  2. Scans entire codebase:      │
│     • grep for route registrations│
│     • grep for CREATE TABLE      │
│     • grep for tag definitions   │
│     • cat full options files     │
│     • grep for capability checks │
│     • grep for hooks/filters     │
│     • cat migration classes      │
│  3. Writes fresh raw/*.txt files │
│  4. Reports: "3 files changed"   │
└──────────┬───────────────────────┘
           │
           ▼
┌──────────────────────────────────┐
│  Diff raw/ against .prev/        │
│                                  │
│  rest-routes.txt: 12 lines changed│
│  db-tables.txt: 5 lines changed  │
│  smart-tags.txt: no changes      │
│  settings-defaults.txt: no changes│
│  hooks-filters.txt: 3 lines changed│
│  capabilities.txt: no changes    │
│  migrations.txt: 8 lines changed │
└──────────┬───────────────────────┘
           │
           ▼
┌──────────────────────────────────┐
│  Update curated docs (AI)        │
│                                  │
│  rest-api.md:                    │
│    + Added 2 new POST routes     │
│    + Updated access level on 1   │
│    - Removed 1 deprecated route  │
│                                  │
│  database-schema.md:             │
│    + Added new_column to posts   │
│    + Added migration entry v4.8  │
│                                  │
│  feature-interactions.md:        │
│    + Added 2 new filter hooks    │
│    + Updated dependency chain    │
│                                  │
│  smart-tags.md: SKIPPED          │
│  settings-map.md: SKIPPED        │
└──────────┬───────────────────────┘
           │
           ▼
┌──────────────────────────────────┐
│  Summary output                  │
│  Files left uncommitted for      │
│  human review via git diff       │
└──────────────────────────────────┘
           │
           ▼
Next time /qa-pr runs, the agent reads updated docs
and generates accurate test cases for new endpoints,
new columns, new hooks, etc.

Step-by-Step Implementation Guide

Prerequisites

Claude Code installed
A codebase with patterns you can grep for (routes, models, settings, etc.)

Step 1: Identify Your Knowledge Categories

Before writing any code, list the categories of knowledge your QA agent needs. Common ones:

Category	What to Extract	Why the Agent Needs It
API endpoints	Route registrations, HTTP methods, auth	API test cases: auth, validation, response format
Database schema	Table definitions, columns, types	Data integrity tests, boundary testing
Settings/config	Option keys, defaults, types	Save/load tests, default value tests
Permissions	Role checks, capability definitions	Access control tests by user role
Event system	Hooks, filters, event listeners	Cross-feature regression analysis
Migrations	Version entries, schema changes	Upgrade/downgrade test cases
Template system	Variables, placeholders, contexts	Cross-cutting feature tests

Not every project needs all of these. Start with the 2-3 most relevant to your codebase.

Step 2: Create the Directory Structure

mkdir -p .claude/scripts .claude/docs/raw/.prev .claude/commands

Step 3: Build the Extraction Script

Create .claude/scripts/extract-knowledge.sh:

The script should:

Accept target arguments — Allow extracting specific categories or all
Support diff mode — Save previous extraction to .prev/ before overwriting
Discover source directories — Auto-detect modules/addons in the repo
Extract deep, not shallow — For each category, grep multiple patterns (registrations, usages, definitions, modifications)
Report changes — Compare current vs previous extraction, print line counts

The extraction functions are project-specific. Here’s the template:

extract_YOUR_CATEGORY() {
    local outfile="$OUTPUT_DIR/your-category.txt"
    {
        echo "# Category Name"
        echo "# Extracted: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
        echo ""

        echo "============================================"
        echo "PRIMARY PATTERN (e.g., route registrations)"
        echo "============================================"

        for dir in "${ALL_DIRS[@]}"; do
            grep -rn "your_pattern" "$dir/src/" \
                --include="*.php" 2>/dev/null || true
        done

        echo ""
        echo "============================================"
        echo "SECONDARY PATTERN (e.g., permission callbacks)"
        echo "============================================"

        for dir in "${ALL_DIRS[@]}"; do
            grep -rn "another_pattern" "$dir/src/" \
                --include="*.php" 2>/dev/null || true
        done

    } > "$outfile"
    write_output "$outfile"
}

Adapt patterns for your stack:

Express/Node: grep -rn "router\.$get\|post\|put\|delete$" --include="*.ts"
Django: grep -rn "path(\|re_path(" --include="*.py" for routes; grep -rn "class.*models\.Model" --include="*.py" for models
Rails: grep -rn "get\|post\|put\|delete\|resources\|resource" config/routes.rb for routes
Go: grep -rn "HandleFunc\|Handle\|mux\.$Get\|Post$" --include="*.go" for routes
React/Vue: grep -rn "createSlice\|defineStore\|useReducer" --include="*.ts" for state management

Step 4: Create the Curated Reference Documents

For each extraction category, create a curated markdown doc in .claude/docs/. Start by running the extraction script once, then manually structure the raw output into clean tables and sections.

The curated doc should:

Use consistent markdown table format throughout
Group entries by feature area or module
Include contextual notes that raw grep can’t provide (e.g., “This endpoint requires WooCommerce to be active”)
Have a clear section hierarchy that the update command can target

Step 5: Create the Update Command

Create .claude/commands/update-qa-docs.md:

---
description: Re-extract codebase knowledge and update curated QA docs
argument-hint: [target]
allowed-tools: Bash, Read, Write, Edit, Glob, Grep, Agent
---

The command should:

Run the extraction script with --diff flag
Define the raw-to-curated mapping — which raw files feed which curated docs
For each changed raw file: a. Generate the delta: diff .prev/file.txt file.txt b. Read the delta and the current curated doc c. Instruct the AI to add/remove/update entries matching existing format
Show a summary of what was updated
Not commit — leave changes for human review

Key rules to include in the command:

Match exact formatting style of existing doc
Never add temporal commentary (“Added on date”)
Skip docs where raw data hasn’t changed
Handle missing files gracefully

Step 6: Wire It Into Your Workflow

Run /update-qa-docs on a regular cadence:

After merging a batch of PRs
Before a release cycle (to ensure QA docs are current)
When the QA agent produces test plans that seem incomplete

Key Design Decisions and Why

1. Raw + Curated: Two-Layer Architecture

The biggest design decision is separating raw extractions from curated docs.

Why not just use raw grep output directly? Because grep output is noisy, duplicated, and lacks context. A raw extraction might show register_rest_route appearing 5 times in different files — but the curated doc shows one clean table row with method, route, access level, and purpose.

Why not just maintain curated docs manually? Because manual maintenance drifts. Someone adds 3 new endpoints and forgets to update the doc. With raw extractions, the diff makes omissions obvious.

The two-layer approach gives you truthful raw data (what the code actually contains) and useful curated data (what the AI agent can actually work with).

2. Diff-Based Updates, Not Full Regeneration

The system diffs raw files against .prev/ and only updates curated docs where changes exist. Full regeneration would risk losing hand-crafted context notes, reformatting sections unnecessarily, and creating noisy git diffs.

Surgical updates mean your git diff after running the command shows exactly what’s new — making review fast and confident.

3. Extract From Source, Not From PRs

The script scans the actual codebase, not PR diffs or changelogs. This is deliberate:

PRs might not describe all changes accurately
Changelogs often omit internal refactors
The source code is the single source of truth
Running against HEAD captures cumulative changes across many PRs

4. Multi-Directory Scanning

The script auto-discovers addon/module directories with glob patterns. When someone adds a new addon, it’s automatically included in the next extraction without modifying the script.

5. No Auto-Commit

The command updates files but never commits. This forces human review of the changes, which catches:

False positives (grep matching a comment, not real code)
Context that needs manual addition (e.g., “this endpoint is deprecated”)
Formatting issues

6. Selective Targeting

You can update just one doc area: /update-qa-docs rest-routes. This is faster for focused updates and avoids unnecessary diff processing.

Customizing for Your Project

For Different Languages/Frameworks

The extraction script is the only part that needs language-specific customization. The curated docs and update command are framework-agnostic.

Python/Django:

extract_routes() {
    grep -rn "path(\|re_path(\|url(" --include="*.py" ...
}
extract_models() {
    grep -rn "class.*models\.Model" --include="*.py" -A 20 ...
}
extract_settings() {
    grep -rn "^[A-Z_]+ =" settings/*.py ...
}

TypeScript/Express:

extract_routes() {
    grep -rn "router\.\(get\|post\|put\|delete\|patch\)" --include="*.ts" ...
}
extract_models() {
    grep -rn "@Entity\|@Table\|schema\." --include="*.ts" -A 10 ...
}

Go:

extract_routes() {
    grep -rn "HandleFunc\|Handle\|\.Get(\|\.Post(" --include="*.go" ...
}
extract_models() {
    grep -rn "type.*struct" --include="*.go" -A 20 ...
}

For Smaller Projects

If you only have 1-2 knowledge categories that matter (e.g., just API endpoints and database schema), skip the others. Start minimal and add categories as the agent’s blind spots become apparent.

For Monorepos

The multi-directory scanning pattern (for dir in "${ALL_DIRS[@]}") works naturally with monorepos. Just adjust the directory discovery glob:

# Microservices monorepo
for dir in "$REPO_ROOT"/services/*/; do
    ALL_DIRS+=("$dir")
done

# Package monorepo
for dir in "$REPO_ROOT"/packages/*/; do
    ALL_DIRS+=("$dir")
done

For CI Integration

You can run the extraction script in CI to detect doc drift:

- name: Check QA docs are current
  run: |
    .claude/scripts/extract-knowledge.sh all --diff
    # Fail if any raw file has changes not reflected in curated docs

Conclusion

An AI QA agent without up-to-date knowledge is like a QA engineer who hasn’t read the latest PRs. The test plans look reasonable but miss new endpoints, new columns, new settings, and new interaction patterns.

The self-updating knowledge base solves this with a simple architecture:

An extraction script that scans source code and produces raw data files
Curated reference documents structured for AI agent consumption
A diff-based update command that surgically patches curated docs when the raw data changes

The entire system runs with one command (/update-qa-docs), takes seconds, and produces a clean git diff you can review before committing.

The raw extraction script is the only project-specific piece — and it’s just organized grep commands. The curated docs and update command work the same regardless of your language, framework, or architecture. Start with the 2-3 knowledge categories most critical to your project, build the extraction patterns, and iterate from there.

When your knowledge base stays current, your AI agent stays accurate — and that means better test plans, fewer missed edge cases, and less manual QA work for your team.

Apr 3, 2026

If you found this post helpful, consider buying me a coffee. It keeps me writing!

« How I Built an AI-Powered QA Agent That Generates Test Plans From Pull Requests Using Claude Code How to Build a Personal Blog Site with Astro and Deploy to GitHub Pages »

How I Built a Self-Updating Knowledge Base That Keeps My AI QA Agent Always in Sync With the Codebase

Table of Contents

The Problem: AI Agent Knowledge Decay

Architecture Overview

Component 1: The Knowledge Extraction Script

What It Extracts

Key Implementation Details

Usage

Component 2: The Curated Reference Documents

Document Catalog

REST API Endpoint Catalog (rest-api.md)

Database Schema Reference (database-schema.md)

Template Variable Reference (smart-tags.md)

Settings Structure Map (settings-map.md)

Feature Interaction & Dependency Map (feature-interactions.md)

QA Playbook (qa-playbook.md)

Component 3: The Update Command (Orchestrator)

Command Definition

The Update Flow

Strict Formatting Rules

Selective Updates

How It All Works Together

Step-by-Step Implementation Guide

Prerequisites

Step 1: Identify Your Knowledge Categories

Step 2: Create the Directory Structure

Step 3: Build the Extraction Script

Step 4: Create the Curated Reference Documents

Step 5: Create the Update Command

Step 6: Wire It Into Your Workflow

Key Design Decisions and Why

1. Raw + Curated: Two-Layer Architecture

2. Diff-Based Updates, Not Full Regeneration

3. Extract From Source, Not From PRs

4. Multi-Directory Scanning

5. No Auto-Commit

6. Selective Targeting

Customizing for Your Project

For Different Languages/Frameworks

For Smaller Projects

For Monorepos

For CI Integration

Conclusion

REST API Endpoint Catalog (`rest-api.md`)

Database Schema Reference (`database-schema.md`)

Template Variable Reference (`smart-tags.md`)

Settings Structure Map (`settings-map.md`)

Feature Interaction & Dependency Map (`feature-interactions.md`)

QA Playbook (`qa-playbook.md`)