The AI QA Setup I Use Every Day: A PR Test Planner, a QA Executor and a Shared Memory
For months, I have used two Claude Code skills that I built to QA pull requests on a large WordPress plugin:
- The PR Test Planner reads a pull request and writes the test plan.
- The QA Executor runs that plan on a real test site and writes the report.
They work as a pair, and they are how I test every day.
A month ago, I added a third part: a shared memory. Every run of either skill reads what earlier runs learned, and it writes down what it learned itself. The whole team shares that one memory. After a month with it, the results are clearly better, so I want to share the whole setup.
This post is about the ideas, not the code. You can build the same thing for your own project.
The problem: agents that forget
Every new session starts from zero. So an agent makes last week’s mistakes again:
- It spends time on a test-site problem that an earlier run already solved.
- It marks a test as passed, but it tested the wrong page or old code.
- It misses a kind of test case that a person already caught in review.
The worst of these is the false pass: a green result that is not true. It is the one mistake that sends a bug to users. Reference docs cannot stop it, because it is not a fact about the code. It is a fact about how the testing went wrong, and that knowledge exists only in past runs.
Idea 1: Write the spec before you read the code
Before the PR Test Planner looks at the code, it writes a short list of what the fix should do. The list comes from the issue, the comments, the PR description and the screenshots, and each line says where it came from.
The order is the point. When an agent reads the code first, “what it should do” quietly becomes “what the code does”, and a wrong fix then passes its own tests.
After that, the PR Test Planner compares the list with the code. Each gap becomes a test that fails while the gap is real. A gap found this way is often worth more to the PR author than the whole test plan.
Idea 2: Plan, run, and let a second reader check each one
The work is split in two:
- The PR Test Planner writes the test plan. It never touches the site.
- The QA Executor installs the PR’s build on a test site that is used only for QA. Then it runs the plan, looks for more bugs and writes one report. It asks before it posts anything to the PR.
After each step, a second agent reviews the result. It sees only the files, not the conversation, so it cannot copy the first agent’s assumptions. It sends work back more often than I expected: in the first 16 reviewed runs, only 4 passed with no changes.
Idea 3: Give the agents a memory of lessons
The memory is a set of short lessons, in plain text. Each lesson answers the same questions:
- Where in the product does it apply?
- What step leads you into it?
- What does it look like when it happens?
- What is the real cause, and what is the fix?
- When was it last seen to be true, and how many runs did it help?
The last two questions keep the memory honest. A lesson that nobody uses can be removed later. A lesson that is too old can be checked again.
The most valuable lessons are about false passes. When the agent catches itself trusting a green result that was not true, that becomes a check that every later run makes before it trusts a result.
Idea 4: Make lessons easy to find by area and by step
More lessons help only if the agent finds the right one.
At first, each lesson said only where it applies, for example “redirects”. But an agent often searches by what it is doing at that moment, for example “choose a URL to test”. Those words were not in the lesson, so the search missed it.
So each lesson now names both: the area of the product, and the steps that lead to it. The agent searches on both. In my test, the right lesson came first every time.
Idea 5: Make recording a step, not a habit
The weakest link is simple: a lesson exists only if an agent chooses to write it. On a rushed run, nothing gets written.
So recording is its own step at the end of every run:
- Give credit to each lesson that helped.
- Write down each new trap, and each false pass that the agent caught.
- Add one line to a run log.
- Say, in the report, what was recorded. “Nothing new” is allowed, but it must be written, with a reason.
Three guards check this: a script that checks the report, the second reader, and a hook that stops the session from ending while the step is missing. Asking an agent to record is not enough. Something has to check that it did.
Idea 6: Keep the memory outside the skills
My first version kept the lessons inside the skills. Then I found that a skill update replaces the whole skill folder, so it would delete everything a teammate had learned.
Now the memory is kept apart from the skills, in its own branch of the product’s git repo. That branch has no product code. Each person keeps a copy of only that branch, in a separate folder, so it never touches their working copy.
This gives three good things for free:
- The right people already have access, because they already work on the repo.
- Git keeps the history of every lesson, so a bad lesson is easy to fix or undo.
- Nothing else changes. No build and no workflow reads that branch.
Idea 7: One memory for the whole team
This is the part that made the biggest difference. The PR Test Planner and the QA Executor read and write the same memory, so a lesson that one of them learns helps the other. Every run now does two more things by itself:
- At the start, it gets the latest memory, and the first time it sets the memory up.
- At the end, it asks the user, then shares what it learned.
A few rules make it work for many people:
- When two people add lessons at the same time, both are kept. When two people change the same line, both versions stay, and a check marks it for a person to fix.
- Every lesson says which pull request it came from. So anyone can see where it came from, and why.
- A monthly check reports problems in the memory: repeated lessons, missing fields, lessons ready to become standing rules, and runs that did not record. It reports, and a person decides.
- Do not delete too early. A lesson that nobody has used yet usually has not come up yet. It is not a bad lesson.
Now, what one person learns, the next person gets on their next run.
What changed
After a month:
- Old test-site problems no longer get solved twice. The agent finds the old fix first.
- False passes that one run caught became checks that every later run makes.
- Almost every run records something.
- When a person or a reviewer catches a mistake, the question changed. It is no longer “who missed this?” It is “which lesson would have stopped it?”
Using this for your own project
I built the PR Test Planner and the QA Executor for one plugin, but the approach works for any project:
- Spec first: write what the change should do before you read the code.
- Plan, then run on a site used only for QA, with a second reader for each.
- Record as a step, and check that it happened.
- Keep the memory outside the skills, in a place your team can already read, with history.
- Share it, so every run makes the next one better.
To adapt my skills to another plugin, you can ask Claude to do it: change the product docs, the repo and the test-site setup, and give that plugin its own memory.
Conclusion
An AI QA agent without a memory is like a new tester every day: fast and capable, but sure to repeat last week’s mistakes.
The PR Test Planner plans, the QA Executor runs, and the shared memory makes each run better than the last. The agents learn from every run, and the whole team learns with them.
the end