It all started with a dungeons and dragons game, and ended with an innovative approach to solving one of our biggest problems as communicators.
Bridging the Insight-Action Gap: Engineering a Multi-Model AI Governance & Predictive Traffic Framework
Disclaimer: While I take pride in my design abilities, the infographics for this case study are AI-generated.
Impact at a Glance:
- Traffic Optimization: ~30% initial traffic surge across optimized content
- User Satisfaction LFT: 50% lift in customer satisfaction via early-stage Qualtrics data.
Engineering a Predictive Content Framework
How do you provide clear, learnable information to IP (intellectual property) lawyers, novice users, and students all at the same time? By blending multi-model AI workflows with time-series statistical modeling, I engineered a content framework that accurately forecasts user behavior, lifted qualitative satisfaction by 50%, and drove a 32%–40% monthly traffic surge.
Enterprise web properties often collapse under the weight of competing priorities. At the U.S. Patent and Trademark Office, our web ecosystem faced a persistent, systemic challenge:
How do you satisfy completely divergent user profiles on the exact same set of comms products?
IP lawyers have a distinct set of needs: They need accurate, legally bulletproof information that can help them submit their next IP application. They make their money off of knowing the process, having quick access to highly technical documentation, and submitting many applications rapidly. New filers, on the other hand, are small business owners, entrepreneurs, and people who have an idea. They don’t know the process, they may have never submitted an application to register a trademark before, and they need information at every step of the journey. Students, as a third group (and there are more groups), need information similar to new filers; an explanation of how the process works, what protections it provides, and some interesting tidbits to make their papers stand out from other papers.
Everyone has their own set of needs, and if you have one set of pages, with no differentiation by account type or status, you need a way to get people the right information at the right time. You have to do that for everyone, not just one group. It’s a major problem. The client, traditionally an organization driven exclusively by lawyers, has built thousands of pages of dense, legal copy; that doesn’t serve the majority of the country.
To solve the problem, I spent two years designing and iterating a self-correcting, human-in-the-loop framework that treats content optimization as a measurable data-science problem. As a bonus, it uses classical statistical analysis as a sort of checksum and heat map; areas where the numbers don’t line up get flagged for human inspection.
Infographic
The four-stage architecture of my multi-model AI governance & predictive traffic framework. Built by AI and validated by me, because that's the theme of this entire case study. I had to edit the output because some words and numbers didn't quite line up; and that's also the theme. You are responsible for your own outputs.
Benjamin Wilson
Methodology Phase 1: Designing the Validated Rubric & Multi-Model Scoring
Instead of relying on generic AI prompts, the scoring engine is based on user needs.
- Data Scrape & SME Validation: I extracted three years of customer service surveys, cross-referenced Google search trends, and conducted stakeholder interviews to identify user questions. The questions were further validated by legal subject matter experts.
- The Plain-Language Deductive Rubric: I wrote a rubric that grades content across five core axes: Semantic Relation, Completeness, Specificity, Relevance, and Plain Language. To eliminate technical drift, the engine enforces strict rule based penalties: deducting points for every unexplained acronym, initialism, or jargon term.
- Cross-Model Synthesis: To eliminate single-model drift, I use Gemini, Claude, and ChatGPT to score web pages against the target question sets, and then combine outputs. The framework tracks individual scores and median scores, while monitoring standard deviation via heat map. Very low and very high deviations indicate the models agreeing too well or not agreeing, so those are human inspection points.
Methodology Phase 2: Statistical Modeling & The Traffic Bridge
To make it mean anything, the quality score has to have an impact on behavior. The next step is matching content scores and GA data.
- Using historical web trends, I engineered a time-series forecasting model—similar to a difference-in-differences approach—blending three years of historical GA data to smooth out macro seasonality spikes.
- To validate the system’s predictive accuracy, I ran Pearson’s correlation coefficient and Spearman’s rank correlation calculations against active monthly page views. Initial scores indicated a moderate monotonic relationship; r=0.51 and ρ=0.49.
After the initial phase of measuring and identifying where content needed to be improved, we made improvements based on the user questions identified in Phase 1. These weren’t AI-written changes, but they were informed by what we found from the AI content grading. Then, I re-measured the content:
- We saw an improvement of about 20% in the scores of improved content.
- GA traffic showed a surge of 32-40% in traffic on improved content.
- I compared the relation between the scores and traffic and found that the monotonic relationship had improved, with Spearman’s rank correlation coefficient now at ρ=0.60.
This validates the method further; AI-enabled, human-written improvements bring us closer to an ideal state of AI-ready, human-friendly content that gets good SEO surfacing.
Human-in-the-loop implementation and scale
AI illustration
An AI illustration of a human and a computer getting along, generated by Firefly. Note the inaccurate number of fingers. Still, it gets the concept across.
Benjamin Wilson
The framework doesn’t replace human talent; it hyper-focuses it. No jobs lost to AI here. The moral of the story is that AI is a powerful tool, and works extremely well in a rigorously defined framework that is human-implemented. And the other moral is that innovative solutions help defeat problems that existing solutions don’t address.