The question
What happens when you hand a developer a genuinely good AI tool and measure everything?
I wanted to find out. Not through anecdotes (“the AI saved me hours!”), not through YouTube demos. Through data. With real baselines, hour tracking, controlled variables. An actual experiment.
The setting was favorable: I’m a Tech Lead at Cadastra and I had a brand-new FastStore project (Herbíssimo), two solid devs who took on the challenge, and access to the history of 13 previous FastStore projects to use as a comparison. The only thing missing was the infrastructure - and that’s the part I designed.
The name came naturally. We put the team together and I thought: we’re practically the Avengers — or better yet, the AIVengers! The mission? Hunt down the Thanos who snaps his fingers and half the project budget vanishes. The superpowers? Prompts, tokens and context engineering.
What I designed
A scientific experiment doesn’t start with tools - it starts with a hypothesis and a method.
I defined three comparison baselines: the project’s commercial proposal, the historical median across the 13 previous projects, and the granular Jira estimate task by task. Any measured gain would have to be a gain against all three, simultaneously.
I chose Claude Code Max 5x as the main tool - not out of brand loyalty, but because it was the only one with native Figma MCP support and a real 200K context window, with no silent compression. That mattered because my hypothesis depended on context: the AI is only useful if it “knows” what it’s doing. Setting up that context was the core of the work.
I wrote each dev’s CLAUDE.md with the project conventions, the design system patterns, the known FastStore limitations, and direct links to the technical documentation in cadastra-docs. I defined how the Figma MCP would be used for each type of component. I built a script (ai-tokens) to track token consumption in real time - without it, any cost analysis would be guesswork. I set up a Google Sheets with Apps Script for daily logging of hours per task. And I documented the full methodological framework before starting, not after.
It ran for 17 business days. Two devs. A real project with a real client and a real deadline. I wasn’t developing - I was managing the experiment, doing code review, resolving architectural blockers, and tuning the context engineering throughout the process whenever something didn’t work as expected.
What I found
The devs finished the project in 53% of the estimated hours. It wasn’t one dev - it was both, with consistent results (50% and 54%). The AI didn’t help the better dev more; it freed up both of them in similar ways.
What surprised me most wasn’t the speed. It was what the speed made possible. With more time to spare, the devs stopped cutting corners. Cleaner code, fewer hacks, a fix rate of only 10% on commits. When you’re not rushing, you do better work.
The visual components gained the most (79-88% reduction). Structural, intermediate components (30-54%). Complex pages with proprietary VTEX logic, little (10-15%). That makes sense: the AI handles well anything with a clear pattern, and poorly anything that requires proprietary knowledge that isn’t in its context. That variation confirmed for me that the context I had prepared was the deciding factor - not the tool itself.
What didn’t work as expected
Not everything was a win. There were tasks where the AI generated wrong code and the dev lost time fixing what it had invented. There were moments when the dev was busier calibrating prompts than developing. And there was the real risk of credential exposure - the AI doesn’t tell a .env apart from a React component, and if you don’t set the right boundaries, it’ll drop a secret into a commit without blinking.
I documented five risks over the course of the experiment and how each one was mitigated. None were fatal, but all required active attention. The AI doesn’t self-correct. The work of context engineering doesn’t end when the CLAUDE.md is written; it continues throughout the entire project, adjusting what the tool knows and what it shouldn’t touch.
The variation in results by component type revealed something I suspected but now had data to confirm: the deciding factor isn’t the tool, it’s the context you feed it. The components with the biggest time reduction were exactly the ones with the most documentation in cadastra-docs: documented patterns, linked source code, reusable solutions. The ones with proprietary VTEX logic, with no accessible public documentation, gained almost nothing. The knowledge infrastructure I built before the experiment was what made the results possible. Without it, the AI would have been operating in the dark.
What it became
The experiment produced 28 technical documents that stayed in cadastra-docs - from the methodology and the experimental framework to the granular analysis by component type, the tool comparison and an expansion plan for VTEX IO. It became the standard AI-assisted development process at Cadastra, with a CLAUDE.md per project and context engineering as a formal step in the kickoff.
But what matters most to me is simpler: I wanted to know whether the question had an answer. It does.
Pilot: Herbíssimo (Dana Cosméticos) Infrastructure: Cadastra Docs