Story of Building LLM Developers: Glimpsing the Future
The LLM based development process
This is the story of how I created a process to transform short product ideas into fully developed solutions using LLMs—and discovered something unexpected along the way. In a world drowning in headlines about LLMs replacing developers, I stumbled upon a different future: one where we don’t become obsolete, but instead evolve into architects of intelligent systems, creators of roles, and composers of virtual offices. But I’m getting ahead of myself.
Last year, I introduced the prompt-to-PR trend in our company—essentially vibe coding, the fast and loose approach to LLM-based development we all know. When the company decided to build a new trading platform using LLMs, we naturally started with that approach.
Our team cranked out a working POC with trading authentication, purchases, configuration, user management, market data, and pricing in just three weeks. Impressive, right?
Then came the reality check.
Our CEO looked at what we’d built and asked the killer question: “What exactly did the LLM develop?” How could we know what was in this massive codebase? Were we supposed to read all the LLM-generated code? Our traditional documentation—the kind that would work for human developers—was essentially worthless. We had specs, plans, test suites, QA processes… none of it gave us confidence in what the LLM had actually built.
That moment of reckoning led me to develop something entirely different. In May 2025, I introduced the Spec-to-PR concept.
@here For some time, I was pursuing a method to move beyond the prompt-to-pr concept toward specs-to-pr.
prompt-to-pr, or vibe coding, which we are doing now, is equivalent to the same mess we do in normal high code: “What did we implement?”, “How did we generate the code?”, “Where are the exact prompts we used?”, “What should I do if I want to apply a change across the project?” …
specs-to-pr: we define the project module by module with full specs. We let the model implement them.
There were many challenges. The model should be complicated enough to:
- Get the solution with a human-level understanding of specs.
- Read a spec again and judge what changes are requested and only apply those changes.
- Most challenging, read the specs for existing code and judge if the current implementation is enough, and not blabber a new solution every time!
I think Sonnet 3.7 is it.
Specs need to follow some standards and sections. Below example is good enough to me, and we will onboard this method for
#our_dummy_internal_channelwith the team.spec of one module:
platform.md
In this approach, we started to define the trading platform in full specs. Based on the architecture of the trading platform, we divided each service into multiple packages/modules, and for each we wrote hundreds of lines of prompts. We also created a separate process to consume these prompts and produce code.
This was another start for us. For sure, using LLMs for coding has many challenges—especially if you want to leave them with long prompts and come back expecting them to make few enough mistakes (human or LLM developers both make mistakes—the rate and magnitude vary). We started to develop primitive controls and found interesting concepts like: when you give a task to LLMs and get a result, ask them 3 times to review their own output! There are some details here, but basically this is one main concept. We also added verification steps to our process to make sure LLMs actually follow the specs. We found this should be separate from the original 3 loops and mostly with a different LLM at that point. One more major innovation in our process was about how to instruct the LLMs not to create completely new code from scratch every time we ask them to process a spec. We learned how to guide them to read what is done, compare with specs, and decide what is missing, what needs change, and even what needs removal! This, even today, is the most confusing part of our process for outsiders: “Do you create a new trading platform every time you start the process?!” Answer is no :) A proper LLM process can actually check what modifications are needed based on our latest specs. I demonstrated it by adding SSE and WebSocket support and then removing WebSocket support purely by changing the service specs and letting the process and LLM sort out how, in the new spec, there is no WebSocket requirement; how, in the code, there is a WebSocket implementation; and what it needs to do.
We demoed the next POC to our CEO, and he wisely asked: where is the product specification? None of the documents from architecture, service specifications, and module specifications we had was even remotely an easily readable one-pager about the project. He called us a few names, then sat and wrote a one-pager Trading Platform Product Description and told us to convert this into a full trading platform—backend and frontend and everything—only using LLMs!
It was an interesting moment. I was, as you know, looking for an open window, but in our building there is no such luxury. It is a modern office with plenty of glass but no actual windows. So the only other option was to find a way to get a short “product idea,” a human-written document, and somehow output a fully developed solution.
Waterfall Approach
No Human Intervention: The Pure LLM Challenge
So we started around mid-May. My instruction was no human intervention (very little), no vibe coding; everything LLMs did from that starting point should be documented. Step by step, generated by LLMs.
From mid-May, we were trying to create a process based on a small product idea that a human wrote and convert it to a fully developed platform—front end, back end, everything. And we wanted to make sure from that document to the code, the whole path was fully documented. That was crucial because the whole concept was: we needed to show exactly what we have developed, step by step.
My approach was to completely refuse any kind of human intervention, including any kind of vibe coding. So no shortcuts, no human intervention. My idea was to follow the early software development methodologies. Old fashioned, which I knew failed for a reason—a reason that might not be valid anymore. I found: okay, we used to actually start from a product idea and develop on top of it. All kinds of PRD, then domain documents, domain modeling documents, then user stories, then on top of that, architecture, then just go on and on—then API definitions, then after that frontend architecture and design, then backend service specification and design, then after that, in a kind of a different universe, we would go and write the code. Kind of a waterfall approach.
At the start, our aim was still the full trading platform, and that was a big mistake. It was extremely hard to create such an elaborate process to convert that small PRD to a fully fledged trading platform that later we estimated would be a bit less than a medium lines-of-code project. Just imagine 700–1000K lines of code. That is enormous from just 100 lines of instructions from a human.
how one phase looks like
So, the team spent a few months trying to reach that goal. It was very frustrating. Every time we wanted to test whatever we had as a process, the size of the project was so enormous that it would take us a long time to create any kind of meaningful specification—just imagine domain and PRD and API and everything, leave alone the code. So, maximum we were reaching one service or two services’ implementation in a very narrow way, and we were finding more issues, which was actually forcing us to go back, modify the process, and repeat the loop.
Eventually, we were able to create a meaningful process which could narrow down the project into smaller stages, and we had a POC for the first stage of the trading platform again, but the implementation and result were too narrow. It was not meaningful enough for any kind of exciting, impactful demo. That was demoralizing the whole team.
I knew we had something very interesting because as our process was growing, it was getting better and better in results; it was doing a better job translating a human request to a full platform in a huge magnitude. But it was crazy slow to test.
It was around June that I found my mistake. The scope was too large, so I literally went back and thought about the project. I realized I needed two major breakthroughs to make this work.
Two Major Breakthroughs
First Breakthrough: Defining Achievement Levels
I started to think: okay, a trading platform is too large, let’s start with steps and levels. So what I did was define levels of achievement that we want our LLM-based developer to reach.
levels
- L1: a simple project, single service, no authentication, no user management—any kind of solution that needs full following of the specifications and gives you a meaningful, correct result.
- L2: an LLM-based developer that can actually develop three to five services, coordinate between them—designing different services with the LLM and asking it to decide about the calls between them all—and design all of them “in one mind, one eye.” It’s a complex task. At the same time we expected L2 to handle user management, login, sign-up, some kind of admin, and be good enough that security does not find too many mistakes—just imagine a good mid-level developer.
- L3: they should be able to design even more complex platforms with seven or eight services, real-time communication like WebSocket, gRPC, or others, and services that are more or less scalable—acceptably good.
- L4: our final platform—like a trading platform that suddenly explodes in complexity, including risk management, many external limitations that the platform needs to handle—a full-fledged solution for us.
Second Breakthrough: Merging Vibe-Coding with Waterfall
Router
After defining these levels, I didn’t just stop there—I went back and completely revamped the entire process from scratch. This wasn’t a minor adjustment; it was a fundamental redesign using all the learnings from our painful months of trial and error. I brought together everything: my original experience with spec-based development, the lessons from our failed trading platform attempts, the insights about document handovers, the understanding of how different phases need to communicate with each other. This time, I approached it differently—creating something much more elaborate and sophisticated than anything we had attempted before. The process became intelligent, adaptive, and capable of understanding context in ways our previous iterations couldn’t.
The second breakthrough was merging the flexibility of vibe-coding with the structure of waterfall. This gives you both:
- The agility of vibe-coding: Jump in anywhere, make quick changes, iterate fast
- The reliability of waterfall: Full documentation, traceability, systematic progression
The revamped process introduced several key components that enable this hybrid approach:
- An orchestrator that routes requests to the right place
- Complexity assessment that automatically adapts the process to project size (L1 through L4)
- Structured phases with dedicated workers, verification steps, and three operational modes (new, update, review)
- A preference management system that remembers decisions and reduces repetitive questions
- Requirement traceability with unique IDs tracking every artifact from idea to implementation
- A modification router that brings vibe-coding flexibility into our structured process, intelligently classifying changes so bugs go straight to code while features cascade properly
This hybrid approach solved the critical challenges: you can make a quick bug fix like in vibe-coding, but it’s tracked and integrated properly. You can add a major feature that cascades through the entire waterfall, maintaining consistency. You get the speed when you need it, the structure when it matters.
Validating the Process: L1 and L2 Successes
I worked on the process until it could successfully handle L1 problems. The L1 process totaled around 5-6K lines of prompts. First test: a simple PDF to Markdown converter with Postgres storage. It worked perfectly on the first try.
Then I continued developing the process to handle L2 complexity. This required expanding the process significantly—the L2 version grew to around 12K lines of prompts. The test case was a temporary secret sharing solution with end-to-end encryption, user management, and role-based access control. Security found almost no issues, and it works.
What L2 can create on its own
It creates all the PRD, API, Architecture, ..
And code and integrations
Currently working on L3, with the added challenge of incorporating low-code/no-code capabilities due to company policy changes.
Our LLM-Driven Software Development Process
Overview
So what exactly did we build? After all those months of trial and error, we created something that actually works: an LLM-driven software development process that systematically transforms product ideas into fully implemented systems.
This is not a product—it’s a solution. We internally use Roo and Cline (both open source tools) to execute this process. The entire process is now publicly available at: https://github.com/deriv-com/specai-process
Let me walk you through how it all works.
The Orchestrator (start.md)
Everything begins with start.md - our LLM-based orchestrator that accepts all user requests and intelligently manages the entire development lifecycle.
start.md handles three distinct scenarios:
- New projects: Follows the waterfall phases step by step from requirements to implementation
- Interrupted work: Checks the current state, finds where we left off, and continues from there
- Completed projects: Accepts modifications, bug fixes, and security patches, determines which phases need to be involved, and routes accordingly
graph LR
A["User Request"] --> B["start.md<br/>Orchestrator"]
B --> C{"Request Type?"}
C -->|New Project| D["Requirements Phase"]
C -->|Modification| E["Modification Router"]
C -->|Review| F["Review Mode"]
E --> G["Classification"]
G --> H["Appropriate Phase"]The orchestrator analyzes the request context, determines the development state, routes to the appropriate phase, delegates work to specialized prompts, and monitors the cascade through dependent phases. It never executes directly - it always delegates to specialized workers.
Complexity Assessment
The process automatically adapts to project complexity using a complexity scoring system that kicks in during the Requirements phase:
## Complexity Scoring (0-10 scale)
1. **Technical Complexity (0-3)**
- 0: Single service, simple logic
- 1: Multiple components, moderate logic
- 2: Distributed services, complex logic
- 3: Microservices, advanced patterns
2. **User Complexity (0-2)**
- 0: Single user type, no auth
- 1: Multiple user types OR auth required
- 2: Complex roles/permissions AND auth
3. **Integration Complexity (0-2)**
- 0: Standalone application
- 1: 1-2 external integrations
- 2: Multiple integrations or complex APIs
4. **Regulatory/Compliance (0-2)**
- 0: No compliance requirements
- 1: Basic compliance (data privacy)
- 2: Strict compliance (financial, health)
5. **Scale/Performance (0-1)**
- 0: Small scale, standard performance
- 1: High scale or performance critical
The complexity score determines:
- PRD Template Selection:
- 0-3: Minimal template (simple apps)
- 4-7: Standard template (business applications)
- 8-10: Comprehensive template (complex systems)
- Architecture Patterns: From flat structure to full microservices
- Implementation Approach: From single files to modular architectures
- Framework Selection: From vanilla JS to enterprise frameworks
The Pipeline
After countless iterations, we settled on a consistent pattern for every phase. This makes the process predictable and manageable:
Phase Structure:
├── prompt.md # Main orchestration logic
├── guideline.md # Standards and best practices
├── verify.md # Verifies requirements are followed (no more, no less)
└── {worker}.md # Phase-specific implementation files
Each phase operates with three operational modes:
-
New Mode 🆕
- Starting fresh with no existing outputs
- Comprehensive analysis and creation
- Asks all necessary questions
-
Update Mode 🔄
- Outputs exist but requirements changed
- Gap analysis between existing and new
- Only asks about changes needed
-
Review Mode 👁️
- Review and modify existing preferences
- Understand current decisions
- Help user modify choices
graph TD
A["Product Brief"] --> B["Requirements"]
B --> C["Domain Modeling"]
C --> D["User Stories"]
D --> E["Architecture"]
E --> F["API Specifications"]
F --> G["Service Specifications"]
G --> H["Backend Development"]
F --> I["Frontend Architecture"]
I --> J["UI Specifications"]
J --> K["Frontend Development"]
H --> L["Integration"]
K --> LRequirement Traceability
Our process generates unique IDs for every artifact, creating a traceable path from requirements to implementation:
## ID Format Examples
REQ-01-001 → Requirement: User Authentication
ENT-02-003 → Entity: UserAccount
US-03-005 → User Story: Admin manages users
SVC-04-002 → Service: Authentication Service
API-05-007 → API Endpoint: POST /auth/login
TC-06-009 → Test Case: Login validation
Each ID flows through the system:
REQ-01-001 (User Authentication)
↓
ENT-02-003 (UserAccount entity in Domain)
↓
US-03-005 (User login story)
↓
SVC-04-002 (Auth service in Architecture)
↓
API-05-007 (Login endpoint in API spec)
↓
Implementation with all references preserved
This enables:
- Impact Analysis: Change REQ-01-001? Know exactly what’s affected
- Coverage Verification: Every requirement has implementation
- Debugging: Trace issues back to requirements
- Documentation: Self-documenting system
Preference Management: The Memory System
The system needs to remember user decisions without repeatedly asking the same questions. preferences.md files record user decisions, directives, and clarifications throughout the development process.
The system uses a two-tier hierarchy:
- Workspace preferences (
workspace/preferences.md) - Default settings that persist across all projects - Phase preferences (
workspace/output/{phase}/preferences.md) - Project-specific decisions for each phase
Both work the same way—they store the current state of decisions. The only difference is scope: workspace preferences apply globally across all projects, while phase preferences are specific to the current project and phase.
workspace/
├── preferences.md # Global defaults (persistent)
└── output/
├── requirements/
│ └── preferences.md # Requirements phase decisions
├── domain/
│ └── preferences.md # Domain modeling decisions
├── api/
│ └── preferences.md # API design decisions
└── ... (one for each phase)
Example preference entry:
### Technology Stack
- **Primary Language**: Go
- **Database**: PostgreSQL
- **API Style**: REST
Key principles:
- Current state only: Records what the system should be NOW, not change history
- Override hierarchy: Phase preferences always win over workspace preferences
- Resolution order: Phase preferences → Workspace preferences → Ask user
This hierarchy eliminates repetitive questions and maintains consistent state tracking across the development process.
Modification Management
The prompts/modify/ directory contains the modification management system that handles changes to an already-developed application while maintaining consistency across the entire codebase.
This system merges vibe coding flexibility with our structured process. After the first iteration, you can request any modification. The process analyzes your request and determines if any requirements or specification phases need updates. If they do, it updates them and propagates changes through all dependent phases to the final code. This maintains the current desired state of the solution - you always know what the LLMs have developed because you have updated specs for all phases.
The system records current state, not history. When you say “change authentication from JWT to OAuth”, it records “OAuth is the current authentication method”, not “changed JWT to OAuth”.
When users request changes after initial development is complete:
graph TD
A["User: Add email notifications"] --> B["Modification System"]
B --> C["Analyze & Classify Request"]
C --> D["Determine Entry Point"]
D --> E["Record as Current State Directive"]
E --> F["Return to Orchestrator"]
F --> G["Trigger Update Cascade"]
G --> H["Each Phase Updates in<br/>Modification Mode"]Different types of modifications route differently:
- Bug fixes go straight to development code
- New features cascade from requirements through the entire pipeline
- Architecture changes update from architecture phase down
This maintains consistency while allowing flexibility - you get vibe-coding speed when fixing bugs, but proper cascading updates when adding features.
Does History Shine a Light on the Future?
complexity growth showed a historical pattern
The turning point was when my “process” outgrew me.
Look at that graph—the explosion from March to August. What began at ~3k lines of prompts became 8k, then ~13k by the time it performed at an L2 engineer level. The prompts had become a codebase themselves, growing exponentially with each new capability.
My response: I built a tool that builds the process (which I haven’t published yet, but it’s coming). This was the real breakthrough. When you need tooling to manage your prompts like you need IDEs and development tools to manage your code, something fundamental has shifted. It reframed the entire work: the process is the program, the tool is the toolchain.
From there the abstraction snaps into focus: I’m not “writing apps” anymore; I’m authoring roles and composing offices that build apps. The complexity explosion forced me to think at a higher level of abstraction—and that’s when I saw the future.
My Emerging Obsessions
This journey has awakened three distinct interests that I never knew I had:
1. Tools to Write Roles I’m becoming obsessed with creating tooling to author new roles efficiently. Just for the software engineer role alone, I need tools that help me craft different levels (L1 through L4), different specializations (backend, frontend, DevOps), different regulatory contexts (finance, healthcare). The meta-tooling becomes crucial—how do you manage 100,000+ lines of role definitions without going insane?
2. The Roles Themselves There’s incredible depth in understanding and optimizing each role. What makes a great LLM-based backend engineer? How do you optimize for current LLM capabilities? How do you balance cost vs. speed vs. accuracy? Each role is a universe of optimization problems:
- Prompt engineering for reliability
- Context management for coherence
- Verification patterns for quality
- Cost optimization without sacrificing output
- Adapting to different LLM models and their quirks
Even within a single role, the complexity is staggering. When my L2 SWE needs to coordinate internally to design 3 services that call each other, the orchestration logic alone is hundreds of lines of careful instructions.
And now? I still need to teach my L2 engineer to think no-code and be a developer who can build those types of apps. This isn’t just about giving it access to browsers, MCP protocols, or fancy tooling—that’s the easy part. It requires engineering an entirely new type of thinking! How do you teach an LLM to conceptualize in visual builders, drag-and-drop paradigms, and workflow automation? How do you make it understand that sometimes the best code is no code at all? This is a fundamental cognitive shift I need to embed in the role definition itself.
3. Virtual Offices But here’s where it gets really interesting: composing these roles into functioning offices. How do you get a Product Manager role to effectively communicate requirements to an Architect role? How does the Architect’s output seamlessly flow to multiple Engineer roles? What about review cycles between QA and Development roles?
The coordination complexity explodes. If internal coordination within ONE role for 3 services is hard, imagine orchestrating an entire office where 5-10 specialized roles need to:
- Share context efficiently
- Maintain consistency across decisions
- Handle conflicting recommendations
- Escalate issues appropriately
- Learn from each other’s outputs
Is this what we’ll do next? Build the tooling layer? Perfect individual roles? Or jump straight to the orchestration challenge?
Maybe all three—they’re interconnected. The tools inform how we write roles. The roles reveal what the tools need. And the office orchestration shows us which roles are missing and how roles need to evolve to work together.
This feels like the birth of an entirely new field: LLM Systems Engineering—not just using LLMs, but architecting elaborate multi-agent systems that mirror and eventually surpass human organizational structures. We’re all pioneers here, whether we realize it or not.
The Pattern is Clear
Just like computing evolved from opcodes to compilers to operating systems to applications, LLM systems are evolving from prompts to roles to offices to solutions. Each layer abstracts the complexity below it.
We’re entering a Role Definition Era. Like early compilers and standard libraries, roles must be carefully authored, evolved, and optimized. Each mature role could comprise hundreds of thousands of lines of carefully crafted instructions. We’ll see families of roles with variants by domain (fintech, healthcare), region (EU privacy, US speed), and style (safety-first, performance-optimized).
This isn’t just about Software Engineers. The catalog will expand: Product Managers, Architects, QA Engineers, Security Reviewers, Data Scientists, even domain-specific roles like Risk Analysts or Clinical Coders. Each role a universe of optimization.
The Future is Being Written Now
The question isn’t whether this future will happen—it’s who will build it, and which layer they’ll own.
I’ve stumbled into all three layers simultaneously:
- The tools that make role creation manageable
- The roles themselves, each a deep optimization challenge
- The offices where roles collaborate to build solutions
What started as trying to code faster has become something much larger—pioneering an entirely new field where we don’t just use LLMs, we architect elaborate systems of specialized agents that work together like the best human organizations, only better.
Posted on September 16, 2025