How to Scale a Prototype to Production
How to scale a prototype to production without a rewrite: what breaks at 100, 1,000 and 10,000 users, and the few early choices worth getting right.
AI Engineer, Viithiisys
How to scale prototype to production: what breaks at each tier
To scale a prototype to production, fix correctness first (around 100 users), then the database (around 1,000), then retries, queues and cost (around 10,000). Each tier breaks a different layer, and only a few early decisions force a rewrite.
Most app scaling problems are visible in a prototype's design if you know where to look. The decisions that force a rewrite are the tenant model, job safety and visibility, covered below. The rest can be fixed in place while the product is live. The user numbers here are rules of thumb, not measurements, and your product may hit each tier earlier or later.
At 100 users the bugs are about correctness
Early users find edge cases, not load limits. Two people edit the same record, a payment webhook arrives twice, a user with a long name breaks a layout.
Prototype architecture mistakes show up here as missing validation and missing ownership checks. If any endpoint trusts a record ID from the browser without checking who owns it, fix that before anything else. It is a data leak, and no amount of infrastructure helps.
At 1,000 users the database and slow requests show up
A query that scanned 500 rows now scans 50,000. Pages that felt instant take two seconds, and someone adds a bigger server.
A bigger server hides the problem for a month. The real fixes are indexes, removing queries inside loops, and moving slow work out of the request.
At 10,000 users retries and queues decide the outcome
Now a slow dependency causes a pile-up. Clients retry, retries add load, and a ten-minute degradation becomes an hour-long outage.
The Amazon Builders' Library describes why retries need backoff and jitter: synchronised retries turn a partial failure into a self-inflicted load spike. Most prototypes retry immediately and without limit.
Database decisions that are expensive to reverse
Schema shape, tenant identifiers and primary key choice are hard to change once real data exists. Indexes, query plans and instance size are cheap to change later. Spend your early design effort on the first group.
Put the tenant and the owner on every table now
Adding a tenant_id or account_id column to forty tables after launch means a migration, a backfill and a rewrite of every query. Doing it on day one costs a column and an index.
If you are building a multi-customer product, this is the single most valuable early decision. Our SaaS development work starts with the tenancy model for exactly this reason: row-level isolation, separate schemas and separate databases each have different costs when customer number 200 asks for their data to be exported or deleted.
Connections run out before CPU does
PostgreSQL's max_connections defaults to 100, and each connection is a separate server process. A prototype that opens a connection per request, across several app instances, reaches that limit quickly.
Put a pooler in front, such as PgBouncer or Amazon RDS Proxy on AWS. It is a configuration change, not a redesign, and it is one of the cheapest fixes on this list.
Background jobs: the thing prototypes never have
A prototype does everything inside the web request. At scale, slow work such as email, AI calls and PDFs belongs in a queue, because a slow request ties up a worker and a failed one loses it.
Move anything slower than a second out of the request
Email, file processing, third-party API calls and LLM calls are the usual candidates. The request writes a job record and returns immediately, and a worker picks it up.
The user sees a status instead of a spinner. Your web tier stays responsive when a vendor API is slow, which is the situation that otherwise takes the whole app down.
Make every job safe to run twice
Queues deliver at least once. Amazon SQS standard queues, for example, can deliver a message more than once, so a job that charges a card or sends an invoice will eventually run twice.
The fix is an idempotency key, the pattern Stripe documents for its API. Record that a job ran, check before acting, and send repeatedly failing jobs to a dead-letter queue where a person can look at them.
When to add caching, and when not to
Cache a specific read that you have measured as slow or repeated, after fixing the query behind it. Caching everything early buys stale data and a second system that can fail.
Fix the query before you cache the result
A missing index turns a 5 ms lookup into a multi-second scan. Caching that result hides the problem until the cache is cold, then the database takes the full hit at the worst moment.
Run EXPLAIN ANALYZE on your ten slowest queries first. Often an index removes the need for a cache entirely.
Where a cache earns its place
Good candidates are reads that are expensive, requested often and tolerant of staleness: a public pricing page, a permissions lookup, a rendered report that updates hourly. Poor candidates are anything a user has just edited and expects to see immediately.
Set an expiry on every key. Check what your cache does when it fills up: Redis's default maxmemory-policy is `noeviction`, which makes writes fail instead of dropping old entries.
Observability before traffic grows
You cannot fix what you cannot see. Before traffic grows, you need request latency, error rate and saturation per endpoint, plus logs that can be traced to a single user action.
Four signals cover most of the questions
Google's SRE book names four golden signals: latency, traffic, errors and saturation. A dashboard with those four per service answers most of the "is it us or is it them" questions at 2 a.m.
Track latency as percentiles, not averages. An average of 300 ms can hide a p99 of eight seconds that your biggest customer sees every day.
Add tracing before the first incident
Pass a request ID through every log line and every job. OpenTelemetry gives you a vendor-neutral way to do this for most languages.
Teams that add this after an outage spend the outage guessing. If you want a second pair of eyes on what you can and cannot currently see, our DevOps consulting work usually begins with exactly that inventory.
How infrastructure cost grows by user tier
Infrastructure cost grows in steps, not smoothly. This is a general pattern for a typical web product, not a measurement from any specific system, and it carries no figures because load shape varies between products.
- Around 100 users: the first thing to fail is usually a missing ownership check or an unhandled edge case. Add validation, access tests and error tracking. Cost stays close to flat: one small app server and a managed database.
- Around 1,000 users: slow queries and exhausted connections appear. Add indexes, a connection pooler and a first background queue. Cost steps up a little, with a larger database and one worker process.
- Around 10,000 users: retry storms, hot rows and slow third-party calls appear. Add rate limits, backoff, a read replica and targeted caching. This is the largest step, with replicas, more workers and monitoring spend.
The cost that surprises teams is usually observability and third-party usage, not compute. Per-call fees for LLM and payment APIs scale directly with users, so measure them per request from the start.
A simple habit helps: record the cost of each external call next to its latency. When a bill jumps, you can see which feature caused it and decide whether to cache, batch or limit that call.
Three early choices that prevent a rewrite
Get the data model and tenancy design, the job-handling contract, and basic visibility right at the start. Each is cheap early and painful later. Almost everything else can be improved while customers are using the product.
- Put a tenant and an owner on every table.
- Make every background job idempotent, retried with backoff and dead-lettered.
- Measure latency, errors and saturation per endpoint.
Choose the schema as if it will outlive the code
Application code can be refactored module by module. Data cannot be un-mixed after two years of customers share tables without a tenant column.
Treat job safety as a contract
Decide once that every job is idempotent, retried with backoff and dead-lettered on repeated failure. New features then inherit the rule instead of reinventing it.
Instrument before you optimise
Latency, errors and saturation per endpoint tell you which of the other fixes you need. Without them you will optimise whatever is loudest, not whatever is slowest.
Skip heavy architecture until you need it
Teams get stuck weighing Kubernetes, microservices and event sourcing when a single well-indexed Postgres database, a queue and a dashboard would carry many products well past 10,000 users.
Fewer moving parts means fewer things to fail at 2 a.m. and fewer systems for a small team to learn. Add a new component only when a measurement shows the current one is the limit, not because a larger company uses it.
Where to start
Viithiisys builds software from Mohali and Markham, Ontario. To find out which of these decisions your product is missing, start with the broken workflow assessment. It ends with a one-page list of what is likely to fail first at your next tier, and what each fix involves.
FAQ
- Do I need to rewrite my prototype before it can handle 10,000 users?
- Usually not. Most prototypes need targeted fixes: indexes, connection pooling, a job queue for slow work, and monitoring. A rewrite is justified when the data model or tenancy design is wrong, because those are the parts that are expensive to change once real data exists.
- What breaks first when a prototype gets real traffic?
- Typically the database. Slow queries without indexes and exhausted connection limits appear well before CPU becomes a problem. After that come long requests that should have been background jobs, and retries that multiply load during a partial outage.
- When should I add caching to a growing app?
- Add it when you have measured a specific slow or repeated read and an index or query fix will not solve it. Caching adds invalidation bugs and a new failure mode, so it should answer a measured problem, not an expected one.
- How do I know my architecture is ready for scale?
- Check whether you can see latency, errors and saturation per endpoint, whether every background job is safe to run twice, and whether your schema carries tenant and ownership identifiers. If all three are true, you can scale most other parts incrementally.