Making 768 database servers look like one: transparent sharding
Making 768 database servers look like one: transparent sharding
Source: https://x.com/i/status/2077435501922140450
📌 PlanetScale engineer Ben Dicken explains how a routing layer can make hundreds of sharded database servers look like a single database to applications. The post covers the path from one node to petabyte-scale fleets, and how Vitess (MySQL) and Neki (Postgres) deliver transparent sharding without rewriting app code.
📈 Why shard at all
A single Postgres or MySQL primary eventually hits hard limits on writes and storage. Read replicas help reads but not write throughput—horizontal sharding splits rows across independent servers so each holds only a subset of data.
🔀 The routing layer
Apps keep one connection string. A proxy/gate parses queries, routes by shard key, and scatter/gathers when a query spans shards—so the app never manages multi-server topology.
🛠️ MySQL: Vitess
VTGates sit in front of shards, using explicit sharding keys (vindexes). This is the battle-tested pattern behind large production MySQL fleets on PlanetScale.
🆕 Postgres: Neki
PlanetScale is building Neki from first principles for sharded Postgres—not a Vitess fork—applying lessons from operating large clusters. Available in early access.
🎯 Transparent but intentional
You still choose a sharding strategy per table; the platform hides connection management, routing, and much of the ops burden. Architecture scales from a few shards to ~768 servers / ~1 PB.
Key facts
| Fact | Value |
|---|---|
| Author | Ben Dicken (PlanetScale) |
| Published | July 15, 2026 |
| Core problem | Make ~768 servers / ~1 PB look like one cohesive DB to apps |
| MySQL solution | Vitess (VTGate proxies, explicit shard keys / vindexes) |
| Postgres solution | Neki (sharded Postgres; not a Vitess fork; early access) |
| App interface | Single connection string via load balancer + routing layer |
Details
Database sharding splits a relational database’s data across many independent servers so each holds only a subset of rows. As traffic and storage grow, a single Postgres or MySQL primary hits hard limits—especially on writes, which cannot be fixed by adding read replicas alone. PlanetScale’s post walks from a single-node database through multi-terabyte setups on a few shards up to a petabyte-scale cluster on hundreds of machines.
The practical answer is a routing layer between the app and the shards. Applications keep a single connection string; the proxy parses queries, uses a chosen shard key to pick the right servers, and can scatter/gather when a query spans shards. For MySQL that layer is Vitess (VTGates); for Postgres, PlanetScale is building Neki from first principles rather than forking Vitess.
This matters because vertical scaling eventually plateaus, and naive app-level sharding is error-prone and expensive to operate. Transparent, explicit sharding keeps schema design intentional—you still pick a sharding strategy per table—while hiding connection management, routing, and much of the operational complexity. The 768-server, ~1 PB illustration is one configuration among many; smaller systems may start with a handful of shards, but the architecture is the same pattern.