Building AI Products in a World of Rapidly Improving Models

Assume the model you use today is the worst you will ever use.

Every two months computers can do something they have never done before, forcing constant re-evaluation of what is possible.

Why: The exponential pace of model improvement means capabilities that are marginal today will soon be excellent, so products should be designed for the next model, not the current one.

"The AI models that you're using today is the worst AI model you will ever use for the rest of your life."

Build on the bleeding edge of current capabilities.

If your product barely works with today's model, keep going—future models will make it sing.

Why: Being slightly ahead of the curve positions you to ride the next capability jump rather than being disrupted by it.

"If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right, give it another couple months and the models are going to be great."

Writing evals is becoming a core product skill.

Evals are like unit tests for models—benchmarks that measure how well a model performs on specific tasks.

Why: Product decisions depend on knowing whether a model gets something right 60%, 95%, or 99.5% of the time, which determines the entire product design.

"Writing evals is going to become a core skill for product managers."

Stay PM-light and empower high-agency engineers.

OpenAI runs ~25 PMs for the whole company; each PM covers more engineers than typical to avoid micromanagement.

Why: Ambiguous, fast-moving problems require teams that can make decisions without waiting for permission; too many PMs create friction.

"My personal belief is that you want to be pretty PM light as an organization... too many PMs causes problems. You know, we'll fill the world with decks and ideas versus execution."

Plans are useless, planning is essential.

Quarterly roadmaps are written but expected to be thrown out halfway as new models arrive.

Why: The technology underneath changes so fast that rigid long-term plans become obsolete; the act of planning aligns dependencies and direction even if the specifics change.

"I think it's like an Eisenhower quote, plans are useless, planning is helpful, which I totally subscribe to, especially in this world."

Fine-tune and ensemble models for specific use cases.

Instead of one generic model, break problems into sub-tasks and use specialized, fine-tuned models plus ensembles to solve each part.

Why: Tailoring models to narrow domains with company-specific data yields far higher accuracy and lower cost than relying on general-purpose models.

"You're going to want sort of quasi-researcher, machine learning engineer types as part of pretty much every team, because fine-tuning a model is just going to be part of the core workflow for building most products."

Design AI interactions like you would human interactions.

When a model needs to think, give human-like updates (“hmm, let me see…”) instead of silence or full chain-of-thought dumps.

Why: Humans communicate with nuance; mirroring that in AI interfaces feels natural and sets appropriate expectations for wait times.

"You can often reason about it the way you would reason about another human. And it kind of works."

Build in verticals and with proprietary data.

OpenAI will never cover every industry-specific use case; startups win by fine-tuning on private, domain-specific data.

Why: Foundation models lack access to most of the world’s non-public data; whoever owns that data and fine-tunes on it creates defensible value.

"There are way more smart people outside your walls than there are inside your walls... there are immense opportunities in every industry and every vertical in the world to go build AI-based products."