Skip to main content

​
The Problem

Building AI applications outside the chat paradigm means fighting your tools. If you’re creating a document editor with inline AI suggestions, a research tool that accumulates sources in real-time, or a code analyzer that displays results across multiple files, you’ve discovered that most AI development tools assume you’re building a chatbot. This creates friction at every layer of your application.

​
The Chat Assumption

The Vercel AI SDK is well-designed and production-ready, but it’s built around a chat interaction model. That design choice permeates everything the SDK provides: the hooks, the components, the state management, the type definitions. When you try to use it for something other than chat, you discover how deeply embedded that assumption is. Consider building a document editor that streams AI suggestions inline. Your application has a document—a string of text that the user edits. The SDK, however, expects messages: an array of user and assistant turns representing a conversation. To bridge this gap, you create synthetic messages: 
// Your application has a document
const document = "The quick brown fox...";

// The SDK expects messages
const { messages, append } = useChat();

// So you fake a conversation
const fakeUserMessage = { role: "user", content: `Improve this: ${document}` };
const response = messages.find((m) => m.role === "assistant");
const improvedDocument = response?.content;
This mismatch extends beyond the API. The SDK provides hooks optimized for rendering chat logs, but your application needs an editor panel where suggestions appear inline. Your mental model treats content as a document, but the API forces you to think in messages. You can work around these mismatches, but each workaround adds code whose only purpose is bridging the gap between your application’s structure and the chat-shaped hole the tools expect. This pattern appears across many application types: calendar apps that stream scheduling suggestions, dashboard widgets showing AI-generated analysis, email clients surfacing relevant context. None of these are conversations. All of them get contorted into chat shapes.

​
Streaming Infrastructure

Streaming is fundamental to good AI user experiences. Users expect to see results appearing in real-time rather than waiting for a complete response. Implementing streaming outside the chat paradigm requires substantial infrastructure work. Suppose you want to stream tokens into a result field on your state object. Here’s what you write on the server: 
// Server: Create an SSE stream
const encoder = new TextEncoder();
const stream = new ReadableStream({
  async start(controller) {
    for await (const chunk of llm.stream(prompt)) {
      controller.enqueue(encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`));
    }
    controller.close();
  },
});
And the corresponding client-side code to consume that stream: 
// Client: Parse the SSE stream
const response = await fetch("/api/generate");
const reader = response.body?.getReader();
const decoder = new TextDecoder();

let result = "";
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const lines = decoder.decode(value).split("\n").filter(l => l.startsWith("data: "));
  for (const line of lines) {
    result += JSON.parse(line.slice(6));
    setResult(result);
  }
}
That’s roughly thirty lines of boilerplate to stream tokens into a single text field. You’re parsing Server-Sent Events manually, handling chunk boundaries, and accumulating state on the client. You’ll spend hours debugging edge cases—TextDecoder splitting chunks mid-character, JSON parsing failing on malformed data—that have nothing to do with your application’s purpose. The situation becomes worse when you need multiple streaming fields. If your application streams codePreview, explanation, and confidence simultaneously, you either stream sequentially (tripling user wait time), duplicate your infrastructure for each field, or build your own multiplexing protocol. At that point, you’re maintaining custom infrastructure for the lifetime of your application.

​
Type Safety at the Boundary

TypeScript provides excellent type safety within your codebase, but that safety evaporates at the client-server boundary. When you call server methods from React, you’re often back to string-based APIs: 
const result = await agent.call("analyzeDocument", [document]);
//                              ^^^^^^^^^^^^^^^^   ^^^^^^^^^
//                              string literal     array of any

const analysis = result as Analysis; // Hope this matches the server
Method names are string literals, so a typo means a runtime error rather than a compile-time error. Arguments are untyped arrays, so passing the wrong type won’t be caught until the code runs in production. The return type is any, requiring an unsafe type assertion that the compiler can’t verify. This makes refactoring risky. If you rename analyzeDocument to analyze on the server, you need to search every string literal in your frontend code. If you add a required parameter, you won’t know about missing arguments until users hit those code paths. If you change the return type’s structure, the type assertion silently becomes incorrect.

​
Manual State Synchronization

Real-time AI applications need to keep server state synchronized with client state. When the server’s processing status changes, or results become available, or an error occurs, the client needs to reflect those changes immediately. Without framework support, you implement this synchronization manually: 
const [status, setStatus] = useState("idle");
const [progress, setProgress] = useState(0);
const [results, setResults] = useState([]);

useEffect(() => {
  const ws = new WebSocket(url);

  ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    setStatus(data.state.status);
    setProgress(data.state.progress);
    setResults(data.state.results);
    // Did I forget a field? Will I remember to add new ones?
  };

  ws.onerror = (e) => setError(e);
  ws.onclose = () => setStatus("disconnected");

  return () => ws.close();
}, [url]);
This code handles WebSocket lifecycle, JSON parsing, and field-by-field state mapping. None of it is type-safe. If you add a field to server state, you need to hunt through client code to find everywhere that field should be handled. The compiler won’t help you find the places you missed.

​
The Cumulative Cost

Each of these problems is solvable in isolation. You can write the SSE parser. You can create synthetic chat messages. You can manually synchronize state. But you solve all of them on every project, and the time adds up. The days spent on streaming infrastructure are days not spent on your product. The bugs debugged in state synchronization don’t make your AI smarter or your user experience better. None of this work differentiates your application from competitors. This is infrastructure that someone should have built once, correctly, so that application developers never have to think about it again.

​
FAQ

​
Doesn’t the Vercel AI SDK solve streaming?

The AI SDK handles streaming well for chat interfaces, where tokens appear in a message bubble as the assistant responds. For applications that don’t fit the chat model—streaming into arbitrary state fields, multiple simultaneous streams, or custom UI patterns—you’re back to implementing SSE parsing yourself.

​
What about LangChain or LangGraph?

LangChain and LangGraph solve orchestration: composing prompts, managing context, coordinating agent behavior. But they don’t address the full-stack concerns that AI applications face. You still need to build streaming infrastructure to send tokens from your pipeline to React components. You still need state synchronization to keep clients consistent. Those problems remain.

​
Can’t I use tRPC or GraphQL?

They help with type safety at the API boundary, which is valuable. But they don’t address the streaming model that AI applications need. You still build multiplexing for multiple simultaneous streams and accumulation logic for partial updates. tRPC and GraphQL give you typed pipes, but you still need to build what flows through them.

Next: The Solution

How Idyllic addresses these problems