The fastest way to add LLM features to a Rails app is to keep the request/response cycle synchronous and free of AI calls entirely: push every generation, embedding, or classification job onto a background queue (Sidekiq or GoodJob), stream partial results back to the browser over Turbo Streams, and cache aggressively. Rails doesn't need a rewrite to become an AI-powered system — it needs the same async discipline you already use for email and PDF generation, applied to a slower, less predictable dependency.

Why shouldn't LLM calls run in the request cycle?

A GPT-4-class completion routinely takes several seconds, and OpenAI's own documented retry behavior can add up to 300% more latency on top of that when a rate limit is hit mid-request. Put that inline in a Rails controller action and you've turned a 100ms page load into a multi-second hang that ties up a Puma worker the whole time. Under load, a handful of slow LLM calls can starve your entire web tier of workers, since Puma (and most Rails app servers) allocate a fixed thread/process pool.

The fix is the same pattern Rails developers already use for slow third-party APIs: push the call to a background job, return a "processing" state immediately, and update the UI when the result lands. This isn't AI-specific advice — it's the standard mitigation for any external dependency with unpredictable latency, and it happens to solve the two biggest LLM integration complaints (timeouts and cost spikes) at the same time.

What background job architecture actually works for LLM calls?

A production-grade pattern looks like this:

1. Enqueue, don't call inline. The controller creates a placeholder record (e.g., Message.create!(status: :pending)) and enqueues a job — GenerateResponseJob.perform_later(message.id).
2. The job calls the LLM provider directly, with its own timeout and retry configuration separate from your web-request timeouts.
3. Idempotency keys matter. If a job retries after a network blip, you don't want to pay for (and display) two completions. Store a request-scoped idempotency key and check for an existing result before calling the provider again.
4. Backoff on rate limits, not just generic errors. Providers return 429 with a Retry-After header. Sidekiq's built-in exponential backoff handles generic failures, but a rate-limit-aware job should read that header and requeue precisely, rather than guessing.
5. Separate queues by cost and priority. Route cheap classification calls (e.g., spam detection on a small model) to a default queue, and expensive generation calls to a dedicated llm queue with its own concurrency limit — so one runaway feature can't crowd out another.
6. Set a hard job timeout. Sidekiq doesn't kill long-running jobs on its own; wrap the provider call in Timeout.timeout or the client library's own request timeout, or a stuck HTTP connection can hold a worker indefinitely.

Sidekiq is the default choice here for a reason: RubyGems.org lists it at over 327 million cumulative downloads, making it the de facto standard for background processing in Rails, and its Redis-backed queues handle the retry/backoff logic described above with minimal custom code. GoodJob (Postgres-backed, no Redis dependency) is a credible alternative if you want one fewer moving part in your infrastructure and can tolerate slightly lower throughput.

Which Ruby gems should you actually use for LLM integration?

Three tiers exist:

• Provider-official and community SDKs. ruby-openai (the alexrudall/ruby-openai gem) is the most-used direct client for OpenAI's API in Ruby, with more than 44 million downloads recorded on RubyGems.org — a reasonable proxy for how many Rails apps have integrated OpenAI directly rather than through a heavier framework. Anthropic and other providers ship or maintain comparable Ruby clients.
• Provider-agnostic wrappers. ruby_llm and llm.rb give you one interface across OpenAI, Anthropic, Gemini, and others, which matters if you expect to swap models based on cost or capability — a near-certainty in a market where pricing and benchmarks shift every few months.
• RAG and embeddings tooling. langchainrb and neighbor (a Postgres/pgvector-backed nearest-neighbor gem) cover retrieval-augmented generation without leaving your existing Rails database. If your app already runs Postgres, adding the pgvector extension avoids standing up a separate vector database for a first version.

Pick the provider-agnostic wrapper unless you have a hard reason to lock into one vendor's client — model and pricing volatility is high enough right now that flexibility is worth the thin abstraction cost.

How do you stream LLM responses to a Hotwire or Turbo view?

Streaming token-by-token output to the browser is the single biggest UX difference between an app that "feels like ChatGPT" and one that feels like a slow form submit. The pattern:

1. The background job opens a streaming connection to the LLM provider (stream: true in most SDKs) and, for each token or chunk received, broadcasts a Turbo Stream update via Turbo::StreamsChannel.broadcast_append_to (or broadcast_replace_to for a full re-render of the growing text block).
2. The browser subscribes to that stream via turbo_stream_from in the view, so no custom JavaScript or WebSocket handling is required beyond what Hotwire already ships.
3. Because Action Cable (or a Redis/Postgres-backed adapter) is doing the transport, this works the same whether the LLM call takes 2 seconds or 20 — the user sees continuous progress instead of a spinner.
4. Debounce broadcasts. Sending a Turbo Stream frame for every single token can flood Action Cable under load; batching every 3-5 tokens or every ~100ms keeps the perceived streaming smooth without the message-rate overhead.

This is a meaningfully lighter-weight approach than standing up a separate Next.js or React frontend purely to get streaming UX — Hotwire already solves the "update the DOM without a full page reload" problem, and LLM token streaming is just a fast-arriving special case of that.

What do cost and latency actually look like in production?

Two things should shape your architecture decisions:

• Latency is provider-side, not something you can fully engineer away. A single GPT-4-class request commonly takes multiple seconds end-to-end even without rate-limit retries; add a rate-limit retry and that can grow substantially, per OpenAI's own documented retry behavior. Design for the slow case as the normal case, not the exception.
• Rate limits are tiered and will bind before you expect. OpenAI's lower usage tiers cap GPT-4-class models at roughly 500 requests per minute with comparatively low tokens-per-minute allowances — numbers a moderately busy Rails app can hit during a traffic spike or a batch backfill job. Build queue-level concurrency limits (Sidekiq queue-size limits, or a custom Redis-backed rate limiter) before you need them, not after a production incident.

On cost: cache aggressively at two layers. Cache embeddings by content hash so you never re-embed unchanged text, and cache full completions for prompts that repeat (FAQ-style queries, standard classification tasks) with a TTL appropriate to how often the underlying answer should change. A Rails Rails.cache read is close to free; a cached-away LLM call is the cheapest LLM call you'll ever make.

FAQ

Do I need a vector database to add RAG to a Rails app? No. If you're already running Postgres, the pgvector extension plus the neighbor gem gives you nearest-neighbor search on embeddings without adding new infrastructure. Dedicated vector databases (Pinecone, Weaviate) earn their keep at a scale most Rails apps haven't reached yet.

Should LLM calls go through Sidekiq or ActiveJob directly? Use ActiveJob as the interface and Sidekiq (or GoodJob) as the backend — that's standard Rails practice and it keeps your job code portable if you ever change queue adapters. The queue-isolation and rate-limit-aware retry logic described above sits inside the job class regardless of adapter.

How do I keep an LLM feature from blowing up my hosting bill? Cache prompts and completions where the answer doesn't need to be fresh, set hard per-user or per-endpoint request caps, and route cheaper tasks (classification, short summarization) to smaller/cheaper models while reserving frontier models for tasks that actually need them.

Can Hotwire really handle streaming as well as a React chat UI? For token-by-token text streaming into a page, yes — Turbo Streams over Action Cable covers the core case with far less client-side code. Where React-style frontends still win is highly interactive, client-state-heavy UI (drag-and-drop prompt builders, multi-pane comparison views) that goes beyond appending text.

What's the biggest mistake teams make integrating LLMs into Rails? Calling the provider synchronously inside a controller action. It works in a demo and fails in production the first time a request queues behind a slow completion, dragging down response times for unrelated pages served by the same worker pool.

Do I need a separate microservice for AI features? Usually not. A background job queue plus a provider-agnostic gem covers the vast majority of Rails AI use cases. Reach for a separate service only if you need GPU-bound custom model inference that doesn't belong in a Ruby process at all.