Frameworks
Verdict. The Swiss-army framework — huge ecosystem and integrations, with a learning curve and API churn to match
What it is. Python and JavaScript framework for building LLM apps with chains, agents, tools, retrieval and memory, from LangChain Inc.
Good.
- Largest integration ecosystem of any LLM framework
- Python and JS/TS support, with LangGraph underneath for stateful agents
- LangSmith adds tracing, evals and deployment in one place
Bad.
- API has changed often; older tutorials go stale fast
- Abstractions can obscure what is actually happening
- Production observability and deployment push you toward paid LangSmith
Best for. Teams building varied LLM apps who want the biggest ecosystem and don't mind the learning curve
Price: Free / open source (MIT); paid LangSmith platform (Developer free, Plus $39/seat/mo) · Official site · Docs · GitHub
Verdict. The control freak's agent runtime — explicit stateful graphs with durable execution, if you want to build it yourself
What it is. Low-level framework from LangChain for stateful, multi-actor LLM apps modelled as graphs of nodes and edges, with persistence and human-in-the-loop.
Good.
- Durable execution: runs survive crashes and resume
- Checkpointing, time-travel debugging and interrupts for approval
- Fine-grained control over branching, cycles and streaming
Bad.
- You build almost everything yourself — few opinions, lots of code
- Steeper learning curve than higher-level agent frameworks
- Best experience is tied to the LangChain/LangSmith stack
Best for. Teams shipping long-running, production agents that must be reliable, resumable and auditable
Price: Free / open source (MIT); managed deployment via LangSmith (Developer free, Plus $39/seat/mo) · Official site · GitHub
Verdict. The RAG specialist — the fastest route from a pile of documents to a grounded answer
What it is. Data framework for LLM apps focused on ingesting, indexing and retrieving private data, with query engines, agents and workflows.
Good.
- Best-in-class document ingestion, parsing and retrieval tooling
- Huge connector and integration library (LlamaHub)
- LlamaCloud adds managed parsing and retrieval when you outgrow DIY
Bad.
- Less suited to general agent orchestration than LangGraph or CrewAI
- Hosted LlamaCloud pricing is credit-based and can be hard to predict
- Smaller agent ecosystem than LangChain
Best for. Teams whose core problem is RAG over documents — search, Q&A and grounded answers
Price: Free / open source (MIT); hosted LlamaCloud has a free tier, Starter from $50/mo · Official site · Docs · Pricing
Verdict. The pipeline engineer's pick — declarative, auditable RAG pipelines built for production
What it is. Open-source framework from deepset for building LLM pipelines — search, RAG, agents — as explicit, inspectable component graphs, in Python.
Good.
- Declarative pipelines that are easy to audit, version and test
- Strong retrieval and search heritage
- YAML pipeline definitions make review and reuse straightforward
Bad.
- Smaller community and fewer tutorials than LangChain or LlamaIndex
- Fewer third-party integrations out of the box
- DAG mental model takes getting used to
Best for. Platform teams that need inspectable, production-grade RAG pipelines they can audit
Price: Free / open source (Apache 2.0); deepset AI Platform has a free Studio tier, Enterprise custom · Official site · Docs · GitHub
Verdict. The optimizer, not the plumber — program your prompts and let it tune them for you
What it is. Stanford-origin framework that treats LLM programs as code: declare signatures and modules, then automatically optimise prompts and weights against a metric.
Good.
- Prompt optimisation is built in, not bolted on
- Clean separation of program logic from specific prompts and models
- Strong for RAG and classification tasks with measurable quality bars
Bad.
- Different paradigm — a real learning curve if you think in chains
- No built-in observability or deployment story
- Smaller ecosystem and fewer integrations than mainstream frameworks
Best for. Builders who care about measurable output quality and want prompts tuned systematically
Price: Free / open source (MIT); no paid tier · Official site · Docs · GitHub
Verdict. Microsoft's enterprise SDK — the natural choice if your stack is .NET or Azure
What it is. Microsoft's open-source SDK for adding LLM capabilities to apps in C#, Python and Java, with plugins, planners, memory and agents.
Good.
- First-class .NET/C# support — unique among major frameworks
- Plugin model turns ordinary code into AI-callable functions
- Enterprise-friendly: Azure integration and Microsoft backing
Bad.
- Smaller community and ecosystem than LangChain
- Multi-language support isn't perfectly even across C#, Python and Java
- Agent features have lagged the Python-first frameworks
Best for. Enterprise .NET/Azure teams adding AI to existing applications
Price: Free / open source (MIT); you pay only for the underlying model APIs · Official site · Docs · GitHub
Verdict. The TypeScript default for AI in the browser — streaming UIs with almost no boilerplate
What it is. TypeScript toolkit for building AI-powered web apps: unified model calls across providers, streaming, structured output, agents and UI hooks for React/Next.js.
Good.
- One API across many model providers
- Excellent streaming and React/Next.js UI primitives
- Lightweight and easy to start — no heavy framework to learn
Bad.
- TypeScript/JavaScript only
- Thinner than LangChain for complex server-side agent orchestration
- Closest integrations assume the Vercel ecosystem
Best for. TypeScript teams building AI features into web apps, especially on Next.js
Price: Free / open source (Apache 2.0); Vercel hosting and AI Gateway billed separately · Official site · Docs · GitHub
Agent frameworks
Verdict. The TypeScript agent framework — agents, workflows and evals in one tidy SDK
What it is. TypeScript-first open-source framework for building agents, workflows, RAG and tool use, with built-in evals, observability and a local dev playground.
Good.
- TypeScript-native with type-safe tools and workflows
- Agents, workflows, memory and evals in one SDK
- Local dev server with playground UI for testing agents
Bad.
- Younger project — smaller ecosystem than LangGraph or CrewAI
- Multi-agent orchestration is lighter than the Python heavyweights
- Enterprise features live under a separate commercial licence
Best for. TypeScript/Node teams shipping agents inside web apps
Price: Free / open source (Apache 2.0 core; enterprise features under commercial licence); Mastra Cloud Starter free, Teams $250/mo · Official site · Docs · Pricing
Verdict. The org-chart framework — cast agents as roles and let the crew run the job
What it is. Python framework for multi-agent systems where agents have roles, goals and backstories and collaborate as a 'crew' on sequential or hierarchical tasks.
Good.
- Intuitive role-based mental model — fast to prototype
- Memory, guardrails and human-in-the-loop built in
- YAML config for defining crews without boilerplate
Bad.
- Sequential and hierarchical processes feel limiting for free-form agent conversations
- Less fine-grained control than LangGraph when workflows get complex
- Hosted platform's free tier is capped at 50 executions a month
Best for. Teams that want a working multi-agent prototype fast, especially research and content pipelines
Price: Free / open source (MIT); hosted CrewAI AMP free tier (50 executions/mo), Enterprise custom · Official site · Docs · Pricing
Verdict. The conversation-first framework — agents that talk to each other until the job is done
What it is. Microsoft's open-source framework for multi-agent applications built around conversational, event-driven agents that message each other, with async execution and observability.
Good.
- Conversation-driven model maps well to collaborative agent work
- Async, event-driven runtime with good observability
- Microsoft backing and active research community
Bad.
- Older tutorials target the pre-rewrite API, so docs and examples can disagree
- More moving parts than a simple sequential crew
- Conversation loops can be harder to debug and bound
Best for. Developers and researchers building conversational multi-agent systems, especially on Microsoft's stack
Price: Free / open source; no paid tier · Official site · Docs · GitHub
Gateways & routers
Verdict. The universal translator — one OpenAI-compatible API for 100+ providers, self-hosted
What it is. Open-source Python library and proxy that gives every LLM provider a single OpenAI-compatible interface, with load balancing, fallbacks, caching and spend tracking.
Good.
- One API shape for 100+ providers — swap models by changing config
- Fallbacks, retries and load balancing built in
- Spend tracking and per-key budgets for teams
Bad.
- You operate it: self-hosting means owning the infra and upgrades
- Enterprise features (SSO, RBAC, audit logs) are paid and behind a licence key
- Python-centric; very high throughput needs careful tuning
Best for. Teams that want provider freedom and cost control without vendor lock-in, and can run a proxy
Price: Free / open source (MIT core; enterprise features under commercial licence); Enterprise paid, pricing on request · Official site · Docs · Pricing
Verdict. The model marketplace — every frontier model behind one key and one bill
What it is. Hosted gateway that routes requests to hundreds of models from many providers through a single API key, with per-model pricing and automatic fallbacks.
Good.
- Instant access to hundreds of models with no per-provider setup
- Pay-as-you-go per token; free models available for trying things
- Built-in fallbacks and usage analytics
Bad.
- Some models carry a markup over the provider's direct price
- Hosted service — traffic and keys flow through a third party
- No self-hosted option for strict data-residency needs
Best for. Builders who want to try or switch models quickly without managing provider accounts
Price: Pay-as-you-go per token, rates vary by model; no subscription, some models free · Official site · Docs · Pricing
Local runners
Verdict. The easy button for local models — one command and you're chatting offline
What it is. Free tool for running open-weight models locally on macOS, Windows and Linux, with a CLI, REST API and model library.
Good.
- Simplest local setup: pull a model, run it
- OpenAI-compatible API for apps and agents
- Models stay on your machine — private by default
Bad.
- Not built for multi-user production serving or horizontal scaling
- One request per model by default; concurrent load queues
- Signed-in cloud models can send traffic off-device
Best for. Developers trying local models, prototyping offline, or giving agents a private local backend
Price: Free / open source (MIT) for local use; optional Ollama Cloud from $20/mo · Official site · GitHub · Pricing
Verdict. The friendly desktop for local models — browse, chat and serve with sliders, not terminals
What it is. Desktop app for discovering, downloading and running open models locally, with a chat UI, an OpenAI-compatible local server and MCP support.
Good.
- Best GUI for exploring models — search, chat and tweak settings visually
- One-click local OpenAI-compatible server for apps
- Runs fully offline; good Apple-silicon (MLX) support
Bad.
- Proprietary app — you can't use it as a SaaS or service-bureau backend
- Bundled runtimes can lag the newest model architectures
- Desktop-first; headless serving needs the separate daemon
Best for. People who want to try local models hands-on without the command line
Price: Free for personal and internal business use; paid Enterprise tier · Official site · Docs · Pricing
Verdict. The engine under everything — raw speed and control, with no hand-holding
What it is. Open-source C/C++ inference engine for running LLMs on CPU and GPU, the core that powers Ollama, LM Studio and many other tools; includes llama-server.
Good.
- Fastest to support new model architectures and quantisations
- Runs anywhere: CPU, CUDA, Metal, Vulkan and more
- Full control over sampling, context and performance tuning
Bad.
- Command-line first — no polished app or chat UI
- You configure everything yourself, including GGUF quantisation
- API surface moves fast; docs assume technical users
Best for. Engineers who want maximum control and early access to new models, or who embed inference in their own products
Price: Free / open source (MIT); no paid tier · Official site · Docs
Verdict. The production serving engine — built for throughput when one GPU must serve many users
What it is. High-throughput open-source LLM serving engine (PagedAttention, continuous batching) that exposes an OpenAI-compatible API for production deployments.
Good.
- Best-in-class throughput via PagedAttention and continuous batching
- OpenAI-compatible API — drop-in for existing clients
- Wide hardware support: NVIDIA, AMD, Intel and more
Bad.
- Linux-focused; Windows needs WSL
- More moving parts to tune and maintain than Ollama
- API-only — no chat UI or model library included
Best for. Teams serving open models to many users at scale on their own GPUs
Price: Free / open source (Apache 2.0); no paid tier · Official site · Docs · GitHub