Durgesh Tiwari
Author
Building an AI agent requires more than a Large Language Model (LLM). An LLM can understand language and generate responses, but real-world agents also need technologies to access data, use tools, maintain context, and complete multi-step tasks.
For example, an AI agent may use:
APIs and tools to interact with external systems.
Vector databases to retrieve relevant information from documents.
Memory systems to maintain useful context.
Agent frameworks to coordinate workflows, tools, and agent behavior.
Together, these technologies help developers build AI assistants, research agents, coding assistants, customer-support systems, and business automation applications.
In this chapter, we will explore the main technologies behind Agentic AI and the role of popular AI agent frameworks in building modern agent-based applications.
An AI agent combines multiple technologies to understand requests, access information, make decisions, use tools, and complete tasks.
The LLM is a core component, but depending on the use case, an agent may also need:
Large Language Models (LLMs) — Understand instructions and generate responses.
APIs and Tool Integration — Allow agents to interact with external applications and services.
Vector Databases — Store and retrieve embeddings for semantic search and RAG-based applications.
Knowledge Graphs — Represent entities and relationships as structured knowledge.
Memory Systems — Maintain relevant information across interactions or tasks.
Planning and Reasoning — Help agents break complex goals into steps and decide what action to take next.
AI Agent Frameworks — Provide building blocks for tools, workflows, state, orchestration, and agent execution.
These technologies can be combined based on the application's requirements. Not every AI agent needs every technology; a good agent architecture uses only the components required for its tasks.
A Large Language Model (LLM) can understand instructions and generate responses, but it cannot perform every real-world action by itself. An AI agent uses APIs and tools to interact with external systems, retrieve information, and take actions.
An API (Application Programming Interface) provides a way for the agent to communicate with another application or service.
For example, an AI agent may use APIs to:
Check current weather
Search the web
Send emails
Schedule meetings
Process payments
Access business applications
Retrieve customer information
Agents can also work with tools such as databases, file systems, search services, calculators, code execution environments, and cloud storage.
Suppose you ask:
"Email today's sales report to my manager."
The agent could perform the following steps:
Retrieve the sales data from a database.
Generate a short summary of the report.
Use an email tool or API to send it.
Return the result of the operation to the user.
Here, the LLM helps understand the request and decide which actions are needed, while the tools actually perform those actions.
This ability to select and use external tools is commonly called tool use or tool calling, and it is a key capability of modern AI agents.
An AI agent may need information that is not available in the LLM's current context. One way to retrieve relevant information from a large knowledge base is through vector search.
Content such as documents can be converted into embeddings—numerical representations that capture useful semantic information. These embeddings can then be stored and searched using a vector database or another system that supports vector search.
A simplified flow looks like this:
Documents
↓
Create Embeddings
↓
Store in Vector Database
User Question
↓
Create Query Embedding
↓
Similarity Search
↓
Retrieve Relevant Content
↓
Provide Context to LLM
↓
Generate ResponseSuppose a company has thousands of internal documents.
When an employee asks:
"What is our refund policy for annual subscriptions?"
The system can convert the question into an embedding and search for semantically similar document chunks. This means relevant content can be found even when the document does not use exactly the same words as the user's question.
Popular technologies used for vector search include Pinecone, Weaviate, Milvus, Qdrant, and Chroma.
Vector search is commonly used in Retrieval-Augmented Generation (RAG) systems. The retrieved information is added to the LLM's context before generation, helping the agent answer using relevant external knowledge rather than relying only on the model's built-in knowledge.
A Knowledge Graph organizes information as entities and relationships. Instead of treating information as isolated records, it connects related entities so an AI agent can understand how they are related.
For example:
Aman ──works_at──► ABC Company
│
develops
▼
Product X
│
belongs_to
▼
CategoryHere, Aman, ABC Company, Product X, and Category are entities, while works_at, develops, and belongs_to represent relationships between them.
Knowledge Graphs can help AI agents:
Understand relationships between entities
Retrieve connected information
Combine information from multiple sources
Answer questions that depend on relationships and context
They are commonly useful in areas such as:
Enterprise search
Recommendation systems
Healthcare
Financial analysis
Customer support
Business intelligence
When combined with Large Language Models (LLMs), a knowledge graph can provide structured and connected information that the model can use while answering questions. This can be especially useful when an agent needs to reason over entities, relationships, and domain-specific knowledge rather than relying only on unstructured text.
Building an AI agent from scratch requires developers to manage several components, such as LLMs, prompts, tools, state, workflows, and external integrations.
AI agent frameworks simplify this process by providing reusable components and abstractions for building and coordinating agent-based applications. Instead of implementing every capability from scratch, developers can focus more on the application's actual logic.
Depending on the framework, common capabilities may include:
Tool integration and tool calling
Agent state and memory management
Workflow and task orchestration
Multi-agent coordination
LLM and model integration
Observability, testing, and debugging support
Different frameworks focus on different problems. Some are useful for building simple tool-using agents, while others are designed for multi-agent systems, structured workflows, or production AI applications.
In the following sections, we will explore some popular AI agent frameworks, understand what each one provides, and see when it can be useful.
LangChain is an open-source framework for building applications powered by Large Language Models (LLMs).
It provides reusable components and integrations for working with models, prompts, tools, retrieval systems, and external data sources. This makes it easier to connect an LLM with the other components required by an AI application.
For example, a LangChain application can:
Receive a user request
Retrieve relevant information from external data
Let the model call available tools
Process the results
Generate the final response
LangChain is commonly used for building RAG applications, AI assistants, document-based applications, and tool-using agents.
AI Assistants
RAG Applications
Document Question Answering
Tool-Using Agents
LLM-Powered Applications
LangGraph is a framework for building stateful, long-running agents and workflows using a graph-based execution model.
Instead of limiting an agent to a simple sequence of steps, LangGraph allows a workflow to contain different paths, decisions, loops, and state.
A simple workflow might look like:
User Request
↓
Agent
↓
Need a Tool?
/ \
Yes No
↓ ↓
Use Tool Respond
↓
Agent
↓
RespondThe ability to return to an earlier step is useful for agents that may need to use multiple tools, evaluate results, and continue working until a task is complete.
LangGraph is particularly useful when an application needs more control over agent execution, state, and multi-step workflows.
Multi-Step AI Agents
Stateful Agent Workflows
Multi-Agent Systems
Long-Running Workflows
Human-in-the-Loop Applications
Decision-Based Workflows
CrewAI is an open-source framework for building multi-agent systems where multiple AI agents collaborate to complete a task.
Each agent can be given a specific role, goal, and set of tools. The agents can then work together as a team, with different agents handling different parts of a workflow.
For example:
User Task
↓
Research Agent
↓
Writer Agent
↓
Reviewer Agent
↓
Final OutputHere, the Research Agent collects information, the Writer Agent creates the content, and the Reviewer Agent checks the result.
CrewAI is useful when a complex task can be divided into specialized responsibilities handled by multiple collaborating agents.
Research Automation
Content Creation
Business Workflows
Multi-Agent Collaboration
Task Automation
AutoGen is an open-source framework from Microsoft for building agent-based and multi-agent applications.
It allows developers to create agents that can communicate with each other, use tools, and collaborate on tasks. Different agents can be assigned different responsibilities depending on the workflow.
For example, a software-development workflow might involve:
User Request
↓
Planning Agent
↓
Coding Agent
↓
Review Agent
↓
Final ResultAutoGen is useful for applications where multiple agents need to exchange information, coordinate actions, and work together on multi-step tasks.
Multi-Agent Applications
Research Assistants
Coding Workflows
Task Automation
Collaborative AI Systems
The OpenAI Agents SDK is an open-source framework for building agentic AI applications. It provides a small set of building blocks for creating agents that can use tools, maintain context, and collaborate with other agents.
An agent can be configured with instructions and tools, while the SDK manages the agent loop and tool execution. It also supports capabilities such as handoffs, guardrails, sessions, human-in-the-loop workflows, and tracing. Sessions can provide persistent conversation context across agent runs.
For example:
User Request
↓
OpenAI Agent
↓
Choose Tool
↓
Execute Tool
↓
Process Result
↓
Final ResponseThe SDK is useful when developers want a lightweight way to build tool-using or multi-agent applications while having the runtime manage common agent behavior.
AI Assistants
Customer Support Agents
Tool-Using Agents
Multi-Agent Workflows
Task Automation
Voice Agents
OpenAI Agents SDK documentation
Semantic Kernel is an open-source SDK from Microsoft for integrating AI models with application code, services, and external tools.
A central concept in Semantic Kernel is the Kernel, which manages AI services and plugins used by the application. Plugins expose functions that allow AI applications and agents to interact with existing APIs, databases, and business logic.
Semantic Kernel also provides an Agent Framework for creating agents and supporting agent-based workflows, including scenarios where multiple agents collaborate.
A simplified architecture looks like:
Application
↓
Semantic Kernel
/ \
↓ ↓
AI Model Plugins
↓
APIs / ServicesSemantic Kernel is especially useful when developers need to integrate AI capabilities with existing application code and enterprise systems.
Enterprise AI Applications
AI Assistants
Business Process Automation
Tool and API Integration
Multi-Agent Applications
LlamaIndex is a framework for building LLM-powered applications and AI agents that work with external data.
It provides tools for connecting data from sources such as documents, databases, APIs, and cloud services, then processing and retrieving the relevant information when an application needs it.
A simple flow looks like this:
External Data
↓
LlamaIndex
↓
Index / Retrieve
↓
Relevant Context
↓
LLM / AI Agent
↓
ResponseLlamaIndex is especially useful for Retrieval-Augmented Generation (RAG), where an application retrieves relevant information from its own data before asking the LLM to generate a response.
It can also be used in agentic applications where agents need to search, retrieve, and work with private or domain-specific knowledge.
RAG Applications
Enterprise Search
Knowledge Management
Document Intelligence
Data-Driven AI Agents
Research Assistants
There is no single AI agent framework that is best for every project. The right choice depends on factors such as application requirements, workflow complexity, data sources, tool integration, and whether you need single-agent or multi-agent workflows.
Here is a simple comparison:
Framework | Best For |
|---|---|
LangChain | General LLM applications, tool integration, and RAG |
LangGraph | Stateful agents and complex, controllable workflows |
CrewAI | Role-based multi-agent collaboration |
AutoGen | Conversational and multi-agent applications |
OpenAI Agents SDK | Lightweight tool-using and multi-agent applications with OpenAI's agent stack |
Semantic Kernel | Integrating AI with enterprise applications and existing business systems |
LlamaIndex | RAG, data retrieval, and knowledge-intensive AI applications |
For beginners, LangChain can be a useful starting point for learning common LLM application concepts and integrations. However, the best framework should be chosen based on the problem rather than popularity.
For example, LangGraph is useful when you need explicit control over state and multi-step execution, CrewAI focuses on role-based multi-agent collaboration, and LlamaIndex is a strong choice when the application mainly needs to work with external or private data.
In real-world projects, these technologies are also not always mutually exclusive. For example, an application may use an agent framework for orchestration while using a separate retrieval system for RAG.
AI agents combine multiple technologies to understand goals, access information, make decisions, use tools, and complete tasks.
Technologies such as APIs and tool integration allow agents to interact with external systems, while vector databases and knowledge graphs help them work with external knowledge and connected information. Depending on the application, agents may also use memory, planning, reasoning, and workflow orchestration.
Frameworks such as LangChain, LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, Semantic Kernel, and LlamaIndex provide reusable components that simplify the development of agentic applications.
There is no single framework that is best for every project. The right choice depends on the application's data, tools, workflow complexity, state-management needs, and single-agent or multi-agent architecture.
Understanding these technologies and frameworks provides a strong foundation for designing reliable, scalable, and production-ready AI agent systems.