Each month, I'll share my five seven top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Over the past few months, I’ve been developing and experimenting with several open-source packages while creating a few AI agents. This post covers six of my most popular LinkedIn posts on the topic, each showcasing the package’s core features and my key takeaways. Plus, a bonus tip at the end: a toolkit that automatically converts prompts from GPT and Claude into Llama-optimized versions, boosting performance by up to 45%.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Recent posts:

(1) LangMem

LangMem is a new open-source library that gives LLM agents long-term memory and it’s refreshingly easy to use.

It’s built for developers working with LangGraph or custom agents, and it solves a persistent problem: how to make agents remember and adapt across sessions without bloated prompts or manual hacks.

LangMem introduces a clean memory API that works with any storage backend and includes tools for:

1. Storing important information during conversations - agents decide what matters and when to save it

2. Searching memory when relevant - retrieving facts, preferences, or prior context

3. Running background memory consolidation - automatically refining and updating knowledge over time

It integrates natively with LangGraph’s memory store, but you can also plug it into your own stack using Postgres, Redis, or in-memory stores.

This design is especially useful for building agents that need to:

-> Personalize interactions across sessions

-> Maintain consistency in long-running workflows

-> Adapt behavior based on evolving user input

Unlike Mem0, which requires explicit memory updates, LangMem handles memory automatically in the background, storing and retrieving key details as needed, and integrates with LangGraph out of the box.

GitHub repo https://github.com/langchain-ai/langmem

(2) Browser Use

LLM agents can read the web, but few can truly use it. This open-source package changes that.

Browser-use is an open-source library that turns any LLM into a browser-native agent, with first-class support for real UI actions and multi-step tasks.

Out of the box, it supports:

1. Direct interaction with Chromium via Playwright - no extra scripting layers or wrappers

2. Seamless LLM integration - use GPT-4o, DeepSeek-V3, Claude, Gemini, or even Grok

3. Ready-to-run UX - spin up agents with a single function, or test flows in a ready-to-run Web UI or CLI

Browser use can handle real-world tasks like:

-> Checking your latest Stripe payouts and updating a financial tracking sheet

-> Logging into your CMS, creating a draft blog post, and uploading media

-> Scraping product reviews across sites and summarizing them in a shared doc

-> Tracking changes to your competitors’ pricing pages and alerting your team

For those building AI agents that go beyond chat, this gives you a direct bridge to the real web, not a sandbox.

GitHub repo https://github.com/browser-use/browser-use

My recent post on coding with AI:

(3) OpenAI Agents SDK

OpenAI has one of the most useful frameworks for multi-agent workflows, and it’s open-source.

Building production-ready agent systems has been notoriously complex, requiring deep knowledge of orchestration patterns, handoff mechanisms, and debugging distributed AI behavior. The new OpenAI Agents SDK simplifies this complexity with a remarkably clean Python interface that handles the heavy lifting.

Why I find this framework so useful:

1. Provider-agnostic design - works with OpenAI's APIs plus 100+ other LLMs, so you're not locked into a single provider

2. Built-in handoffs - agents can seamlessly transfer control to specialized agents based on context, like routing Spanish queries to Spanish-speaking agents

3. Integrated tracing - every agent run is automatically tracked using popular tools such as AgentOps, Braintrust, and Arize AI Phoenix, making debugging multi-agent conversations straightforward instead of impossible

4. Guardrails by default - configurable safety checks for input and output validation prevent runaway behavior

Setting up a triage system that routes conversations to language-specific agents takes just a few lines of code, with the SDK handling message persistence, context switching, and execution flow automatically.

I'm particularly impressed by the tracing capabilities - the framework integrates with popular observability tools like Logfire, AgentOps, and Braintrust, giving you visibility into exactly what your agents are doing and why.

For developers who have been intimidated by the complexity of multi-agent architecture, this SDK removes the final barrier to building sophisticated agent workflows that actually work in production.

GitHub repo https://github.com/openai/openai-agents-python

(4) Agno

Most agent frameworks I've used struggle with performance at scale, but I recently tested one that achieves microsecond-level instantiation.

The math doesn’t lie: if each agent takes seconds to spin up and consumes megabytes of memory, running the thousands needed for complex workflows becomes infeasible.

A new library called Agno addresses this through architectural decisions that prioritize performance without sacrificing functionality. The framework supports 23+ model providers and implements a progressive five-level agent architecture, from basic tool-enabled agents to coordinated multi-agent workflows.

Key technical capabilities include:

1. Native multimodal processing - handles text, image, audio, and video inputs without additional preprocessing layers

2. First-class reasoning implementation - agents can explicitly "think through" problems using built-in reasoning tools or custom chain-of-thought approaches

3. Agentic search with hybrid retrieval - combines vector search with keyword matching and re-ranking for improved RAG performance

The performance difference is substantial. In head-to-head comparisons with LangGraph, Agno completes instantiation benchmarks before competing frameworks reach halfway through their measurement cycles.

Agno also includes pre-built FastAPI routes, structured output handling, session storage, and monitoring capabilities.

GitHub repo https://github.com/agno-agi/agno

(5) Agents Towards Production

A new, comprehensive, open-source playbook has just solved the biggest challenge in developing AI agents: transitioning from experimentation to production-ready systems.

Unlike scattered documentation or theoretical frameworks, this resource provides executable tutorials that guide you from zero to a working implementation in minutes.

The playbook covers the entire agent lifecycle:

1. Orchestration fundamentals - build multi-tool workflows with memory persistence and agent-to-agent messaging using frameworks like Xpander and LangChain

2. Production deployment - containerize agents with Docker, scale on GPU infrastructure via Runpod, or run on-premise with Ollama for privacy-sensitive applications

3. Security and observability - implement real-time guardrails against prompt injection, add comprehensive tracing with LangSmith and Qualifire, and automate behavioral testing

4. Advanced capabilities - enable dual-memory architectures with Redis for semantic search, integrate real-time web data through Tavily, and deploy agents as APIs with FastAPI

What makes this resource invaluable is its tutorial-first approach. Each concept comes with runnable notebooks and production-ready code.

Whether you're building customer service agents, research assistants, or autonomous workflows, the playbook provides tested patterns for tool integration, multi-agent coordination, and model customization.

GitHub repo https://github.com/NirDiamant/agents-towards-production

(6) Docling

Read more

This one is an updated version of one of my recent popular posts, which outlined the most popular and useful open-source libraries for AI agent builders:

The landscape of AI agent tools is evolving rapidly. After publishing my previous post, I received suggestions for additional packages I hadn't encountered. Over the past few months, I've tested these tools and documented new, valuable agent-related libraries. This post shares my updated insights.

If you often wonder, "What tools are people actually using to build voice agents or understand documents?"—this post is for you. With new packages emerging almost daily, it can be challenging to determine what's state-of-the-art and truly usable. This list is deliberately selective, focusing on the libraries I've personally found most effective, or those recommended by colleagues I trust.

In this post, I'll provide a curated and updated overview of the open-source ecosystem for developers building AI agents. While there’s no shortage of AI agent market maps, most are geared toward non-builders who need actionable tools and frameworks to launch functional AI agents today.

Every package listed in this post allows commercial use and has a permissive open-source license.

Categories covered in this piece:
→ Building and Orchestrating Agents (10)
→ Computer Use (5)
→ Browser Automation (5)
→ Voice (12)
→ Document Processing (7)
→ Memory (3)
→ Testing, Evaluation, and Observability (6)
→ Vertical Agents (7)

Plus:

• Real-world agent stacks: Voice agent that answers phone calls + Browser agent that crawls LinkedIn URLs

• Curated guides and tutorials to get started building agents

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Building and Orchestrating Agents

To build agents that go beyond simple prompting, you need infrastructure for planning, memory, and tool use, and a way to hold it all together.

As more developers started shipping real-world agents, new frameworks popped up and older ones evolved to meet the actual challenges of agentic AI. This section covers the tools I’ve found most effective for building agents that can think, remember, and act with minimal hand-holding.

How to choose?

• For beginners or rapid prototyping, consider Langflow for its intuitive visual interface and OpenAI’s Agents SDK, or LangChain for their simplicity and flexibility.

• For enterprise applications, Portia and CrewAI offer robust features suitable for production environments requiring control and scalability.

• For multimodal or memory-intensive agents, Agno provides lightweight support for agents needing persistent memory and multimodal inputs.

• For complex simulations or data generation, Camel excels in creating customizable multi-agent systems for simulating real-world interactions.

• For autonomous task execution, AutoGPT is designed for agents that need to operate without continuous human input.

Computer Use

AI agents become far more useful when they can operate computers like humans: clicking, typing, browsing, and running programs. The libraries below make that possible, letting agents bridge the gap between language output and real-world action.

How to choose?

• For local code execution via natural language, go with Open Interpreter – it’s fast to set up and great for command-driven agents.

• For agents that need to see and control a computer screen like a human, Self-Operating Computer is your best bet.

• If your agent needs to run in a secure, fast, sandboxed environment, use CUA.

• For dynamic multi-step tasks on irregular interfaces, Agent-S offers the most flexibility with its planning and learning capabilities.

• If your agent relies on interpreting UIs from screenshots (e.g., grounding actions in visual layouts), OmniParser adds critical visual parsing capabilities.

Browser Automation

As AI agents move from passive reasoning to active execution, the browser becomes their gateway to the internet. Whether scraping data, submitting forms, or navigating complex workflows, browser automation tools let agents interact with web pages just like a human would—with clicks, scrolls, and typed input. These libraries differ in abstraction level, performance, and agent integration, so choosing the right one depends on your goals.

How to choose?

• For a low-code, declarative approach where the LLM plans the steps, try Stagehand.

• If you're building agents that need to deeply understand and extract content from websites, Firecrawl offers the cleanest pipeline.

• For LLM-friendly control over browser actions with integration hooks, I’d recommend the popular browser-use.

• Choose Playwright if you need more low-level control over browser actions across browsers.

• Use Puppeteer if you need fast, scriptable Chrome automation in a Node.js environment.

Voice

Voice is still the most intuitive interface for humans, and increasingly, for agents too. These tools let agents handle speech in and out: understanding spoken language, keeping track of conversations, and responding naturally.

How to choose?

I wrote a whole post covering open and commercial packages and APIs for voice agents, including my guide for choosing the right ones:

Dia compared to ElevenLabs and Sesame 👆

Document Processing

Modern AI agents must process and comprehend documents in various formats, from PDFs to images containing text. The following open-source tools empower agents to extract, interpret, and act upon information from unstructured documents, facilitating real-world business processes.

How to choose?

• Long-form PDFs such as contracts, research papers - use Qwen2.5-VL or mPLUG-DocOwl2 for efficient multi-page understanding without relying on OCR. And, as of a few months ago, you can also easily fine-tune a DocOwl2 model on your own data with ms-swift.

• Text + image docs such as medical reports, annotated diagrams - try Molmo for high-resolution multimodal inputs, visual QA, and GUI parsing.

• Layout analysis & table extraction - use Docling for JSON/Markdown conversion, or LayoutLMv3 for form understanding and layout-aware modeling.

• Lightweight multimodal with speech - Phi-4 handles text, vision, and speech in a compact model—great for on-device agents.

Memory

To feel truly intelligent, AI agents need memory. Without it, they’re stuck in single-turn loops, forgetting what just happened, what the user wants, or what they already did. The libraries below help agents remember, adapt, and personalize, enabling everything from contextual conversations to long-term planning.

How to choose?

• For long-term, personalized memory that improves with use, Mem0 is a solid choice—especially if you want minimal overhead.

• Use Letta when building agents that need persistent memory across sessions and integration with tools or APIs.

• To enable active memory management and knowledge sharing among agents, LangMem facilitates dynamic memory operations and shared knowledge bases.

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Testing, Evaluation, and Observability

As agents grow more complex, they need to be tested, measured, and monitored like any serious software system. These tools help you catch edge cases, debug behavior, and track performance, both during development and in production.

How to choose?

• To monitor and benchmark agent performance in production environments, AgentOps provides robust tracking and analysis tools.

• When comparing various agent configurations or conducting A/B tests, Agenta facilitates structured evaluations.

• To integrate observability into LLM applications, OpenLLMetry leverages OpenTelemetry for seamless monitoring.

• If detecting and addressing performance, bias, or security issues is a priority, Giskard offers automated scanning capabilities.

• For comprehensive LLM observability and debugging, Langfuse provides an open-source platform tailored for LLM applications.

• For voice agent evaluation across different models and prompts, VoiceLab offers a comprehensive testing framework.

Vertical Agents

The open-source world is full of vertical agents: purpose-built tools for coding, research, data analysis, and more. I’ve tested a bunch. These are the ones I’d actually reach for when building something real.

How to choose?

• Goose allows custom workflow integration to build extensible AI coding assistants.

• For comprehensive coding agents with GUI capabilities, OpenHands offers a full-stack solution inspired by Devin.

• If you prefer a Claude Code-like terminal-based pair programming, aider provides Git integration and multi-file editing.

• To convert UI designs from images to code, screenshot-to-code automates the prototyping process.

• For autonomous research tasks, GPT Researcher can scrape, summarize, and export findings efficiently.

• For conducting in-depth, privacy-focused research using local LLMs, Local Deep Research offers iterative analysis and comprehensive, cited reports.

• If your focus is on generating SQL queries from text, Vanna offers customizable and database-integrated solutions.

  

Lastly, here are real-world stacks and beginner-friendly tutorials to help you launch your first AI agent.

Real-world agent stacks

All the tools above are powerful on their own, but how do they actually fit together in practice? What does a real architecture look like when you're stitching these components into something usable, testable, and shippable? I’ve compiled a few concrete examples from recent open-source projects and builders in the space. If you're trying to move from “exploring tools” to “building real systems”, these will give you a head start.

Voice agent that answers phone calls

Read more

A NotebookLM-powered podcast episode discussing this post:

December 2028. Maria's AI agent is negotiating simultaneously with twelve different vendors for her daughter's upcoming birthday party. Within minutes, it secured the perfect cake from a local bakery (after verifying their nut-free certification), booked an entertainer with stellar safety ratings, and coordinated custom goodie bags filled with each child's favorite treats (after checking allergies and dietary restrictions with the other parents' agents)—all while staying 15% under budget. What would have taken Maria hours of calls, emails, and anxiety about vendor reliability now happens seamlessly through a web of agent-to-agent interactions powered by the new infrastructure we explored in our previous post.

The revolution in web infrastructure we discussed in previous posts isn't just theoretical—it's enabling fundamental changes in how commerce, marketing, and customer service function. As agent passports and trust protocols become standardized, we're witnessing the emergence of entirely new commercial paradigms.

With the recent release of Tasks by OpenAI, which equips ChatGPT—its consumer-facing AI—with the ability to perform tasks behind the scenes on behalf of users, it's now easier than ever to envision a future where ChatGPT seamlessly navigates the web and handles complex operations for us.

Today, we'll explore how an agent-first internet will reshape domains like payments, marketing, support, and localization.

Agentic payments

Remember when online shopping first emerged, and entering your credit card details on a website felt risky? Card networks like Visa and Mastercard and banks like Chase and Barclays had to rapidly adapt to the digital realm, introducing new protocols like CVV codes and secure payment gateways to protect consumers and merchants alike. This evolution was pivotal in building trust and facilitating the e-commerce boom of the early 2000s, giving birth to digital payment giants like Stripe and PayPal.

Similarly, the payment infrastructure that powers today's internet was built around a simple dichotomy: card-present versus card-not-present transactions and the assumption that all payments are human-initiated. The internet already faced a similar shift with the introduction of subscription payments in the early days of e-commerce, in which customers input their payment credentials once, allowing a merchant to charge them repeatedly in the future. Recurring payments, now a dominant commerce mechanism projected to surpass $2.4T in spending by 2028, underscore how critical adaptive payment systems are to supporting global e-commerce. Yet, even more than a decade later, there are still countries like India that impose strict controls on recurring payments, highlighting the complexities of adapting systems to new paradigms.

But what happens when the legitimate transacting party isn’t human at all?

Digital payment systems are built around human users, employing measures like CVV codes and billing address verification to prevent fraud. These methods assume a human is initiating the transaction, making them ill-suited for autonomous agent transactions.

One major challenge is fraud detection and resolution—an area I intimately understand from my time as a PM at Stripe, where I worked closely with card issuers to develop sophisticated fraud prevention systems. The current paradigm relies heavily on human behavioral patterns: typing speed, mouse movements, time spent reviewing checkout pages, and other signals that indicate legitimate human activity. But in an agent-driven world, these signals become obsolete.

During my time at Stripe, we saw how crucial these behavioral signals were for differentiating legitimate transactions from fraudulent ones. The shift to agent-driven commerce demands an entirely new approach. Rather than looking for signs of human behavior, we'll need systems that verify agent legitimacy, authorization scope, and decision-making logic.

This shift raises critical questions for fraud prevention across banks (Chase, Barclays), card networks (Visa, Mastercard), and payment processors (Stripe, PayPal). For instance, how do you establish a chain of trust when an agent purchases on behalf of a user? How do you verify that an agent hasn't been compromised or hijacked? These challenges require fundamentally rethinking our approach to transaction security and fraud prevention in an agent-driven ecosystem.

Future payment systems could introduce 'agent wallets' with granular spending controls, such as $100 limits for trusted merchants like Amazon and stricter caps for lesser-known websites. These wallets would integrate real-time fraud detection, submitting cryptographic evidence for disputes and maintaining transparent, auditable records of agent actions tied to human authorization.

Visa could introduce an agent-specific flag to the existing payment protocols, indicating this payment was initiated autonomously, along with a trial of reasoning and actions leading to this payment. Meanwhile, Stripe might expand its SDKs to enforce programmable payment rules, ensuring alignment with user-delegated instructions (Stripe has already made its foray into Agentic payments with its recent SDK release).

Beyond preventing fraud, agentic payments face fundamental economic and infrastructural challenges. The existing payment infrastructure wasn't architected for the high-frequency, low-latency transactions that characterize agent interactions. Consider the standard pricing model of payment processors like Stripe: a 2.9% fee plus 30¢ per transaction. While manageable for traditional e-commerce, this fee structure becomes prohibitively expensive when scaled to the myriad micro-transactions that agents might need to execute.

This pricing isn't arbitrary—it reflects the complex web of stakeholders in the traditional payment chain. Card networks like Visa and issuers like Chase have built their business models around these transaction fees. Interestingly, Stripe's recent acquisition of Bridge, a stablecoin payment infrastructure provider, hints at a potential solution. By leveraging blockchain-based payment rails, companies could facilitate agent-to-agent transactions without incurring the expensive overhead of traditional payment networks. This move suggests a growing recognition that the future of payments may require completely new infrastructure, optimized for the unique demands of autonomous agents.

Consider how a new payment protocol might work in practice. An "Agent Payment Protocol" (APP) could include:

• Delegation Chain Verification - a cryptographic proof chain showing the agent's authorization to make specific types of purchases

• Transaction Context Object - machine-readable metadata including:

  • Reasoning trail that led to the purchase decision

  • Reference to specific user preferences/rules that were satisfied

  • Confidence score for the decision

• Smart spending controls with programmable constraints like:

  • Category-specific limits (e.g., $200 for groceries, $50 for entertainment)

  • Merchant-specific trust scores

  • Required human confirmation above certain thresholds

  

Major payment providers could implement this through extensions to existing standards. For instance, Visa's existing 3D Secure protocol could add an agent verification layer, while Stripe's API could introduce new parameters for agent-specific transaction metadata.

While payment infrastructure provides the foundation for agent-driven commerce, the very nature of how we complete transactions must also evolve. The familiar checkout process—a hallmark of e-commerce for decades—is about to undergo its own transformation.

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Redefined checkout experience

In an agent-first environment, the concept of a traditional checkout—where a human user confirms their purchase by clicking a “Buy” button—fades into the background. Instead, agents operate with predefined goals and parameters, continuously evaluating whether a proposed transaction aligns with those objectives. Rather than halting everything at a payment prompt, agents could integrate a “stop and reflect” step into their workflows. For example, if a user’s agent is tasked with booking a flight seat that’s both a window seat and close to an exit, the agent pauses before completing the reservation. It double-checks that the seat assignment matches the user’s criteria and only then proceeds, ensuring flawless execution of the user’s intent and mitigating the probability of the agent going off the rails due to hallucinations.

This reflective process transforms the final authorization into a subtle verification loop rather than a jarring user interruption. The agent reviews the selected attributes—price, seat location, baggage allowance, and cancellation policy—and compares them against the user’s stored preferences and constraints. It confirms not only that the requested outcome has been met but also that it falls within acceptable spending limits and trust parameters. This transforms purchasing from a manual “Are you sure?” prompt into a nuanced, data-driven decision matrix.

As these agent-mediated transactions proliferate, payment providers and merchants might offer additional layers of context-aware validation. For instance, when an agent chooses a specific insurance add-on, the payment system could prompt the agent to confirm whether its logic correctly interpreted the user’s needs. This transparent chain of reasoning, visible to the agent and logged for future reference, ensures that each transaction stands up to scrutiny. Ultimately, the checkout step evolves from a user-facing choke point to an agent-managed quality control measure, minimizing errors and elevating the overall integrity of automated commerce.

As agents reshape how we complete purchases, they're also forcing us to rethink how businesses attract and engage customers in the first place. The era of human-centric marketing campaigns is giving way to something far more systematic and efficient.

Agent-driven marketing and promotions

Marketing campaigns and promotions will evolve radically in an AI agent-mediated economy.

Traditional email marketing and coupon distribution systems, designed around human attention and impulse, will give way to programmatic offer networks where consumers’ AI agents maintain persistent queries about their principals' needs and preferences. These agents subscribe to vendor APIs that broadcast real-time offers matching specific criteria, enabling hyper-personalized deal discovery that transcends the limitations of batch email campaigns.

Vendors might maintain agent-first promotional channels that communicate in structured data formats, allowing instant price comparison and benefit analysis. This ecosystem could enable “intent casting", where agents broadcast shopping goals to trusted vendor networks, receiving precisely targeted offers that align with the principal's timing, budget, and preferences—all without cluttering a human inbox or requiring manual coupon management.

Consider a practical example: A user instructs their agent to monitor high-end fashion retailers for specific items within their style preferences and budget constraints. Rather than the user repeatedly checking websites or subscribing to countless email lists, their agent maintains persistent monitoring across multiple vendors:

• Real-time inventory tracking across size, color, and style variations

• Dynamic price monitoring, including flash sales and member-exclusive discounts

• Evaluation of shipping times and costs to the user's location

When ideal conditions align—perhaps a preferred sweater hits the target price point during an end-of-season sale—the agent can either notify the user or execute the purchase automatically based on pre-authorized parameters. This transforms shopping from an attention-demanding activity into an efficient background process governed by clear rules and preferences.

Major retailers like Nordstrom or ASOS could expose agent-specific APIs that provide structured access to:

• Real-time inventory and pricing data

• Detailed product specifications and measurements

• Membership program benefits and restrictions

• Regional availability and shipping constraints

This evolution democratizes personal shopping, allowing everyone to benefit from persistent, intelligent monitoring of their fashion preferences—not just those who can afford human personal shoppers. It also enables retailers to better match inventory with actual customer intent, reducing overhead from unsold merchandise and improving supply chain efficiency.

Other companies like Honey may need to pivot to offer agent-optimized tools that integrate directly with vendor APIs, allowing agents to query real-time discounts and rewards. Similarly, Mailchimp and HubSpot could develop agent-oriented campaign frameworks that distribute offers as structured data streams rather than traditional email blasts, ensuring seamless integration with agent-driven workflows.

Agent-native customer support

Customer support today centers on human interactions or user-facing chatbots. In an agent-first paradigm, this shifts to agent-to-agent communication. Personalized AI agents will directly engage with business systems to resolve issues, retrieve shipping or refund policies, or autonomously initiate returns. This evolution will streamline processes, reduce human intervention, and enhance efficiency in support workflows.

In an agent-first paradigm, customer support is no longer solely about human users contacting businesses through chat widgets or call centers. Instead, autonomous agents interact directly with enterprise systems, pulling diagnostic information, requesting refunds, or escalating complex issues to a more constrained/expensive resource like a human or a superior model (e.g., o1 over GPT-4o). This shift encourages platforms like Intercom to develop agent-oriented communication layers—specialized APIs that allow autonomous agents to navigate support options, retrieve knowledge base articles, and submit detailed troubleshooting requests without human intervention.

These agent-facing APIs would streamline issue resolution, allowing routine queries, such as package tracking, account verification, or policy clarifications, to be handled agent-to-agent, drastically reducing response times. As soon as a problem arises, the user’s agent can pinpoint the issue and connect with the business’s support agent (be it a specialized LLM or a human representative), negotiating resolutions or applying discounts as needed. The result is a fluid, automated dialogue that bypasses human frustration and latency.

Over time, companies could implement reputation scoring systems that measure how efficiently their support agents (both human and AI) interact with consumer agents. Metrics like resolution speed, policy clarity, and refund accuracy become machine-readable signals, informing user agents which vendors offer superior support experiences. As more vendors embrace these standards, the entire support ecosystem evolves: prompt, well-structured responses become the norm, and agent-native customer support becomes a hallmark of high-quality digital services.

Imagine a complex warranty claim scenario: Your agent detects that your new laptop's battery is degrading unusually fast. It immediately:

1. Collects diagnostic data and usage patterns

2. Cross-references warranty terms with actual performance

3. Initiates a support interaction with the manufacturer's agent

4. Negotiates a resolution based on precedent cases

5. Arranges shipping for replacement parts or full device replacement

6. Schedules a technician visit if needed

This entire process happens without human intervention unless exceptional circumstances arise. The interaction generates a complete audit trail, including all diagnostic data, communication logs, and decision points—valuable data for improving both product quality and support processes.

One example in this space of agentic customer support is Sierra, a startup taking aim at the expansive market of customer support by embedding AI agents into business workflows. Their conversational agents handle complex queries with contextual precision, managing tasks such as processing returns or updating subscriptions. While their primary focus remains on serving human customers, the foundation they’ve built is clearly aligned with an AI agent-driven future. With access to company policies (e.g., refund and shipping rules) and robust conversational AI infrastructure (spanning LLMs and voice interfaces), Sierra is well-positioned to seamlessly transition to support agent-to-agent interactions as demand evolves.

The end of language-optimized interfaces

As AI agents seamlessly translate and interpret information on the fly, the need for painstakingly maintained multilingual websites diminishes. Instead of forcing businesses to host separate English, French, or Mandarin versions of their interfaces, agents handle language conversion dynamically. This capability allows brands to maintain a single, streamlined codebase while ensuring that users, regardless of location, receive content and instructions in their preferred language—instantly and accurately.

Website builders like Webflow and Wix could evolve into platforms that generate “universal templates” optimized for agent interpretation rather than human linguistic preferences. Instead of focusing on localized landing pages, these platforms would produce standardized, machine-readable structures enriched with metadata and semantic cues. Agents, armed with cutting-edge language models, would then adapt the presentation layer for each user, including local dialects, cultural nuances, and even personalization cues drawn from the user’s profile.

The transformation goes beyond simple translation. Agents will handle complex cultural adaptations across multiple dimensions simultaneously. They'll dynamically adjust pricing strategies for different markets while modifying product descriptions to reflect local preferences and purchasing patterns. These agents will intelligently adapt imagery and design elements to ensure cultural appropriateness, automatically managing regional compliance requirements such as privacy policies or consumer protection disclosures. They'll even personalize communication styles based on cultural norms, shifting between formal and casual tones and adapting messaging cadence to match local expectations. This comprehensive cultural intelligence transforms what was once a labor-intensive localization process into a fluid, automated system that maintains cultural authenticity across all customer touchpoints.

In this new reality, the value proposition of website builders shifts from localization to robustness, structure, and data integrity. Rather than wrestling with manual translations or commissioning multiple language variants, businesses can rely on well-defined data schemas and agent-ready manifests. As a result, the concept of “language-optimized” sites becomes obsolete, replaced by fluid, dynamic interfaces that transcend linguistic barriers.

Small vs. large business impact

This transition creates both opportunities and challenges across the business spectrum. Large enterprises can invest in building sophisticated agent interfaces and maintaining complex agent-ready APIs. However, small businesses might initially struggle with the technical requirements and infrastructure costs.

To bridge this gap, we will likely see the emergence of "agent-enablement platforms"—services that help small businesses become agent-ready without significant technical investment. Think of them as the Shopify of the agent era, providing standardized tools that level the playing field. These platforms would offer pre-built solutions for creating agent-readable product catalogs and managing automated pricing and inventory systems. They would include standardized support protocols that small businesses can easily implement, along with simplified integration paths to agent payment systems. By democratizing access to agent-ready commerce capabilities, these platforms will play a crucial role in preventing a digital divide between large and small businesses in the agent economy.

This democratization of agent-ready commerce will be crucial for preventing a digital divide between large and small businesses in the agent economy.

The Great Rewiring

The transition to an agent-first internet represents more than just a technological shift—it's a fundamental reimagining of how commerce functions in the digital age. We're moving from a web optimized for human attention and interaction to one built for efficient, automated decision-making. This transformation touches every aspect of online business:

• Payment systems evolve from human-verification models to agent-oriented protocols with built-in delegation and accountability

• Marketing shifts from attention-grabbing campaigns to structured, machine-readable offer networks

• Customer support transforms from human-to-human interaction to efficient agent-to-agent problem resolution

• Language barriers dissolve as agent-mediated communication enables seamless global commerce

Companies that quickly adapt to this new paradigm—implementing agent passports, embracing agent-to-agent protocols, and restructuring their services for machine readability—will shape the next era of online interaction. Just as the mobile revolution created trillion-dollar opportunities, the agent revolution opens new horizons for innovation and value creation. The businesses that thrive won't just be those with the best products or prices, but those that best enable and embrace agent-driven commerce.

This is the third essay in a five-part series exploring the future of AI agents and their impact on the internet. Having established the foundational shifts and their practical implications across multiple domains, our next post will examine early case studies and implementation challenges as businesses begin this transformation. Subscribe and follow as we continue to navigate the technical, social, and economic implications of this new paradigm.

A NotebookLM-powered podcast episode discussing this post:

Over the past few weeks, I've explored how AI agents will fundamentally reshape the internet. From introducing the concept of "agent-responsive design" - where websites optimize for AI interaction - to examining the infrastructure needed for agent-to-agent communication protocols, my previous posts laid the groundwork for understanding the agent-centric future of the web.

Now, it's time to tackle perhaps the most natural and transformative interface for these agents: voice.

While text-based agents have dominated the early wave of AI applications, voice represents the next frontier in human-AI interaction. It's not just another interface - it's the most intuitive and accessible way for humans to interact with AI systems. This intersection of voice technology and AI agents creates unprecedented opportunities for developers, so I'm excited to share this comprehensive guide to the voice AI ecosystem.

After decades of frustrating experiences with scoped voice assistants that cannot be interrupted and follow a narrow rule-based script, we're witnessing a fundamental shift in what's possible. Three key developments drive this shift:

1. Breakthrough in speech-native models - the release of OpenAI's Realtime API last October and Google's Gemini 2.0 Realtime Multimodal API last week mark a transition from traditional "cascading architectures" (where speech is converted to text, processed, and converted back) to speech-native models that can process audio directly with unprecedented quality. With OpenAI's recent 60% Realtime API price reduction and the hiring of WebRTC's founder, we're seeing a clear industry push toward making real-time voice interactions accessible and affordable.

2. Dramatic reduction in complexity - what previously required hundreds of data scientists can now be achieved by small teams of AI engineers. We're seeing companies reach substantial ARR with lean teams by building specialized voice agents for specific verticals - from restaurant order-taking to lead qualification for sales teams.

3. Infrastructure maturity - the emergence of robust developer platforms and middleware solutions has dramatically simplified voice agent development. These tools handle complex challenges like latency optimization, error handling, and conversation management, allowing developers to focus on building unique user experiences.

This convergence creates a unique opportunity for builders. For the first time in human history, we have a god-like AI systems that converse like humans. The era of capable voice AI has arrived, opening up vast opportunities for innovators and developers alike.

Unlike web or mobile app development, where patterns are well-established, voice AI is still in its formative stage. The winners in this space will be those who can combine technical capability with a deep understanding of specific industry needs.

In this post, I'll provide a well-curated overview of the open-source and commercial tools available for developers building voice agents. While VCs segment the market based on investment opportunities, I'll map the ecosystem based on what matters to developers: APIs, SDKs, and tools you can actually use today. What is the go-to model for speech-to-text? The API for synthesis speech? Which tools do other builders rely on to develop voice agents? With the holiday season upon us, there's no better time to build your voice agent, turn it into a company, or automate a personal workflow.

Categories covered in this piece, along with my recommended stack and tips for choosing the right architecture at the end:

Foundations
→ Speech-to-text
→ Text-to-speech

Voice AI Middleware
→ Speech-to-speech
→ Conversational Agents Frameworks
→ Evaluation

End-to-end Automation
→ Full Stack

Missing package or tool you think should have been part of this list? Comment on this post and I’ll take a look.

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Foundations

Speech-to-text (STT)

Automated transcription models have improved significantly over the past couple of years, featuring low latency and cost along with a decreasing WER (Word Error Rate). When selecting a speech-to-text model, consider these key factors:

• Latency requirements - for real-time conversations, you need models that can process speech with < 300ms latency. Cloud APIs like Assembly AI and Deepgram excel here, while Whisper variants are better suited for async transcription.

• Deployment constraints - open source models like Whisper.cpp offer flexibility for edge deployment and privacy-sensitive applications but require more engineering effort to optimize. Cloud APIs provide easier scaling but with higher operational costs.

• Language and accent support - while most models handle standard English well, performance varies significantly for other languages and accents. Whisper has broad language support, while specialized APIs like Assembly AI may offer better accuracy for specific use cases.

Open-source (mostly Whisper-based)

1. Whisper.cpp

2. WhisperX

3. RealtimeSTT

Commercial

1. Deepgram1

2. Assembly AI

3. OpenAI’s hosted Whisper

4. Cloud Providers: Azure, Google, AWS. There don’t seem to be significant differences between the cloud providers, so I usually opt for the one I already use for other services like storage and compute.

👆 RealtimeSTT featured low-latency transcription with wake word activation

Text-to-speech (TTS)

Text-to-speech, once dominated by ElevenLabs’ costly API as of June 2023, has evolved into a competitive market with numerous low-latency, cost-effective alternatives. Modern TTS systems have evolved far beyond simple voice synthesis, offering capabilities like:

• Voice cloning - creating custom voices from just a few minutes of audio samples, enabling personalized agent voices

• Emotional synthesis - adjusting tone, pace, and emphasis to convey different emotions and speaking styles

• Multi-speaker synthesis - seamlessly switching between different voices in a single conversation

The most significant recent advancement is the emergence of "speech-native" models that can generate highly natural speech without the traditional text-intermediate step. This enables more fluid conversations with lower latency and better preservation of emotional nuances.

Open-source

1. Coqui TTS

2. ChatTTS

3. OpenVoice

Commercial

1. Cartesia

2. OpenAI TTS

3. Deepgram Aura

4. ElevenLabs (just released Flash—a 75ms latency model in 32 languages)

Other popular APIs: Play, Amazon Poly, Azure AI Speech, Google

👆 Hello Patient’s voice agent is powered by Cartesia

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Voice AI Middleware

Speech-to-speech

OpenAI sparked everyone’s imagination when they demoed GPT-4o last April, showcasing real-time conversational AI that can be interrupted. Until a few months ago, this kind of technology wasn’t available to developers. Now, we have commercial APIs at a reasonable price (OpenAI dropped the price for its Realtime API by 60% last week) and open-source alternatives.

Speech-to-speech models are specialized AI systems that process and respond to voice input directly in the audio domain without converting to and from text. They are superior to cascading workflows that involve sequential speech-to-text, text-to-text, and text-to-speech models for several key reasons:

1. Ultra-low latency - by processing audio natively, these models achieve response times of ~300 milliseconds, matching natural human conversation speeds. Traditional cascading approaches often exceed 1000ms, creating noticeable delays.

2. Better contextual understanding - speech-to-speech models preserve important non-textual elements like tone, emotion, and conversation dynamics that are often lost in text conversion.

3. Natural interruptions - unlike cascading systems that require rigid turn-taking, speech-native models can listen and process input even while speaking, enabling natural interruptions and more fluid conversations.

4. Improved conversation quality - by maintaining the audio context throughout the interaction, these models better handle overlapping speech, background noise, and the natural rhythm of human dialogue.

Open-source

• Ultravox

• Moshi

Commercial

• OpenAI’s Realtime API

• Gemini 2.0 Multimodal Live API

👆 Conversing with Google’s new Gemini 2.0 (source: Gemini Playground)

Conversational Agents Frameworks

Conversational agent frameworks provide the essential scaffolding needed to build production-ready voice AI applications. These frameworks abstract away much of the complexity in managing real-time voice infrastructure, handling edge cases, and orchestrating the various components of a voice application. Pipecat offers an open-source approach for developers who want complete control over their stack and need flexibility to build custom multimodal experiences. Vocode provides a more opinionated toolkit that simplifies the process of creating voice-based LLM agents, handling much of the complexity around conversation management and error handling. Vapi takes this further by offering a full-featured platform that abstracts away infrastructure challenges while providing enterprise-grade reliability and scalability. The choice between these frameworks often comes down to the specific requirements of your project - whether you need the flexibility of open-source, the simplicity of a focused toolkit, or the robustness of a complete platform.

Open-source

• Pipecat

• Vocode

Commercial

• Vapi

Evaluation

One of the most significant challenges in voice AI development is accurately measuring agent performance. Unlike text-based interactions, where metrics like response accuracy and latency are relatively straightforward, voice agents require a more nuanced evaluation approach that considers elements like conversation flow, emotional appropriateness, and real-world task completion.

Technical performance metrics form the foundation of evaluation, starting with response latency - which should target under 300ms for natural conversation and be measured across different percentiles, while speech recognition accuracy is tracked through Word Error Rate (WER) with special attention to domain-specific terminology. The second dimension focuses on conversational quality metrics that measure the agent's ability to handle natural dialogue. This includes turn management metrics like interruption handling and end-of-speech detection, along with task completion metrics such as success rate and the number of turns needed to complete a task. User experience metrics round out the evaluation framework, tracking call abandonment rates, average call duration, user sentiment scores, and the frequency of repeat requests.

When done right, proper evaluation becomes a game-changing unlock - it enables developers to identify subtle issues that could frustrate users (like awkward pauses or mismatched emotional responses), optimize for natural conversation patterns, and ensure consistent performance across different accents and background noise conditions.

Open-source

• Voice Lab

Commercial

• Coval

End-to-end Automation

Full Stack

The Stripe for voice agents. From prompt to a call.

While building with individual components offers maximum flexibility, full-stack solutions make sense when you need to quickly deploy production-ready voice agents without managing infrastructure complexity. These platforms are particularly valuable for teams that need to focus on their core business logic and customer experience rather than wrestling with the intricacies of voice infrastructure, latency optimization, and scaling challenges.

These solutions typically provide end-to-end capabilities, including pre-built integrations with popular business systems (CRMs, scheduling tools, payment processors), sophisticated error handling, automated failover, and comprehensive analytics. They handle complex engineering challenges like maintaining low latency during peak loads, graceful degradation during API failures, and automated quality monitoring. While you trade some flexibility compared to building your own stack, these platforms can dramatically accelerate time-to-market and reduce engineering overhead.

• Bland

• Deepgram Voice Agents API

• Eleven Labs Conversational AI

👆 A Deepgram-powered voice agent handles a food order

Choosing the right architecture for your voice agent

When selecting a voice AI architecture, developers face a fundamental choice between using full-stack platforms and assembling custom solutions from individual components. This decision ultimately comes down to three key factors: latency requirements, cost constraints, and the need for fine-grained control over the conversation flow.

Read more

A NotebookLM-powered podcast episode discussing this post:

December 2028. Sarah's AI agent encounters an unusual situation while booking her family's holiday trip to Japan. The multi-leg journey requires coordinating with three different airlines, two hotels, and a local tour operator. As the agent begins negotiations, it presents its "agent passport"—a cryptographic attestation of its delegation rights and transaction history. The vendors' systems instantly verify the agent's authorization scope, spending limits, and exposable metadata like age and passport number. Within seconds, the agent has established secure payment channels and begun orchestrating the complex booking sequence. When one airline's system flags the rapid sequence of international bookings as suspicious, the agent smoothly provides additional verification, demonstrating its legitimate delegation chain back to Sarah. What would have triggered fraud alerts and CAPTCHA challenges in 2024 now flows seamlessly in an infrastructure built for autonomous AI agents.

—> The future, four years from now.

In my previous essay, we explored how websites and applications must evolve to accommodate AI agents. Now we turn to the deeper infrastructural shifts that make such agent interactions possible. The systems we've relied on for decades: CAPTCHAs, credit card verification, review platforms, and authentication protocols, were all built with human actors in mind. As AI agents transition from experimental curiosities to fully operational assistants, the mechanisms underpinning the digital world for decades are beginning to crack under the pressure of automation.

The transition to an agent-first internet won't just streamline existing processes—it will unlock entirely new possibilities that were impractical in a human-centric web. Tasks that humans find too tedious or time-consuming become effortless through automation. Instead of clicking 'Accept All' on cookie banners, agents can granularly optimize privacy preferences across thousands of sites. Rather than abandoning a cart due to complex shipping calculations, agents can simultaneously compare multiple courier services and customs implications. Even seemingly simple tasks like comparing prices across multiple vendors, which humans typically limit to 2-3 sites, can be executed across hundreds of retailers in seconds. Perhaps most importantly, agents can maintain persistent relationships with services, continuously monitoring for price drops, policy changes, or relevant updates that humans would miss. This shift from manual, limited interactions to automated, comprehensive engagement represents not just a change in speed, but a fundamental expansion of what's possible online.

Amid these sweeping changes, a new gold rush is emerging. Just as the shift to mobile created opportunities for companies like Uber and Instagram to reinvent existing services, the transition to agent-first infrastructure opens unprecedented possibilities for founders. From building next-generation authentication systems and trust protocols to creating agent-mediated data marketplaces, entrepreneurs have a chance to establish the foundational layers of this new paradigm. In many ways, we're returning to the internet's early days, where core infrastructure is being reimagined from the ground up—this time for an autonomous, agent-driven future.

In this second post of the AI Agents series, we’ll focus on the foundational infrastructure changes that underlie the agent-first internet: new authentication mechanisms, trust systems, novel security challenges, and agent-to-agent protocols, setting the stage for the more commerce-oriented transformations we’ll explore in the following post.

Proving you're a human an agent

Remember when "proving you're not a robot" meant deciphering distorted text or selecting crosswalk images? Those mechanisms become obsolete in a world where legitimate automated actors are the norm rather than the exception. Today’s CAPTCHAs, designed to block bots, have become increasingly complex due to advances in multimodal AI. Paradoxically, these mechanisms now hinder real humans while sophisticated bots often bypass them. As AI outpaces human problem-solving in these domains, CAPTCHAs risk becoming obsolete, reducing website conversions and frustrating legitimate users.

The challenge shifts from proving humanity to verifying the agent has been legitimately delegated and authorized by a human user.

Today’s rate-limiting mechanisms assume human-paced interactions, relying heavily on IP-based throttling to manage access. But in a world of AI agents, what constitutes "fair use" of digital services? In an agent-driven internet, automated browsing will become not just accepted but essential. Cloudflare, Akamai, and similar services will need to pivot from simplistic IP-based throttling to sophisticated agent-aware frameworks.

As businesses grapple with these challenges, a new solution is emerging—one that shifts the paradigm from blocking automated traffic to authenticating and managing it intelligently. Enter the Agent Passport.

Imagine a digital credential that encapsulates an agent's identity and permissions—cryptographically secured and universally recognized. Unlike simple API keys or OAuth tokens, these passports maintain a verifiable chain of trust from the agent back to its human principal. They carry rich metadata about permissions scope, spending limits, and authorized behaviors, allowing services to make nuanced decisions about agent access and capabilities.

By integrating Agent Passports, business websites like airlines can distinguish between legitimate, authorized agents and malicious actors. New metrics, such as agent reliability scores and behavioral analysis, could ensure fair access while mitigating abuse, balancing security with the need for allowing agent-driven traffic.

Authentication mechanisms, such as signing up and signing in, must also evolve for an agent-first internet. Websites will need to determine not just an agent's identity but also its authorized scope—what data the agent is authorized to access (‘read’) and what actions it is permitted to execute (‘write’).

Google Login revolutionized online authentication by centralizing access with a single credential, reducing friction and enhancing security. Similarly, agent passports could create a universal standard for agent authentication, simplifying multi-platform access while maintaining robust authorization controls.

Companies like Auth0 and Okta could adapt by offering agent-specific identity frameworks, enabling seamless integration of these passports into their authentication platforms. Meanwhile, consumer companies like Google and Apple could extend their authentication and wallet services to seamlessly support agent-mediated interactions, bridging the gap between human and agent use cases.

A new protocol for Agent-to-Agent communication

In the early days of the web, protocols like HTTP emerged to standardize how browsers and servers communicated. In much the same way, the rise of agent-mediated interactions demands a new foundational layer: an Agent-to-Agent Communication Protocol (AACP). This protocol would formalize how consumer agents and business agents discover each other’s capabilities, authenticate identities, negotiate trust parameters, and exchange actionable data—all while ensuring both parties operate within well-defined boundaries.

Just as Sarah's travel agent from the intro paragraph seamlessly coordinated with multiple airlines and hotels, AACP enables complex multi-party interactions that would be tedious or impossible for humans to manage manually.

Much like HTTPS introduced encryption and certificates to authenticate servers and protect user data, AACP would implement cryptographic attestation for agents. Trusted third-party authorities, similar to today’s certificate authorities, would issue digital “agent certificates” confirming an agent’s legitimacy, delegation chain, and operational scope. This ensures that when a consumer’s travel-planning agent communicates with an airline’s booking agent, both sides can instantly verify authenticity and adherence to agreed-upon standards.

Without such a protocol, a rogue agent might impersonate a trusted retailer to trick consumer agents into unauthorized transactions, or a malicious consumer agent could spoof credentials to overwhelm a merchant’s infrastructure. By mandating cryptographic proof, robust authentication handshakes, and behavior logs, AACP mitigates these threats before meaningful data or funds change hands.

The handshake phase in AACP would include mutual disclosure of the agents’ technical stacks—such as which LLM or language configuration they use—and their supported capabilities. Once established, the protocol would also govern “write-like operations” (e.g., initiating a payment or updating account details) by enforcing strict sign-offs with auditable cryptographic signatures. Every action would leave a verifiable trail of authorization that can be reviewed and validated after the fact.

Finally, AACP would incorporate locale and language negotiation at the protocol level. Although agents can translate and interpret content dynamically, specifying a preferred language or locale upfront helps streamline interactions. This new protocol weaves together trust, authentication, and contextual awareness, forging a resilient substrate on which the agent-first internet can reliably function.

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Trust and reputation reimagined

When we navigate the internet, our judgment of a website's credibility hinges on a blend of visual and social cues. We look for secure HTTPS connections, professional design, and familiar branding to assure us that a site is trustworthy. No one wants to input their credit card information on a site that looks like it was built in the early 2000s. User reviews and star ratings on platforms like Trustpilot and G2 further influence our decisions, offering insights drawn from shared human experiences.

Perhaps no aspect of online commerce requires more fundamental reimagining than trust and reputation systems. In an agent-mediated economy, traditional cues for reliability fall short. AI agents can't interpret visual aesthetics or branding elements–they operate on data, protocols, and cryptographic proofs. Trust mechanisms must pivot from human perception to machine-readable verifications. For instance, an agent might verify a seller's identity through cryptographic attestations and assess service quality via automated compliance records, ensuring decisions are based on objective, tamper-proof data. Traditional review platforms like Trustpilot and G2, built around subjective human experiences and star ratings, will also become increasingly obsolete.

The emerging alternative is a new trust infrastructure built on quantifiable, machine-readable metrics. Instead of relying on potentially AI-generated reviews, a problem that has already undermined traditional review systems, agents could assess services using benchmarks like delivery time reliability, system uptime, or refund processing speed—measurable metrics that ensure objective evaluations rather than subjective human reviews. This could involve decentralized reputation networks where trust is established through cryptographically verified interaction histories and smart contract execution records. Such systems would offer objective assessments of service quality, enabling agents to make informed decisions without relying on potentially biased or manipulated human reviews.

Moreover, the feedback loop between consumers and businesses will evolve dramatically. Instead of sending generic emails requesting reviews—a method often resulting in low response rates—commerce websites can engage directly with your AI agent to collect timely feedback about specific topics like shipping or product quality. They might offer incentives like future store credit to encourage participation. The human user could provide a brief impression, such as "The cordless vacuum cleaner works well, but the battery life is short." The agent then takes this input, contextualizes it with additional product data, and generates a comprehensive review that highlights key features and areas for improvement. This process not only saves time for the user but also provides businesses with richer, more actionable insights.

Trustpilot and G2 could pivot by introducing agent-oriented verification systems, such as machine-readable trust scores derived from operational metrics like service accuracy, delivery consistency, and customer support responsiveness, enabling agents to evaluate businesses programmatically.

The new data-sharing economy

Information sharing in the age of AI agents demands a fundamental reinvention of the current consent and data access model. Rather than blunt instruments like cookie banners and privacy policies, websites will implement structured data requirement protocols—machine-readable manifests that explicitly declare what information is needed and why.

This granular control would operate at multiple levels of specificity. For example, an agent could share your shirt size (L) with a retailer while withholding your exact measurements. It might grant 24-hour access to your travel dates but permanent access to your seating preferences. When a service requests location data, your agent could share your city for shipping purposes but withhold your exact address until purchase confirmation. These permissions wouldn't be just binary yes/no choices—they could include sophisticated rules like "share my phone number only during business hours" or "allow access to purchase history solely for personalization, not marketing." Such granular controls, impossible to manage manually at scale, become feasible when delegated to AI agents operating under precise constraints.

AI agents would also act as sophisticated information gatekeepers, maintaining encrypted personal data vaults and negotiating data access in real time.

These mechanisms will fundamentally shift the balance of power in data-sharing dynamics. GDPR-like frameworks may evolve to include provisions for dynamic, agent-mediated consent, allowing for more granular data-sharing agreements tailored to specific tasks. Websites might implement real-time negotiation protocols, where agents can evaluate and respond to data requests based on their principal's preferences, preserving privacy while optimizing functionality.

New attack vectors

The shift to agent-mediated interaction introduces novel security challenges. Agent impersonation and jailbreaking agents are two examples.

Jailbreaking AI agents pose significant risks, as manipulated agents could act outside their intended scope, leading to unintended purchases or other errors. Techniques like instruction-tuning poisoning or adversarial suffix manipulation could alter an agent’s behavior during critical tasks. For example, adversarial instructions embedded in websites’ HTML might influence an agent’s purchasing logic, bypassing its human-defined constraints. Robust safeguards and continuous monitoring will be essential to prevent these vulnerabilities.

Agent impersonation adds a complex layer to cybersecurity challenges. Malicious actors could spoof an agent's credentials to access sensitive data or execute fraudulent transactions. Addressing this threat demands robust multi-layered verification protocols, such as cryptographic identity verification paired with continuous behavioral monitoring, to ensure authenticity and safeguard sensitive interactions.

Building the new web - opportunities for founders

The web’s agent-first future has no established playbook—and that’s exactly where founders thrive. Entirely new product categories are waiting to be defined: agent-to-agent compliance dashboards, cryptographic attestation services that replace outdated CAPTCHAs, and dynamic data-sharing frameworks that make “privacy by design” a reality. Platforms that offer standardized “agent passports,” identity brokerages that verify delegation rights, agent-native payment gateways, and trust ecosystems driven by machine-readable performance metrics—each of these represents a greenfield opportunity to set the standards of tomorrow’s internet. Startups anticipating these shifts can position themselves as foundational players in an agent-driven economy, opening new channels of value creation and establishing a competitive edge before the rest of the market catches up.

Some concrete areas include:

• Trustpilot for agents - creating machine-readable trust metrics and reputation systems that help agents evaluate services and vendors

• Okta for AI agents - building the identity and authentication layer that manages agent credentials, permissions, and delegation chains

• OneTrust for agents - creating the new standard for privacy preference management, turning today's basic cookie banners into sophisticated data-sharing frameworks where agents can negotiate and manage granular permissions across thousands of services

• Cloudflare for agent traffic - developing intelligent rate-limiting and traffic management systems designed for agent-scale operations

• LastPass for agent permissions - building secure vaults that manage agent credentials and access rights across services

• AWS CloudFront for agent data - creating CDN-like infrastructure optimized for agent-readable formats and rapid agent-to-agent communication

• McAfee security for agents - developing security platforms that protect against agent impersonation and novel attack vectors

Go build.

This is the second essay in a five-part series exploring the future of AI agents and their impact on the internet. In Part 3, we'll move from infrastructure to commerce, exploring critical questions that emerge in an agent-first economy: How will payment systems evolve to support autonomous purchasing decisions? What happens to marketing and customer engagement when AI agents become the primary decision-makers? And perhaps most intriguingly, how will businesses adapt their revenue models when agents can negotiate prices and terms at superhuman speed and scale? Subscribe and follow as we continue to examine the technical, social, and economic implications of this transformation.

Welcome to a new post in the AI Agents Series - helping AI developers and researchers deploy and make sense of the next step in AI.

My last post explored how the internet will trans

form for an agent-first future - from websites optimizing for AI interaction through "agent-responsive design" to the emergence of Agent Engine Optimization (AEO) as the next SEO. We saw how tech giants like Google, Apple, OpenAI, and Anthropic are racing to define this next evolution of digital interaction, with Gartner projecting that by 2028, 33% of enterprise software applications will include agentic AI.

In this post, I'll outline a curated, though non-exhaustive, overview of the open-source ecosystem for developers creating these AI agents. While numerous market maps exist for AI agents, they often cater more to venture capitalists than builders. Developers need actionable tools and frameworks to launch functional AI agents today.

Which tools do other builders rely on to develop voice agents? What’s the leading open model for document understanding? With new packages emerging almost daily, I’ll focus solely on the libraries I’ve personally found most effective. This list is, therefore, intentionally selective rather than exhaustive.

Every package included here supports commercial use and has a permissive open-source license.

With the holiday season coming, there's no better time to dive into these tools and start building.

Categories covered in this piece:
→ Frameworks for Building and Orchestrating Agents
→ Computer and Browser Use
→ Voice
→ Document Understanding
→ Memory
→ Testing and Evaluation
→ Monitoring and Observability
→ Simulation
→ Vertical Agents

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Frameworks for Building and Orchestrating Agents

Building AI agents requires robust frameworks that can handle complex workflows, memory management, and tool integration. These foundational frameworks serve as the backbone for creating agents that can understand, plan, and execute tasks autonomously.

• CrewAI - a framework for orchestrating role-playing, autonomous AI agents

• Phidata - build AI assistants with memory, knowledge, and tools

• Camel - build customized multi-agent systems to generate data, complete tasks, or simulate real-world interactions

• AutoGPT - create, deploy, and manage continuous AI agents that automate complex workflows

• AutoGen - develop LLM applications using multiple agents that can converse with each other

• SuperAGI - build, manage, and run autonomous AI agents quickly and reliably

• Superagent - an open framework for building AI assistants

• LangChain & LlamaIndex - the usual suspects, facilitating AI Agents through composability

Computer and Browser Use

For AI agents to be truly useful, they need to interact with computers and browsers just like humans do. These tools enable agents to navigate websites, control applications, and execute commands programmatically, bridging the gap between AI reasoning and real-world actions.

• Open Interpreter - turn natural language commands into code that runs on your local machine

• Self-Operating Computer - enables multimodal models to operate a computer

• Agent-S - an open agentic framework that uses computers like a human

• LaVague - create web agents that take actions on websites using LLMs as their reasoning engines

• Playwright - a framework for web testing and automation

• Puppeteer - a JavaScript library that provides a high-level API to control Chrome or Firefox

Voice

Voice interfaces represent the most natural way for humans to interact with AI agents. These tools enable the creation of agents that can understand spoken language, maintain context in conversations, and respond with natural-sounding speech, making AI interaction more accessible and intuitive.

Speech2speech

• Ultravox - a speech2speech model for real-time voice interaction, superior to Moshi for now

• Moshi - a speech2speech model for real-time voice interaction

• Pipecat - a framework for voice and multimodal conversational AI, supporting speech2text, text2speech, video, etc.

Speech2text

• Whisper - OpenAI's speech2text model

• Stable-ts - a lightweight Whisper wrapper with timestamps and more

• Speaker diarization 3.1 - pyannote’s flagship model for speaker detection

Text2speech

The only decent open model I came across was ChatTTS, which is satisfactory for production. I, therefore, default to ElevenLabs or Cartesia.

Misc

• Vocode - a toolkit for building voice-based LLM agents

• Voice Lab - a comprehensive testing and evaluation framework for voice agents across language models, prompts, and agent personas

Document Understanding

Modern AI agents need to process and understand documents in various formats, from PDFs to images with text. These tools provide the crucial ability to extract, comprehend, and act on information from unstructured documents, enabling agents to handle real-world business processes.

• Qwen2-VL - vision language model from Alibaba outperforming GPT-4o and Claude 3.5 Sonnet

• DocOwl2 - an efficient multimodal LLM for OCR-free document understanding

Memory

Without memory, AI agents are limited to single-turn interactions. These memory tools enable agents to maintain context over long conversations, remember user preferences, and learn from past interactions, making them truly personal assistants rather than just query responders.

• Mem0 - provides an efficient, self-improving memory layer for LLMs, enabling personalized AI experiences

• Letta (fka MemGPT) - create LLM agents with long-term memory and custom tools

• LangChain - offers memory components to manage conversation history and context

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Testing and Evaluation

As AI agents become more complex, robust testing becomes critical. These tools help developers evaluate agent performance, identify failure modes, and ensure reliability across different scenarios and environments.

• Voice Lab - a comprehensive testing and evaluation framework for voice agents

• AgentOps - tools for monitoring and benchmarking agent performance

• AgentBench - a benchmark to evaluate LLMs as agents across various environments (Web, Minecraft, Visual Design, etc.)

Monitoring and Observability

Understanding how AI agents perform in production is crucial for maintaining reliability and optimizing costs. These tools provide insights into agent behavior, resource usage, and performance metrics essential for running agents at scale.

• openllmetry - an open-source, OpenTelemetry-based end-to-end observability tool for LLM applications

• AgentOps - agent monitoring, LLM cost tracking, benchmarking, and more

Simulation

Before deploying agents to real-world scenarios, testing them in controlled environments is crucial. These simulation tools allow developers to validate agent behavior, test edge cases, and refine decision-making capabilities in safe, reproducible environments.

• AgentVerse - facilitates the deployment of multiple LLM-based agents in various applications, including simulations

• Tau-Bench - a benchmark and testing code for agent-user interactions in real-world domains like retail and airline

• ChatArena - multi-agent language game environments for research on autonomous LLM agents

• AI Town - A virtual town where AI characters live, chat, and socialize

• Generative Agents - Stanford’s Interactive simulacra of human behavior

Vertical Agents

There are dozens of open vertical agents out there, so here are just a few select ones I’ve tinkered with and found the most useful:

• OpenHands (Coding) - a platform for software development agents powered by AI

• aider (Coding) - pair programming in your terminal

• GPT Engineer (Low code) - build applications using natural language. Specify what you want to build, and the AI will ask for clarification before building it.

• screenshot-to-code - convert screenshots into a functioning website using HTML/Tailwind/React/Vue

• GPT Researcher (Research) - an autonomous agent that performs comprehensive research on any given topic

• Vanna (SQL) - chat with your SQL database

Looking Ahead

While this post focused on open-source packages with permissive licenses, I plan to publish another comprehensive list specifically for engineers building voice agents. This upcoming guide will include both open-source and commercial tools, covering solutions like OpenAI's Realtime API (speech2speech) and ElevenLabs (text2speech), along with detailed comparisons of their capabilities, pricing models, and ideal use cases.

Stay tuned for more deep dives in the AI Agents Series.

Comprehensive list of open-source packages for AI engineers (last update: Aug ‘23)

A NotebookLM-powered podcast episode discussing this post:

November 2028. Maya's personal AI agent quietly handles her holiday shopping, easily navigating dozens of e-commerce sites. Unlike the clunky chatbots of 2024, her agent seamlessly parses product specifications, compares prices, and makes purchase decisions based on her preferences. "The boots for your sister," it explains, "are from that sustainable brand you both discussed last month - I found them at 20% off and confirmed they'll arrive before your family gathering." What would have taken Maya hours of manual searching now happens automatically, thanks to a web rebuilt for agent-first interaction.

—> The future, three years from now.

As we approach the end of 2024, a new paradigm shift is emerging in how we build and interact with the internet. With rapid advances in AI reasoning capabilities, tech giants and innovative startups alike are racing to define the next evolution of digital interaction: AI agents. Google, Apple, OpenAI, and Anthropic have all declared AI agents as their primary focus for 2025. This transformation promises to be as significant as the web and mobile revolutions were and represents perhaps the most natural interface for LLM-powered technology—far more intuitive and capable than the chatbots that preceded it.

In the recent No Priors Podcast, Nvidia’s CEO Jensen Huang stated that "there's no question we're gonna have AI employees of all kinds” that would "augment every single job in the company”.

Moreover, Gartner projects that by 2028, 33% of enterprise software applications will include agentic AI, up from less than 1% today, enabling 15% of day-to-day work decisions to be made autonomously. This rapid adoption mirrors the mobile revolution of the early 2010s but with potentially more far-reaching implications for how we interact with digital services.

Anthropic’s Claude submits a vendor request autonomously using the recently announced Computer Use capability

This essay, the first in a five-part series, explores how web and mobile will evolve to accommodate an agent-first future and how businesses can prepare for this seismic shift.

AI Agents

While there's ongoing debate about what an AI Agent is, at its core, what sets agents apart from traditional software is their ability to autonomously plan and adapt. Unlike rule-based systems that follow predetermined paths, agents can formulate strategies, execute them, and—most importantly—adjust their approach based on outcomes and changing circumstances. Think of them as digital assistants that don't just follow a script, but actually reason about the best way to achieve your goals. If a planned action fails or yields unexpected results, an agent can reassess and chart a new course, much like a human would. This flexibility and autonomous decision-making capability marks a departure from traditional software, which can only respond in pre-programmed ways.

Central to agents' capabilities is their sophisticated use of tools. Much like a handyman who knows when to use a screwdriver versus a hammer, agents must determine which tools to use, when to use them, and how to use them effectively. For instance, when helping you plan a trip, an agent might first use a calendar tool to check your availability, then a flight search API to find options, and finally a weather service to ensure you pack appropriately. The key isn't just having access to these tools—it's the agent's ability to reason about their use and orchestrate them intelligently to accomplish complex tasks.

From mobile-first to agent-first

Remember when 'www' stood for something closer to 'Wild Wild West' than 'World Wide Web'? The early 2000s internet was an untamed digital frontier, where users navigated through a maze of pop-ups, fought off malware, and relied on bookmarked URLs just to find their way around.

The early 2010s, when mobile exploded, weren’t that different as businesses scrambled to make their websites mobile-responsive. That shift wasn't just about resizing content for smaller screens–it fundamentally changed how we approached web design, user experience, and digital strategy. It created a whole new field of website and mobile optimization: choosing the best colors and text copy to increase traffic, conversion rates, and stickiness. 

Today, we stand at a similar inflection point with AI agents.

Just as mobile-responsive design emerged from the need to serve smartphone users better, "agent-responsive design" is emerging as websites adapt to serve AI agents. But unlike the mobile revolution, which was about accommodating human users on different devices, the agent revolution requires us to rethink our fundamental assumptions about who – or what – is consuming our digital content.

In this agent-first era, websites will undergo a dramatic transformation. Gone are the days of flashy advertisements, elaborate typography, and resource-heavy images—elements that consume bandwidth but provide little value to AI agents. Instead, we're moving toward streamlined, efficient interfaces that prioritize function over form. These new websites will feature minimalist designs optimized for machine parsing, structured data layers that enable rapid information extraction, standardized interaction patterns that reduce processing overhead, and resource-efficient components that minimize token usage and computation costs.

This evolution extends beyond traditional websites. Mobile applications are already being reimagined with agent-interaction layers, as evidenced by recent novel methods like Apple's Ferret-UI 2 and CAMPHOR, enabling seamless agent navigation of mobile interfaces while maintaining human usability. Google and Microsoft also invest in this space, as demonstrated in their recent papers AndroidWorld and WindowsAgentArena, respectively. Both are fully functional environments for developers to build and test agents.

The incentives are becoming clear: optimize for agents, and you'll unlock new channels of engagement and commerce. Ignore them, and you risk becoming invisible in the emerging agent-first internet.

Blueprint of Agent-Responsive Design

At its core, agent-responsive design represents a radical departure from traditional web design principles. Instead of optimizing for human visual perception and engagement, websites must provide clear, structured interfaces that agents can efficiently navigate and interact with.

This transformation will likely unfold in two phases:

Phase 1: Hybrid optimization

Initially, websites will maintain dual interfaces: one optimized for human users and a "shadow" version optimized for agents. This agent-optimized version will feature:

• Enhanced semantic markup with clear structure and purpose

• Unobfuscated HTML that welcomes rather than blocks automated interaction

• Well-defined aria-label labels and metadata to help agents choose and interact with the right UI components

• Direct access to knowledge bases and documentation by exposing information beyond what’s visible on the “website interface”, giving the querying agents access to their RAG to easily retrieve information such as refund policy or answer questions the agent has based on their help docs. Also, after being authenticated, providing easy access to user-related information such as last purchases or stored payment methods.

• Streamlined authentication and authorization protocols

Phase 2: API-first architecture

The second phase will move beyond traditional UI components, focusing on exposing clean, well-documented APIs that agents can directly interact with. Consumer websites like Amazon, TurboTax, and Chase will:

• Provide clear documentation of available tools and capabilities. The agent will leverage its reasoning engine and the task the human delegated to plan the tools and sequence that it needs to use.

• Offer structured workflows with explicit input/output specifications

• Enable direct access to business logic and user data

• Support sophisticated authentication mechanisms for agent-based interactions

The death of traditional A/B testing

In an agent-first world, the traditional approach to A/B testing becomes obsolete. Instead of testing different button colors or copy variations for human users, companies like Amazon will need to optimize for agent interaction efficiency and task completion rates.

These A/B tests will target similar metrics as today: purchases, sign-ups, etc., employing LLMs to generate and test thousands of agent personas without the need for lengthy user testing cycles.

This new paradigm of testing will require new success metrics such as:

• Model compatibility across different AI providers (GPT, Claude, etc.) - each language model has its own nuances. Optiziming can help businesses squeeze a few more percentage points for conversion, bounce rate, etc.

• Task completion rate for the human-delegated task at hand, like purchasing a product or subscribing to a newsletter

• Token efficiency and latency optimization, enabling lightning-fast interactions while minimizing computational overhead and associated costs

• Authentication and security protocol effectiveness, ensuring robust protection while maintaining frictionless agent operations

The competitive landscape in this new era will be shaped significantly by model providers' unique advantages. Companies like OpenAI and Google, with their vast user interaction data, will possess an inherent edge in creating agents that deeply understand user preferences and behaviors. However, this also creates an opportunity for innovation in the form of universal memory and context layers, like what mem0 is pitching with their recently released Chrome extension—systems that can bridge different models, devices, and platforms to create a cohesive user experience.

Drawing from Sierra's τ-bench research, we can anticipate the emergence of standardized benchmarks for measuring agent-readiness across verticals and task types, similar to how we currently measure mobile responsiveness or page load times.

New discovery protocol - Agent Engine Optimization (AEO)

Just as websites evolved from manually curated directories to sophisticated search engine optimization, the agent era demands a new discovery mechanism. The question isn't just about findability—it's about actionability: how do agents identify and interact with the most relevant and capable digital services?

In 2005, Google introduced the Sitemap protocol to improve search engine crawling efficiency, enable discovery of hidden content, and provide webmasters with a standardized method for communicating site structure and content updates to search engines. What is the Sitemap equivalent for AI agents?

Just as SEO emerged to help websites become discoverable in search engines with Google’s inaugural PageRank algorithm, Agent Engine Optimization (AEO) will become crucial for visibility in an agent-first web. Back in Aug 2023, I called it Language Model Ranking Optimization.

This new protocol will go beyond traditional sitemaps, providing agents with structured information about websites:

• Available services and capabilities like signing up, placing an order, booking a flight seat

• Authentication requirements - what actions require authentication

• Data schemas and API endpoints - what data does each action/endpoint need? What is mandatory vs. optional?

• Privacy and security protocols - how information is being stored 

• Service level agreements like refund and shipping guidelines and data retention policy

Exposing such information will become a standard feature in website builders like Shopify and Wix, much like mobile responsiveness is today. These platforms will automatically generate and maintain agent-interaction layers, democratizing access to the agent-first economy for businesses of all sizes.

Companies will need to optimize not just for search engines but for an emerging ecosystem of agent directories and registries that help autonomous agents discover and interact with digital services.

More on the PageRank for AI Agents will be explored in the next post of this series.

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like OpenAI's DevDay and autonomous agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

The web's great reversal - from defense to embrace

This transformation marks a seismic shift in how the internet treats automated interactions. We're moving from an era of aggressive bot blocking and CAPTCHA barriers to one where automated agents are not just tolerated but actively embraced and optimized for. This shift necessitates new quality assurance practices, with websites being tested and validated for agent-readiness just as rigorously as they're tested for human usability today. The implications are profound: businesses that master agent-responsive design will gain a competitive edge in an increasingly automated digital economy.

2025: Year of the Agent

As we enter 2025, the pieces are falling into place for a fundamental shift in how we interact with the internet. Google's Jarvis, Anthropic's Computer Use capabilities, Apple Intelligence, and OpenAI's recently announced Operator all point to a future where autonomous agents become the primary interface between humans and digital services.

This transformation will require businesses to rethink their digital presence from the ground up. Just as the mobile revolution forced companies to adopt responsive design principles, the agent revolution will require the adoption of agent-responsive design patterns and practices.

In the next essay in this series, we'll explore the components of such a new ecosystem, examining how authentication, payments, and reputation systems must evolve to support this new paradigm.

This is the first essay in a five-part series exploring the future of AI agents and their impact on the internet. Follow along as we examine this transformation's technical, social, and economic implications.

Last February, Stanford published a paper that sparked everyone’s imagination. In this paper, the researchers leveraged ChatGPT to power human-like agents. A mini-simulation of humanity.

A few weeks later, two viral open-source autonomous agents frameworks were introduced to the world - AutoGPT and BabyAGI. Their goal? Build fully autonomous AI agents. Those libraries generated a lot of excitement, but have yet to spawn useful agent-powered applications.

An autonomous agent, described simply, is an AI program capable of planning and executing tasks based on a given objective. Imagine asking an AI to “book a flight”, or “create a website for people interested in renting their apartments when they’re away” - and the agent goes to work. It does so by repeatedly asking itself “What should be the next steps to achieve the task at hand”, utilizing an LLM to answer this question.

Such objectives are great demo material, but demos are often cherry-picked, and the recent consensus is that AI agents are a great concept, but that’s about it. To me, it felt like agents were becoming the new Crypto - a lot of hype, with no one able to point to a concrete widespread use case.

That might have changed with OpenAI’s recent release of GPTs. GPTs are small ChatGPT wrappers that have the prompt, context, and tools baked into them, ready to roll.

Though powerful, they have yet to earn the “autonomous agents” badge–they still require a high degree of human feedback to achieve their goal. They are, however, capable of connecting to other services and APIs such as your calendar and email.

Combined with a few other headwinds, GPTs might represent the milestone the space has been waiting for, leading to a Cambrian explosion of capable autonomous agents.

Bonus at the end of the post: a list of open-source repositories every agents builder should know and the startups leading the way.

Unlocking autonomous agents

AutoGPT, the largest autonomous agents framework to date, has over 150k stars on GitHub and 50k builders in its Discord server. With such a large community of developers, one might wonder why useful agents have yet to emerge. The answer lies in the inherent limitations of agents, which are slowly being addressed.

Limitation #1 - Small context windows

Read more