This post is a collaboration with Ilia Karelin, who writes Prosper, a newsletter sharing tactical AI workflows with frameworks you can plug into your own setup. What drew me to Ilia’s work is that he goes deep into areas most people only skim the surface of, explaining them in layman’s terms that simplify things for both tech and non-tech folks. His complete setup guide for Claude Chat, Cowork, and Code is one of the most thorough breakdowns I’ve seen. With Cowork gaining traction, I’ve noticed a gap between people who installed it and people who know how to make it useful. Ilia bridges that gap better than anyone I’ve read, so I was glad he was open to writing this together. Subscribe to Prosper here:

Prosper

Every week, practical AI workflows, prompts, and guides you can put to use the same day - no matter your technical background.

By Ilia Karelin

Cowork is Anthropic’s answer to non-technical folks seeking the benefits of Claude Code - an agent inside the Claude Desktop app that reads your files, connects to your tools, and executes complex tasks end-to-end without you writing a single line of code. If Claude Code is built for engineers, Cowork is built for everyone who wants the same capabilities without having to open a terminal.

You point it at a folder and describe what you need in plain language: organize 200 scattered files by type and project, turn receipts into a formatted expense report, batch-rename months of screenshots with consistent dates, or synthesize five research PDFs into a single brief with citations.

Since January, I’ve been running my newsletter production inside Cowork: idea triage, outline generation, draft editing, and automations that process my raw notes every Monday morning without me touching them.

The difference isn’t my prompting skill, but three useful Cowork features: connectors, scheduled tasks, and plugins. This post covers all three with concrete workflows you can reuse.

Connectors: Cowork reads and writes to your actual tools

Connectors let Cowork plug directly into the tools you already use: Gmail, Notion, Slack, and Google Drive. and work with your actual data inside a session.

In February, Anthropic expanded connectors beyond project management tools to include popular everyday apps such as Gmail, Google Calendar, Notion, and more.

The connector I use most is Notion. It handles nested pages well, which matters for any workspace with real depth. The brain dump workflow, where raw ideas I capture on my phone get automatically triaged into structured newsletter briefs every Monday, is the next section that runs entirely through it.

Gmail is the other app worth connecting to. Even without any scheduled tasks, having Cowork read your emails during a Cowork session, rather than you pasting them in, reduces friction enough that you’ll actually use it for email-related work, such as drafting emails in your style or taking bulk actions like archiving all emails from a specific sender.

The real payoff comes when you combine a connector with a scheduled task. That’s where Cowork stops being something you open and starts being something that runs in the background.

Scheduled tasks and automated workflows

In Cowork, each piece of work you hand off to Claude is called a task.

To set one up, type /schedule in any Cowork session and pick a frequency: daily, weekly, monthly, weekdays only, or on demand. That’s useful for anything that needs to happen regularly without you having to start it each time.

How to set it up

Two paths: type `/schedule` in any session, or use the “Scheduled” tab in the sidebar to create tasks directly. You set a name, a prompt, a frequency, and which folder to work from.

Each scheduled task runs as its own full Cowork session with access to all your connectors, plugins, and folder context.

Note that your laptop must be awake for scheduled tasks to run, as they run locally. Anthropic recently added a feature (see the red arrow below) to prevent your computer from going to sleep, so tasks can continue running.

A workflow I actually use

I have a “Quick Brain Dump” widget on my phone. Whenever an idea hits, I write it into a Notion page - a text block of raw and unrefined stream of thought. Those notes will remain on this page until I manually review and organize them. That is, until I plugged in Cowork.

To make that page usable in Cowork, I first connected my Notion workspace on my phone.

Once I connected my Notion workspace to Cowork, I set up a scheduled task that runs every Monday morning.

1. Reads everything added to the brain dump page that week

2. Decides which entries have real newsletter potential

3. Turns the strong ones into a working title, an angle, and 3 questions I’d need to answer to write it

4. Saves them to my Newsletter Ideas database as “Needs Review”

I capture ideas throughout the week. When I sit down on Monday, the triage is already done.

Here’s the exact prompt running inside that task (ok to steal):

You are a newsletter research assistant. Connect to my Notion workspace 
using the Notion connector.

1. Read all entries added to my “Quick Brain Dump” page this week

2. For each entry, decide: does this have real potential to become a
newsletter post? (yes/no + one sentence why)

3. Only for strong “yes” entries: generate a working title, the angle
that makes it interesting, and 3 questions I’d need to answer to
write it

4. Skip entries that look like tasks, reminders, or personal notes -
only process ideas

5. Save all results to my “Newsletter Ideas” Notion database with
Status = “Needs Review” and Source = “Brain Dump [date]”
If an entry is too vague to evaluate, add it to the database with
Status = “Unclear” and a note on what’s missing.

Here are the results in my Notion workspace:

The same pattern works for anything where raw input is collected in one place and needs to be structured: Slack threads into action items, Gmail into expense logs, Linear tickets into weekly reports.

Cowork is like a back-office assistant that once taught, does the work for you on a recurring cadence.

Chaining scheduled tasks

The output of one scheduled task can feed into the next one’s input.

The brain dump task runs early Monday and saves structured briefs to a Notion database. A second scheduled task runs after that, reads those briefs, picks the strongest one based on recency and audience fit, and drops a full working outline into my local drafts folder.

Task #1 processes raw input. Task #2 acts on structured output. No human intervention needed.

Each plugin is a full toolkit for one domain

There are dozens of plugins available for Cowork, each one a specialist kit for a specific domain. A plugin bundles skills, connectors, slash commands, and subagents around a job function like Sales, Data, or Design, so you don’t have to wire them up yourself.

Install one and you immediately get domain-specific commands and integrations tailored to that role’s actual workflows.

Here’s what the Data plugin contains, for example:

Composable Plugins

You can install multiple plugins and commands inside of those plugins, and pull from all of them in a single task with just a prompt. Pretty incredible.

Here’s a real example. I dropped a messy sales CSV into Cowork: missing values, a duplicate row, a negative amount (these numbers are not real), and ran this single prompt with both the Data and Design plugins installed:

I have a sales CSV file (sample_sales.csv) and the Prosper company logo 
(image.jpeg) in this folder.

Step 1 - Data plugin:

/analyze the CSV and find:

- Total revenue by product (excluding refunded transactions)
- Month-over-month revenue trend across January, February, March
- Revenue split by region (North America, Europe, Asia Pacific)
- Any data quality issues worth flagging (nulls, anomalies, duplicates)

Step 2 - Data plugin:

/write-query to calculate the refund rate by product - which product
has the highest percentage of refunded transactions?

Step 3 — Design plugin:

/performance-report using the findings above. Format it as a clean one-page
Q1 sales report for Prosper. 
Include:

- The Prosper logo at the top
- A summary section with 3 headline numbers (total revenue, top product,
top region)
- A data quality callout box flagging the issues found in Step 1
- A refund risk section showing which product needs attention
- Brand colors: blue (#4B4EE4), white background, clean sans-serif font

Save the output as prosper_q1_report.pdf

Step #1 used the Data plugin to analyze revenue by product and region, flag the data quality issues, and calculate month-over-month trends. Step #2 wrote the SQL to calculate refund rates by product, and Step #3 switched to the Design plugin and formatted everything into a branded one-page report, logo included.

One prompt. Three steps. The output went straight from a raw CSV to something you could send to a stakeholder.

Where to start

Today: Install Gmail or Slack connectors. Connect whichever tool you actually use to collect raw inputs. Notion, email, whatever it is.

This week: Set up one scheduled task. Pick something you do manually on a recurring basis: drafting a sales update, triaging weekly tasks from Jira/Linear, summarizing newsletters, and let Cowork run it instead. Use the brain dump prompt above as a template, or use Claude to generate a prompt for you by describing the task you want it to automate

This month: Install two or three plugins that are relevant for your tasks, such as Data Analysis or Designer. Then run a task that pulls data from multiple plugins in a single session. Once one scheduled task is working, consider chaining it as input to another task, e.g., a task to draft a report with another to send it via Slack and email.

The goal isn’t to use every feature. It’s to find one workflow that currently requires your manual attention and hand it off. Once that one runs on its own, the next one is easier to see.

That’s how Cowork becomes a system instead of a chatbot.

Recent related posts

I’m excited to host Jeff Morhous for this edition. Jeff is a senior software engineer and teacher at heart who spends his working hours solving software problems with AI tools, then dedicates the rest of his time teaching other developers to do the same through his newsletter, The AI-Augmented Engineer. I’ve been following his newsletter, The AI-Augmented Engineer, and I keep coming back to how clearly he breaks down AI coding concepts—whether he’s explaining what vibe coding actually is, showing how he vibe-coded an iPhone app he now uses every day, or comparing Codex CLI to Claude Code. His practical approach to helping developers ship faster with AI made him a natural fit for this series. I highly recommend subscribing to his Substack:

The AI-Augmented Engineer

Learn how to use AI tools to write better code, ship faster, and stay ahead of the curve

By Jeff Morhous

Today, Jeff is diving deep into Claude Code’s most powerful features: subagents, skills, and context files. Over to Jeff!

If you’re following AI buzz on the internet, you’ve probably already heard of Claude Code. Whether you’re a professional developer using it for work or a hobbyist using it for non-code agentic workflows, Claude Code has something for everyone, as evidenced by its ~5M weekly downloads.

While it’s simple to get started with the tool, getting the most out of it takes some extra effort as the Claude team has been rapidly launching new features over the last few months. In this post, I’ll walk you through the releases I found most impactful when collaborating with Claude Code:

• Subagents

• Skills

• Context files

Let’s start with subagents!

Understanding subagents in Claude Code

Subagents are custom AI assistants that specialize in particular tasks or domains. They have their own configuration and context, enabling them to tackle subtasks without cluttering the main conversation. When a problem matches a subagent’s expertise, Claude can delegate that task to the subagent, which operates independently and returns the result.

For instance, my code reviewer subagent (more on that later) is invoked when I request a code review, while my Terraform subagent is invoked when I make infrastructure changes. It’s like having a site reliability engineer on staff.

Why you should use subagents

Each subagent has its own context window, separate from the main session. This isolation preserves the main conversation’s focus while the subagent works on a specific problem. It is well established that you get better and more accurate responses when your coding agent’s context window is focused. Mixing a request to write an API endpoint with a later request to refactor some CSS is likely to yield worse results than keeping those conversations separate.

Subagents come with a custom system prompt and can be granted a limited set of tools, making them specialists in their particular problem set.

The benefits of subagents are many, but can be summed up as:

• Context preservation (don’t pollute your main context)

• Specialized expertise (can give specific and detailed instructions)

• Reusability (Can be used across projects or checked into Git)

• Controlled tool access (like shell commands)

Creating subagents

Subagents are defined in Markdown files, typically suffixed with .md. You can create them through Claude Code’s interactive interface (/agents command) or by creating files manually: Project-specific subagents live under the project’s .claude/agents/ directory, while user-wide subagents live under ~/.claude/agents/. If a subagent name exists in both places, the project version takes precedence.

Each subagent file begins with a YAML header defining a unique name and a description of its purpose. You can also specify the tools the subagent can use and the language model powering that agent.

The rest of the file is the agent’s system prompt. This prompt should instruct the subagent on how to perform its role. It may include step-by-step guidelines, checklists, or example behaviors. For instance, a “code-reviewer” subagent might be defined as follows:

---

name: code-reviewer

description: Expert code review specialist. Proactively reviews code for quality, security, and maintainability. Use immediately after writing or modifying code.

tools: Read, Grep, Glob, Bash

model: inherit

---

You are a senior code reviewer ensuring high standards of code quality and security.

When invoked:

1. Run git diff to see recent changes

2. Focus on modified files

3. Begin review immediately

Review checklist:

- Code is simple and readable

- Functions and variables are well-named

- No duplicated code

... (etc)

Claude Code offers a handy interactive menu via the /agents command to manage subagents. Running /agents in the Claude terminal opens a UI where you can view all available subagents (built-in, user-level, project-level), create a new one with a guided wizard, edit their configuration (including tool permissions), or delete them. This is the recommended way to create/edit subagents, as it ensures proper syntax and lets you easily pick from available tools.

Subagent files can also be managed manually. For example, to create a new project subagent, you’d create a Markdown file under .claude/agents/. The YAML frontmatter and content can be written by hand or generated by Claude.

You can find implemented subagents for inspiration in this repository.

Different ways to invoke subagents

Now that you know how to make subagents, you’re probably curious about how to actually use them.

The first and easiest is to explicitly invoke a subagent. Do this by explicitly calling your subagent as follows:

> Use the spec-request-spec agent to write request specs for the UserController

The second way is to rely on Claude to pick an appropriate subagent, similar to how tools are being invoked based on tools’ descriptions. It will do this by matching your prompt/query against the descriptions of available subagents. Similar to tool use, good agent descriptions increase the odds of proper automatic invocation.

Behind the scenes, when a subagent is invoked, Claude starts a fresh context for that subagent using its system prompt. The subagent may then use tools (as allowed), produce an answer, and perform actions. Once done, the conversation with the subagent ends, and Claude returns to the main session, freeing that context window’s tokens. Because subagents start with a clean context each time, they might introduce some latency as they gather any needed information (for example, re-reading relevant files) to carry out their task. The key benefit is that your main session’s context remains free of that transient detail, enabling longer, cheaper, and faster overall interactions.

Best practices for using subagents

1. Scoped - design narrow, focused subagents. Scoped agents are more predictable and performant.

2. Detailed - be as specific as possible about the agent’s procedure, style, and constraints in the subagent’s Markdown content,

3. Tracked - store subagent files (especially project-level ones) in your repository so that your team can use them and improve them over time.

4. Assisted - use Claude’s /agents CLI workflow to have Claude draft a subagent’s initial version

By thoughtfully creating subagents and following these practices, you can offload many specialized tasks from the main Claude Code context, leading to cleaner interactions and more effective AI assistance in your development workflow.

My most used subagent is a code reviewer. I explicitly spin off this subagent so I can get feedback on the changes in my branch before sending it to my peers. It’s saved me from tons of “nit” comments or embarrassing typos.

✨ AI Tidbits has partnered with Anthropic to provide premium members with $500 in Claude API credits, plus an additional $800 in credits across other leading AI tools and APIs including Hugging Face, Deepgram, and Vapi. Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Defining agent skills

While subagents are like specialized personas, Agent Skills are more granular capabilities that Claude can draw upon. Skills package up instructions, examples, and even code or data that help Claude perform specific tasks on demand.

If there’s something that is in your CLAUDE.md that is only relevant to a small fraction of queries, turn that instruction into a skill. If there’s something in your day-to-day prompts that could benefit from a super clean context or has a tendency to fill up your main context, put it in a subagent.

An agent skill is a folder on your filesystem that contains a SKILL.md file (with instructions and metadata) and optional supporting files (scripts, reference docs, templates, etc.).

They allow you to extend Claude’s functionality in a modular way. For example, you might create a skill for “SQL troubleshooting” with instructions and scripts for database queries. Once the skill is set up, you no longer need to prompt Claude with those instructions every time. Claude will autonomously load the skill when needed. This reduces repetitive prompting and can combine multiple skills to handle complex tasks.

Every skill resides in its own directory named after the skill, under either your user or project skills folder:

• Personal skills (locally, only for you) are stored in ~/.claude/skills/<skill-name>/SKILL.md. These are loaded for any project you use Claude Code in.

• Project skills (pushed to Git, shared with a team) are stored in .claude/skills/<skill-name>/SKILL.md within the project directory. These can be checked into git so all team members can access them.

The main file SKILL.md in each skill directory defines the skill. It has a YAML frontmatter for metadata and a Markdown body for the instructions:

---

name: your-skill-name

description: Brief description of what this Skill does and when to use it

---

# Your Skill Name

## Instructions

Provide clear, step-by-step guidance for Claude.

## Examples

Show concrete examples of using this Skill.

Don’t forget that this skills folder can also contain additional files that would help Claude with this skill. If you have an Excel file of financial data you’d like the skill to reference, put it here! If you have a PDF with diagrams describing how something works, it belongs here, too. Anthropic even suggests putting examples as files and suggests Markdown as an appropriate format.

You can list all currently loaded skills by using the slash command in the CLI. In practice, since skills are automatic, the way to test a skill is to pose a query that should trigger it. For example, after creating a “PDF processing” skill, you can ask Claude something involving PDFs to see if it uses the skill. Claude’s response often indicates when it’s using a skill (you might see it mention reading the skill file in the conversation, especially if running in debug/verbose mode).

My favorite skill is for consistent frontend design. I give brand colors, font choices, and other UX specifications in a frontend design skill so any changes Claude makes to my views are aligned with the existing patterns. This saves me some time and keeps my meta CLAUDE.md file from having too much extra context.

Using CLAUDE.md effectively

Claude Code heavily leans on a context file named CLAUDE.md, which provides persistent background information and guidelines for your project. This file is automatically loaded into Claude’s context every time you start a session in that project directory (or at global scope), so you don’t have to repeatedly remind the AI of fundamental details about your project or preferences. Essentially, it’s a way to “inject” important context or rules before any conversation begins.

A context file is where you put fundamental project knowledge, constraints, and style guidelines that you want Claude to always be aware of. Think of it like documentation or notes that are always open to Claude.

You can even start your CLAUDE.md with the /init slash command, but don’t forget to review it before moving forward.

By placing project descriptions, architecture breakdowns, style conventions, and important requirements in CLAUDE.md, you ensure Claude Code always “remembers” this context, leading to more relevant and accurate assistance.

Claude Code is smart about where it looks for context files. It supports multiple CLAUDE.md files in a hierarchy. If present, the global context file is loaded for all sessions (your user-level default context).

The project context file in the current working directory (usually your repository root) named CLAUDE.md is loaded if present. You can also use CLAUDE.local.md in the project directory if you want a version that isn’t checked into git.

If you are working in a sub-folder of a larger repo, Claude will also load any CLAUDE in parent directories up to where you launched it. This is especially useful for monorepos or layered projects. Conversely, Claude can load files from subdirectories when relevant. For example, if you start at the repo root and ask about code in a given directory, and that directory contains its own CLAUDE.md, Claude will pull in that subfolder’s CLAUDE.md content if needed.

What will you build with Claude Code?

Claude Code really pushes an agentic programming model. This is incredibly useful, but comes with a learning curve. Enabling the use of specialized subagents, on-demand skills, and persistent context gives you an edge when tackling complex coding tasks. By understanding how these pieces work together, you can dramatically enhance your productivity and code quality.

If you want more tips on using Claude Code (or other AI tools for software engineers), we’d love to have you as a reader over at The AI-Augmented Engineer.

The AI-Augmented Engineer

Learn how to use AI tools to write better code, ship faster, and stay ahead of the curve

By Jeff Morhous

Recent posts on coding better with AI:

Each month, I'll share my top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and useful tools for folks collaborating with AI.

Over the past month, I’ve focused on exploring advanced coding workflows powered by Claude Code and open-source task orchestration tools. This post covers everything from DeepWiki's instant repo querying to Anthropic’s subagent ecosystem, Claude Code best practices, and structured agent workflows with Task Master AI.

To close things out, I also highlight two powerful tools from Google and Mistral for document parsing, which bring new levels of accuracy and traceability to text extraction.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Recent posts on coding with AI

(1) DeepWiki

One of the most underrated AI coding tools released this year has become an essential part of my coding workflow.

DeepWiki, from the team behind Devin the coding agent, turns any GitHub repository into an instant wiki. Replace github.com with deepwiki.com in any repo URL and start asking questions without digging through files.

I now use it for:

1. Onboarding to unfamiliar codebases - targeted explanations with direct file links

2. Understanding implementation patterns - authentication flows, state management approaches

3. Evaluating open-source projects - licensing, security posture, maintenance status

4. Environment setup - getting exact commands and dependencies with citations

5. Building context for AI coding agents - structured summaries and architectural overviews

Every answer includes clickable, line-level citations that link back to source files. No hallucinated summaries, just grounded responses tied to actual code.

The tool integrates directly into Claude and Cursor via a free MCP server, making it queryable within your existing workflow.

I've documented 8 specific use cases with examples in a detailed breakdown of how I integrate DeepWiki into my development process https://www.aitidbits.ai/p/deepwiki

(2) Claudia - Manage Claude Code easily

Claude Code just got its first serious GUI.

A two-month-old open-source tool called Claudia turns Claude Code into a full-featured desktop environment for agent-based development with custom agents, usage tracking, session versioning, and a visual project browser that actually makes your ~/.claude directory usable.

Key features:

1. Project + session management – browse, resume, and diff past sessions across projects, with smart search and visual timelines

2. Agent creation – build persistent Claude agents with custom prompts, sandboxed permissions, and background execution

3. Usage tracking – real-time analytics dashboard showing cost per model/project, usage history, and exportable data

4. MCP support – manage your Model Context Protocol servers from a dedicated UI, with easy import from Claude Desktop

5. Markdown support – edit, preview, and organize CLAUDE.md files across projects with live rendering and syntax highlighting

Under the hood, Claudia runs on Tauri and supports multi-process execution for non-blocking agent runs.

Claudia makes Claude Code accessible for developers who've been hesitant about command-line AI tools or struggled to track Claude Code sessions across projects.

(My recent approach changed from Claudia + Cursor to Claude Code + Task Manager, but Claudia is still useful for the CLI haters).

GitHub repo https://github.com/getAsterisk/opcode

(3) Anthropic’s tips for using Claude Code

The Claude Code team just revealed how they responsibly merged 22,000 lines of AI-generated code into production.

Their breakthrough? They stopped treating AI like a magic wand and started managing it like you would any capable but inexperienced team member.

Here are the five best practices they shared for "vibe coding" in prod:
(1) Be Claude's product manager - spend 15-20 minutes collecting requirements, exploring the codebase, and building a plan together before letting the AI execute. Just like you wouldn't tell a new hire "build this feature" without context.

(2) Focus on leaf nodes, not core architecture - use AI for end features and components that nothing else depends on. Protect your system's trunk and branches where tech debt would be catastrophic.

(3) Design for verifiability - create systems with easily verifiable inputs/outputs and stress tests. You need to validate correctness without reading every line of generated code.

(4) Prepare for the exponential - AI can handle ~1-hour tasks today, but that capability doubles every 7 months. Next year, it'll tackle day-long projects, then week-long ones. If you insist on reviewing every line of code, you'll become the bottleneck.

(5) Know enough to ask the right questions - You still need technical judgment to guide the AI safely. The vulnerabilities in vibe-coded apps come from people who can't distinguish between what's dangerous and what's safe.

The bottom line: we're moving from individual contributors who understand every line to managers who understand the system. The question isn't whether this will happen, but whether you'll be ready when it does.

Full talk

(4) Claude Code Subagents

Claude Code now supports custom subagents, and a new open-source site makes it dramatically easier to start using them.

Claude Code's sub-agent feature lets you delegate to specialized AI experts instead of relying on one overwhelmed assistant. Each subagent operates within its own context window and focuses on its own expertise, much like having a full development team where every member excels in their specific domain.

This new site features 30+ prebuilt Claude Code agents, from UX Optimizer to System Architect to User Researcher. Each is designed to handle a specific task in your workflow. Just click, copy the prompt, and deploy.

These agents coordinate automatically. Ask Claude to "build a secure payment feature" and watch the system architect design the API, the security auditor validate the implementation, and the test generator create comprehensive coverage - all working in parallel.

No more crafting detailed system prompts or figuring out tool permissions - just copy-paste the configurations you need and integrate them in seconds.

Website https://www.subagents.cc

(5) Claude Code tips

Anthropic just shared how their teams actually use Claude Code internally, revealing how frontier labs operate at the cutting edge. An interesting peek behind the scenes.

I expected the usual suspects: debugging, code reviews, boilerplate generation. But applications go far beyond typical developer workflows. Their lawyers built phone tree systems. Marketing teams generate hundreds of ad variations in seconds using specialized sub-agents that process CSV files. Data scientists without TypeScript knowledge are building complete React applications for ML model visualization.

The performance gains are substantial: security engineers reduced debugging time from 15 minutes to 5 minutes by feeding Claude stack traces during production incidents. New data scientists become productive on complex codebases immediately rather than spending weeks learning architecture. Research time for unfamiliar ML functions dropped by 80%.

What's particularly notable is how this shifts organizational dynamics. Product engineers now tackle bugs in unfamiliar codebases without depending on other teams. The Growth Marketing team built agentic workflows that would traditionally require dedicated developer resources.

Claude Code is not only for coding. My observation: we're seeing the early manifestation of a fundamental change in how companies operate. That has been the vibe for some time, but now we actually see this in practice.

Full piece https://www.anthropic.com/news/how-anthropic-teams-use-claude-code

(6) Task Master AI

Read more

In this post, I break down the shift from desktop to cloud-based coding agents, exploring what makes them different, how they fit into real-world development workflows, and where each leading tool stands today. Whether you’re exploring Devin, Codex, Jules, Factory, or Cursor Background Agents, this guide will help you understand how they work, their strengths and trade-offs, and how to get the most out of them.

A NotebookLM-powered video podcast summarizing this post

Keep exploring this post:

Open in ChatGPT

Open in Claude

Agent-assisted coding is evolving quickly. Tools like Cursor, Windsurf, and Claude Code are already part of many developers’ workflows. These desktop agents run locally and rely on continuous back-and-forth: Developer drafts a coding task prompt → Coding agent generates code → Developer asks for changes/fixes → Coding agent implements change → You commit local changes as part of a pull request.

This pair-programming style boosts productivity, but it doesn’t scale. The interaction is synchronous: you must constantly steer the agent from the initial coding task prompt to creating a pull request. Running multiple coding agents in parallel feels like managing multiple junior developers simultaneously.

Real-world engineering teams work differently.

Enter cloud agents, asynchronous coding agents that better resemble a dev on your team: You assign a task → Cloud agent spins up its own environment in the cloud (as if it had its own laptop) → Cloud agent makes changes → Cloud agent opens a pull request for you to review.

You can request and merge changes once the code meets your standards. Some even integrate with Slack and other collaboration tools such as Linear and GitHub, further streamlining the development and CD/CI cycles.

In 2025, the line between desktop and cloud agents is blurring. Cognition, the creator of Devin, the web-managed cloud agent, acquired Windsurf, a Cursor-like IDE that acts as a desktop agent. Cursor, on the other hand, now offers background agents that run asynchronously both locally and on the web. Factory AI (cloud agent) offers a downloadable bridge that enables asynchronous workflows in local environments. Google’s Jules (cloud agent) just graduated out of Beta to complement Gemini desktop CLI (desktop agent), mirroring OpenAI’s Codex (web) and Codex CLI (desktop) approach.

The path for coding alongside AI is set: as models and tooling improve and best practices solidify, coding agents are shifting to asynchronous-first workflows. To clarify, autonomy isn’t a “web” feature, it’s an agent capability. It just so happens that, today, most fully autonomous agents are delivered as web-based tools.

In this post, I’ll walk you through what it’s like to work with each of the leading cloud agents, including a screen recording of my workflow so you can see how the interfaces look and behave in practice. Whether you’re curious about what these agents can actually do or trying to figure out which one fits best into your development workflow, this guide is for you. I’ve also included a comparison table at the end that makes clear which tools truly stand out.

Evaluation framework

Each agent was evaluated across four criteria:

1. Overall experience - onboarding flow, coding UX, working process smoothness (planning → execution → testing), and pull request clarity.

2. Team integration - how well the agent fits into real workflows: taking tasks, opening solid pull requests, addressing feedback, and communicating through platforms like Slack.

3. Autonomy - the level of independence from assignment to pull request: does the agent require step-by-step guidance and close supervision, or can it deliver end-to-end?

4. Cost - pricing model and the actual cost of completing the benchmark task.

To evaluate the agents, I gave each one the same benchmark assignment: add recurring task support to a lightweight to-do app repository:

Add support for recurring tasks. Users should be able to pick from daily, weekly, or monthly recurrence options when creating or editing a task. When a recurring task is marked complete, create the next occurrence immediately with the due date shifted by the chosen interval. Keep changes simple.

I deliberately chose a more straightforward task that all agents naturally completed successfully. The goal of this post is not to benchmark their performance, but to evaluate the experience of working with them. In future posts, I plan to conduct more complex evaluations to compare these agents on challenging, real-world tasks.

Become a premium member to access the full LLM Builders series, $1k in free credits for leading AI tools and APIs (Claude, Hugging Face, Deepgram), and editorial deep dives covering AI coding and voice agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

The results

Devin

Overall experience

Setup took minutes: sign in, connect GitHub, and Devin is ready to go. It scanned the codebase, created a confident plan, executed the task, and opened a well-structured pull request, all autonomously.

The experience felt like pair programming with a senior engineer: you see the shell (the command line for running code), VS Code (where code is edited), and a browser (for testing), all updating in real-time.

The pull request included a clear summary, test plan, and even a diagram, making review easy.

Devin handled feedback directly through GitHub, just like a real teammate. It felt like collaborating with someone who not only ships quality code, but also knows how to get it merged.

Team integration

Devin slots naturally into team workflows. On GitHub, you can review its code or ask it questions exactly as you would with a colleague. It also integrates with Slack, Linear, and Jira, allowing you to tag it in a thread or assign it to an issue.

Devin can also connect to MCP servers, enabling seamless connections to external tools and internal systems. Through its MCP server, Devin can pull in structured context from documentation, analytics, and monitoring platforms like Notion, Sentry, and Datadog. This makes it easier for Devin to act with deeper awareness of your infrastructure and business logic.

Autonomy

Devin is fully autonomous: once you assign a task, it produces a pull request without further input. For web apps, it can even run and test the app itself. This autonomy is powerful because it allows you to run multiple coding agents that don’t require supervision. The downside is that it can go off-track if your prompt and intentions are vague, wasting time and tokens. Fortunately, Devin has substantially improved since the last time I tested it in December, making it autonomous and useful.

Cost

Devin’s pricing is structured in Agent Compute Units (ACUs). Those units represent the work done by Devin in a single session. Steps like planning, gathering context, running code, or using the browser all consume ACUs.

Each ACU costs $2.25. My benchmark task used 3 ACUs, which comes to about $6.75. That’s steep for a simple job. This novel ACU model also introduces friction. Since no other coding agent uses it, there's no mental benchmark, making it harder for developers to estimate costs. The lack of transparency creates hesitation that hinders adoption, especially when simpler pricing models are the norm.

How to get the most out of Devin

(1) The Prompt Improvement Button

Devin has a built-in prompt improver that refines your instructions before it starts. Running prompts through it clarifies intent and removes ambiguity, which helps Devin produce more accurate, review-ready pull requests.

(2) Leverage Devin’s Knowledge capability

Knowledge lets you onboard Devin with your project’s context, just like you’d ramp up a new engineer. It serves the same purpose as coding agents context files such as cursor.md or AGENTS.md, but with structured triggers built in.

Add information in small pieces, group it in folders, and link it to repositories with triggers so Devin knows when to apply it. Store anything you would want an engineer in your team to know: coding standards, workflows, deployment steps, bug fixes, etc.. Once added, Devin recalls and applies it automatically. More tips covering Knowledge here.

(3) Devin Playbooks, à la Claude Subagents

Playbooks are reusable prompts for recurring tasks. Instead of re-explaining a process to Devin every time, create a playbook and ask Devin to use it. It’s like showing a teammate how to do something once and having them write it down so they never ask again.

(4) Connect Devin to Slack/Linear/Jira

Plug Devin into your team’s task management workflow: Assign it issues or tag it in threads. It will pick up the task immediately and get to work. That is especially useful when on the road, as you can tag Devin in a Slack conversation and ask it to take a first (and last?) pass at fixing a bug or implementing a feature.

Recent posts on coding better with AI

OpenAI Codex

Overall experience

Codex is built into ChatGPT, so setup is quick: load your repo, assign a task, and it gets to work. It scans the code, plans, runs tests, and opens a pull request. Also, as of last week, Codex can run in your IDE of choice as an extension, supporting Windsurf, VS Code, and Cursor. You can use it locally or delegate tasks to the cloud.

The interface is minimal, with a shell view and collapsible logs. When complete, Codex provides a concise summary with direct code references to expedite the review process.

Team integration

Codex doesn’t play well with existing tools such as Slack, Linear, and Jira, which is a major downside given that many teams use Slack to discuss and assign work, and GitHub is the familiar interface for code reviews.

Autonomy

Codex is fully autonomous: once assigned a task, it works in its isolated cloud environment and delivers a complete pull request within minutes without requiring further prompting. Similar to Devin, that could be a neat feat if the agent is capable enough, but dangerous if vague coding tasks lead to poor generated code. For our simple task, Codex performed well enough.

Cost

Codex is included with ChatGPT Plus or Pro, so there is no extra charge beyond your subscription. That said, usage limits are not publicly disclosed, which can be a red flag for many teams that require a reliable coding agent with transparent rate limits.

How to get the most out of Codex

(1) Generate multiple responses simultaneously, choose the best one

Codex’s Best of N feature allows you to generate several independent solutions in parallel for the same task. This lets you quickly explore different approaches and select the one that best fits your needs, without adding time to your workflow.

(2) Enable internet access when needed

By default, Codex runs offline after setup, so it cannot look up documentation or install new packages. Enabling internet access solves this issue, but it should only be used when the task requires it, as it carries security risks. For instance, an attacker could slip in a malicious command that Codex executes, unintentionally leaking sensitive data.

(3) Tag Codex in pull requests

Tag Codex for questions or reviews via @codex <question> or @codex review. You can also tag it for changes, but that spawns a new task with a separate pull request you’ll need to merge back into the original.

Google Jules

Overall experience

Setup was quick: I signed in with Google, linked GitHub, and Jules was ready. I provided the task and it then scanned the codebase, generated a detailed plan, and executed it step by step until a pull request was opened. The interface, structured around the plan, made progress easy to follow with expandable diffs, similar to Claude Code’s and Cursor’s recently released planning feature.

However, the pull request’s description was lacking. For example, it stated “This change adds support for recurring tasks…”, which reads more like a product note than a proper description of a technical feature. I expected at least one section covering the technical components added/changes to facilitate a more efficient and easier review. Code review also felt unnatural. Similar to Codex, I couldn’t review the pull request as I would on GitHub: all reviews had to happen in Jules, where I could only leave a single block of feedback instead of commenting on individual lines. While the grouping of changes by the plan’s action items in the interface made diffs easy to follow, the process was rigid compared to reviewing a colleague’s PR.

Team integration

Jules does not fit smoothly into team workflows. On GitHub, you cannot review its code or collaborate with it the way you would with another developer. It also lacks integrations with tools like Slack, Linear, or Jira, leaving its interface as the primary means of interaction. As a result, Jules feels separate from the normal channels teams rely on to collaborate.

Autonomy

Jules is semi-autonomous. It generates an implementation plan that you approve in the beginning. If you don’t respond, the plan auto-approves and executes, completing the task end-to-end without requiring your input, unless it encounters roadblocks or needs clarifications, similar to how Claude Code prompts the user for input.

Cost

Jules is part of Google’s AI package. It offers the following tiers:

• Free tier - 15 tasks per day, 3 concurrent tasks

• Pro - 100 tasks per day, 15 concurrent tasks

• Ultra - 300 tasks per day, 60 concurrent tasks

The free tier makes it easy to get started. The pricing model is transparent and predictable, as it remains consistent regardless of task complexity and does not depend on unpredictable factors. For developers, this makes Jules straightforward to adopt and scale.

How to get the most out of Jules

(1) Interactive Plan mode

Start tasks using the Interactive Plan mode. Jules will then scan the codebase, ask clarifying questions, and create an implementation plan you’d approve before execution. This prevents ambiguity and ensures the output matches your intent, and is already the way I code nowadays. Feels like a junior developer confirming requirements before writing code.

(2) Configure environment setup

If a repo requires setup commands such as dependencies or environment variables, define them upfront. Jules will run them automatically when working with that repo, keeping environments consistent.

Factory AI

Overall experience

Factory is built around the concept of “Droids”: autonomous agents purpose-built for different roles across the software development lifecycle (Code, Reliability, Knowledge, Product). Each Droid comes with its own architecture, memory, and tools, and can operate independently or in parallel. Unlike traditional coding assistants, Droids are designed to own and execute tasks end-to-end, making Factory feel more like a full-stack engineering team than a co-pilot, similar to the idea behind Claude Code's Subsagents.

When using Factory, it’s clear that the company is focused on enterprise users, rather than indie developers, which could hinder the organic bottom-up growth that Claude Code and Cursor benefit from. For instance, you must enter credit card details for a trial with no way to pre-cancel. Pricing is also somewhat opaque and tied to token usage: $40 for 20 million tokens.

Factory’s most significant drawback, however, is the indexing process. For our small benchmark repository, indexing took anywhere from several minutes to multiple hours. That’s a considerable delay just to get started. In a developer world optimized for instant feedback loops and low-friction experimentation, this delay feels off.

On the other hand, Factory shines in its coding cycle: it builds a clear to-do list, asks intelligent and clarifying questions, and executes with visible reasoning and file-by-file diffs. The pull request is detailed and reads like something a senior engineer would write. It also provides the broadest set of integrations, allowing you to consume relevant context from popular tools such as Notion, Google Drive, and even incident management tools like Sentry and PagerDuty.

Additionally, to Factory’s credit, user experience appears to have become a recent priority. When I first used it back in March, the interface was cluttered and overwhelming: too many panes, not enough clarity. During my coding session for this article, however, the redesign was substantially better: the layout is cleaner, more focused, and thoughtfully organized around the key decisions and information needed at each stage.

Team integration

Integrations is where Factory differentiates itself, showcasing again its enterprise focus. It can consume context from your team’s internal knowledge systems: Slack, Linear, Jira, Notion, Google Drive, Sentry, and PagerDuty. Such integrations infuse the coding agent with context beyond the project’s repository, which improves the agent’s performance.

These integrations allow Factory to enrich its understanding of your engineering workflows far beyond what's available in the repository. However, these integrations are read-only: Factory can ingest context from these platforms but cannot be actively directed through them. You can’t, for example, tag Factory in a Slack thread or assign it a Linear ticket and expect it to take action, unlike agents like Devin.

Cost

Factory offers a 14-day trial, after which pricing starts at $40 per month for 20 million tokens. Our task consumed ~330k tokens. As a full-time engineer, you’ll likely burn through this quota quickly–our simple feature alone ate through over 1.5% of the monthly allowance.

Autonomy

Factory is semi-autonomous and often requires your input. It avoids making assumptions, it often asks clarifying questions before starting, and seeks approval before taking actions, such as creating a pull request. By default, it will not create branches or commit files without your approval, though you can disable this safeguard.

How to get the most out of Factory

(1) Use Factory Bridge

Bridge is a secure connector that links Factory's cloud platform to your local machine. It enables running CLI commands, managing local processes, and accessing local files directly from Factory sessions. Tip: use it within an isolated environment (e.g., Docker) for safe and optimal results. I used Bridge when I couldn’t access my laptop or wanted to split tasks between Factory and Claude Code, saving Factory tokens while running both agents in parallel.

(2) Use a remote machine

When local access is not needed, connect Factory to a remote machine. It still grants full command and workflow access without requiring you to touch your computer.

Cursor Background Agent

Overall Experience

Getting started is easy for existing Cursor users: simply submit a task and select “Send to Background”. On the web, log in and paste your coding prompt.

You can watch the agent’s progress by connecting to its virtual cloud environment (“Open VM”), which mirrors a live coding session. Once complete, you get a change summary and file diffs in both desktop and web.

The pull request description is barebones, often limited to a single sentence. It lacks a detailed context that would streamline reviews. Code review, however, feels seamless: you can comment directly on GitHub, tag the agent, and get a response in-line, just like working with a human teammate.

Team integration

Cursor integrates directly into Slack and Linear, so you can tag it in threads or assign it to issues.

Autonomy

Cursor’s Background version is fully autonomous. Once you assign a task, it analyzes the codebase, executes the plan, and opens a pull request without requiring user input.

Cost

Background agents are billed at the API rate of the model you choose, which is affordable with models like gpt-5-high. The task I ran came at 8¢. Token-based pricing is inherently unpredictable, but that’s already the norm with the popular Claude Code and Cursor desktop agents.

How to get the most out of Cursor Background Agent

(1) Add a .cursorrules file to your repositories

Cursor’s best practices also apply to background agents. Make sure every repository you use includes a .cursorrules file, just as you would when working locally (you can use my recently released tool to generate one).

(2) Connect Cursor to Slack or Linear

If your team uses either tool, integrate Cursor so you can assign it issues or tag it in live discussions. Whether it is a bug in Slack or a ticket in Linear, the agent can pick it up immediately and start working.

More tips here​​ https://www.aitidbits.ai/p/sahar-ai-coding

Choosing the right coding agent

We’re just getting started with coding agents

All of these tools are still in their early stages. We can expect them to evolve quickly—I had to update my post twice over the span of two weeks due to new releases.

Just as importantly, the paradigm of working with autonomous agents is still taking shape. Tools like Claude Code’s Subagents and newcomers like Conductor and Task Master hint at what’s to come. In future posts, I’ll dive deeper into each of the tools reviewed here. Subscribe to follow along as I learn how to collaborate with this new generation of coding agents.

In this post, I break down how I use DeepWiki - my go-to tool for understanding unfamiliar codebases, spinning up dev environments, and generating context for coding agents like Claude and Cursor. Whether you're evaluating an open-source repo, onboarding to a new project, or building an AI-powered dev tool, DeepWiki can save you hours.

Note: This is not a sponsored post or paid collaboration. I use DeepWiki in my day-to-day workflow and wanted to share how it's been useful.

A NotebookLM-powered podcast episode discussing this post:

We are generating more code than ever. With LLMs like Claude already writing most of Anthropic’s code, the challenge is no longer producing code, it is understanding it.

DeepWiki, from Cognition, the team behind Devin, the AI software engineer that went viral with its Fiverr demo in March 2024, turns any GitHub repository into an instant, navigable wiki.

Point it at a repo and start asking questions without digging through files or reading every line. Just replace github.com with deepwiki.com to spawn a wiki page for a repository:

https://github.com/Dicklesworthstone/claude_code_agent_farm → https://deepwiki.com/Dicklesworthstone/claude_code_agent_farm

This is a lightweight field guide to the tool that has become my must‑have coding companion.

Recent posts highlight

How DeepWiki works

• Public vs. private - querying public repos works out of the box. For private repos, you’d need to sign in with a (free) Devin account.

• Fast vs. Deep Research modes - Fast mode answers instantly from the code graph. Deep Research spends extra cycles reading across files to provide higher-confidence, multi-hop answers.

• Grounded answers - every answer includes clickable, line‑level citations back to the source files, so you can jump straight to the code and avoid hallucinated summaries.

Use DeepWiki via web or AI IDE

You can use DeepWiki by pasting a GitHub URL on deepwiki.com or by plugging it directly into your AI coding environment using the official DeepWiki MCP server (my preferred way).

The DeepWiki MCP server requires no authentication and is supported natively by Claude and most AI IDEs, such as Windsurf and Cursor. Just add the MCP server to your config (instructions here).

This turns DeepWiki into a live, embedded research sidekick: always-on, context-grounded, and queryable directly within your workflow.

DeepWiki is how I read other people’s code now

(1) Evaluating an open-source project

Engineering velocity has exploded, and with it, the number of open-source libraries released every day.

Before pulling in a new library, I run a quick eval loop: is it actively maintained? How’s the security posture? Any data sent to third parties? Does the license play well with commercial use?

With DeepWiki, I get these questions instantly answered, linking me to the exact configuration, network calls, and license terms, so I can make a quick go/no-go decision.

(2) Quickly setting up new environments

For public or private repos, ask “how do I run this locally?” and you’ll get the env setup, required services, and dependency graph with citations to README, Dockerfile, and scripts, so you can hit the ground running.

(3) Borrowing implementation details

Suppose you find a clever mechanism in another repository, such as an authentication flow or a clever way to persist state locally. In that case, you can ask DeepWiki to provide a Markdown cheat sheet: a breakdown of how it works, which files define it, and what it depends on. You can then drop that summary directly into Claude Code or Cursor as structured context and ask it to implement it in your project.

For instance, I needed to manage multiple coding agents via the terminal. I found a repo that orchestrated them using tmux. DeepWiki mapped the scripts and config, and within ten minutes, I had the same terminal layout running in my project.

(4) Creating custom onboarding guides

Treat it like a patient senior engineer. Ask targeted questions like “Can you walk me through how the queue processor handles retries?”, “What’s the data flow during user signup?”, or “I want to implement feature X - where should I begin?”, and receive tailored explanations with direct links to the relevant functions.

Become a premium member to access the full LLM Builders series, $1k in free credits for leading AI tools and APIs (Claude, Hugging Face, Deepgram), and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

(5) Surfacing first contributions

New team or contributing to a new open-source library? Ask for “good first issues” based on TODOs, failing tests, flaky areas, or missing docs. DeepWiki surfaces approachable fixes and where to start reading.

(6) Navigating cookbook-style repositories

Some repositories are more like collections of examples than actual packages. Anthropic’s cookbook and Gemini’s cookbook are good examples. DeepWiki can help you find the example you need and even generate the code.

(7) Building context-aware coding agents

If you’re building a project or tool that requires understanding the context of a codebase: its structure, architecture, or coding style, DeepWiki can do the heavy lifting.

I built a tool called Sidekick that generates cursorrules.md and claude.md files: markdown summaries that coding agents use as context during generation. Instead of writing those summaries yourself, Sidekick uses DeepWiki to auto-generate them from the repo.

Since DeepWiki’s MCP API is free and open, you can plug it into any product or library that benefits from code awareness: onboarding flows, test generators, AI pair programmers, or anything else that needs a high-level map of the code.

(8) Reviewing and ramping up on pull requests

So your teammate opens a PR and asks for a review. You weren’t involved in the feature, and now you’re staring at a diff with no context. With DeepWiki, you can quickly ramp up. Just replace github with deepwiki in the PR URL to get a structured summary of the proposed changes. For example:

https://github.com/saharmor/simulatedev/pull/7 → https://deepwiki.com/saharmor/simulatedev/pull/7

DeepWiki helps you understand not just what changed, but how it fits into the broader codebase, saving time, reducing back-and-forth, and making you a better reviewer.

When to use DeepWiki

DeepWiki is now the first thing I reach for when I need to reorient quickly: implementing a new feature that touches unfamiliar parts of the stack, returning to a component I haven’t touched in months, or diving into a dense open-source repo.

Instead of grepping around, I skim the generated wiki, ask a few follow-up questions, and jump straight into the files that matter.

What I wish DeepWiki had

Two features I’d love to see:

1. Conversational sidekick mode - leave DeepWiki running next to your IDE and ask, “Where does this function get called?” or “How do I run the worker locally?” (I hacked a prototype last weekend that I plan to share soon).

2. Task-based onboarding - provide a repository and a goal (“fix this open issue”), and receive a step-by-step path through the exact files, functions, and setup commands needed to contribute.

Try it yourself at deepwiki.com

Each month, I'll share my five seven top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Over the past few months, I’ve been developing and experimenting with several open-source packages while creating a few AI agents. This post covers six of my most popular LinkedIn posts on the topic, each showcasing the package’s core features and my key takeaways. Plus, a bonus tip at the end: a toolkit that automatically converts prompts from GPT and Claude into Llama-optimized versions, boosting performance by up to 45%.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Recent posts:

(1) LangMem

LangMem is a new open-source library that gives LLM agents long-term memory and it’s refreshingly easy to use.

It’s built for developers working with LangGraph or custom agents, and it solves a persistent problem: how to make agents remember and adapt across sessions without bloated prompts or manual hacks.

LangMem introduces a clean memory API that works with any storage backend and includes tools for:

1. Storing important information during conversations - agents decide what matters and when to save it

2. Searching memory when relevant - retrieving facts, preferences, or prior context

3. Running background memory consolidation - automatically refining and updating knowledge over time

It integrates natively with LangGraph’s memory store, but you can also plug it into your own stack using Postgres, Redis, or in-memory stores.

This design is especially useful for building agents that need to:

-> Personalize interactions across sessions

-> Maintain consistency in long-running workflows

-> Adapt behavior based on evolving user input

Unlike Mem0, which requires explicit memory updates, LangMem handles memory automatically in the background, storing and retrieving key details as needed, and integrates with LangGraph out of the box.

GitHub repo https://github.com/langchain-ai/langmem

(2) Browser Use

LLM agents can read the web, but few can truly use it. This open-source package changes that.

Browser-use is an open-source library that turns any LLM into a browser-native agent, with first-class support for real UI actions and multi-step tasks.

Out of the box, it supports:

1. Direct interaction with Chromium via Playwright - no extra scripting layers or wrappers

2. Seamless LLM integration - use GPT-4o, DeepSeek-V3, Claude, Gemini, or even Grok

3. Ready-to-run UX - spin up agents with a single function, or test flows in a ready-to-run Web UI or CLI

Browser use can handle real-world tasks like:

-> Checking your latest Stripe payouts and updating a financial tracking sheet

-> Logging into your CMS, creating a draft blog post, and uploading media

-> Scraping product reviews across sites and summarizing them in a shared doc

-> Tracking changes to your competitors’ pricing pages and alerting your team

For those building AI agents that go beyond chat, this gives you a direct bridge to the real web, not a sandbox.

GitHub repo https://github.com/browser-use/browser-use

My recent post on coding with AI:

(3) OpenAI Agents SDK

OpenAI has one of the most useful frameworks for multi-agent workflows, and it’s open-source.

Building production-ready agent systems has been notoriously complex, requiring deep knowledge of orchestration patterns, handoff mechanisms, and debugging distributed AI behavior. The new OpenAI Agents SDK simplifies this complexity with a remarkably clean Python interface that handles the heavy lifting.

Why I find this framework so useful:

1. Provider-agnostic design - works with OpenAI's APIs plus 100+ other LLMs, so you're not locked into a single provider

2. Built-in handoffs - agents can seamlessly transfer control to specialized agents based on context, like routing Spanish queries to Spanish-speaking agents

3. Integrated tracing - every agent run is automatically tracked using popular tools such as AgentOps, Braintrust, and Arize AI Phoenix, making debugging multi-agent conversations straightforward instead of impossible

4. Guardrails by default - configurable safety checks for input and output validation prevent runaway behavior

Setting up a triage system that routes conversations to language-specific agents takes just a few lines of code, with the SDK handling message persistence, context switching, and execution flow automatically.

I'm particularly impressed by the tracing capabilities - the framework integrates with popular observability tools like Logfire, AgentOps, and Braintrust, giving you visibility into exactly what your agents are doing and why.

For developers who have been intimidated by the complexity of multi-agent architecture, this SDK removes the final barrier to building sophisticated agent workflows that actually work in production.

GitHub repo https://github.com/openai/openai-agents-python

(4) Agno

Most agent frameworks I've used struggle with performance at scale, but I recently tested one that achieves microsecond-level instantiation.

The math doesn’t lie: if each agent takes seconds to spin up and consumes megabytes of memory, running the thousands needed for complex workflows becomes infeasible.

A new library called Agno addresses this through architectural decisions that prioritize performance without sacrificing functionality. The framework supports 23+ model providers and implements a progressive five-level agent architecture, from basic tool-enabled agents to coordinated multi-agent workflows.

Key technical capabilities include:

1. Native multimodal processing - handles text, image, audio, and video inputs without additional preprocessing layers

2. First-class reasoning implementation - agents can explicitly "think through" problems using built-in reasoning tools or custom chain-of-thought approaches

3. Agentic search with hybrid retrieval - combines vector search with keyword matching and re-ranking for improved RAG performance

The performance difference is substantial. In head-to-head comparisons with LangGraph, Agno completes instantiation benchmarks before competing frameworks reach halfway through their measurement cycles.

Agno also includes pre-built FastAPI routes, structured output handling, session storage, and monitoring capabilities.

GitHub repo https://github.com/agno-agi/agno

(5) Agents Towards Production

A new, comprehensive, open-source playbook has just solved the biggest challenge in developing AI agents: transitioning from experimentation to production-ready systems.

Unlike scattered documentation or theoretical frameworks, this resource provides executable tutorials that guide you from zero to a working implementation in minutes.

The playbook covers the entire agent lifecycle:

1. Orchestration fundamentals - build multi-tool workflows with memory persistence and agent-to-agent messaging using frameworks like Xpander and LangChain

2. Production deployment - containerize agents with Docker, scale on GPU infrastructure via Runpod, or run on-premise with Ollama for privacy-sensitive applications

3. Security and observability - implement real-time guardrails against prompt injection, add comprehensive tracing with LangSmith and Qualifire, and automate behavioral testing

4. Advanced capabilities - enable dual-memory architectures with Redis for semantic search, integrate real-time web data through Tavily, and deploy agents as APIs with FastAPI

What makes this resource invaluable is its tutorial-first approach. Each concept comes with runnable notebooks and production-ready code.

Whether you're building customer service agents, research assistants, or autonomous workflows, the playbook provides tested patterns for tool integration, multi-agent coordination, and model customization.

GitHub repo https://github.com/NirDiamant/agents-towards-production

(6) Docling

Read more

Each month, I'll share my five seven top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Over the past month, I’ve been deep in the weeds of coding with AI: tinkering, prototyping, and writing about how coding with LLMs can go beyond vibe coding, making us engineers substantially more productive. This post covers six of my most popular LinkedIn posts on the topic, each packed with practical takeaways for both aspiring and experienced engineers. Plus, a bonus tip at the end: an underrated trick from Anthropic to reduce cost by 90% and latency by 50%.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Post(s) published this month:

(1) Parallel AI pair-programming

Recently, I adopted a coding tip from the Anthropic team that has significantly boosted the quality of my AI-generated code.

Anthropic runs multiple Claude instances in parallel to dramatically improve code quality compared to single-instance workflows.

How it works:

1. One Claude writes the code, the coder - focusing purely on implementation

2. A second Claude reviews it, the reviewer - examining with fresh context, free from implementation bias

3. A third Claude applies fixes, the fixer - integrating feedback without defensiveness

This technique works with any AI assistant, not just Claude. Spin each agent up in its own tab—Cursor, Windsurf, or plain CLI. Then, let Git commits serve as the hand-off protocol.

This separation mimics human pair programming but supercharges it with AI speed. When a single AI handles everything, blind spots emerge naturally. Multiple instances create a system of checks and balances that catch what monolithic workflows miss.

This shows that context separation matters. By giving each AI a distinct role with clean context boundaries, you essentially create specialized AI engineers, each bringing a unique perspective to the problem.

(2) Claude ultra think

One extra word can unlock Claude’s hidden reasoning budget and give developers building with AI the planning superpowers.

According to Anthropic, Claude allocates compute tiers based on trigger words:
think < think hard < think harder < ultrathink.

Drop any of these at the start of a prompt and the model gives itself more “brain cycles” before answering. Perfect for discussions on architecture, API design, or edge-case analysis. I also use it when debugging an issue that the existing models (Gemini, o1) didn’t manage to solve.

It works in Cursor/Windsurf and the Claude web chat; no flags, no config.

Give it a try: open your next coding session, preface the prompt with 'think harder' or 'ultrathink' and see how Claude’s performance improves.

My recent post on coding with AI:

(3) Coding visual interfaces with coding agents

Most devs still eyeball CSS tweaks while it’s possible to achieve pixel-perfect UI with coding models like Claude and o3.

Screenshot-driven iteration works across o3, Gemini, and Claude (I’ve had the best luck with Claude). Process:

(1) Paste the design mock via clipboard, drag-drop, or file path

(2) Ask the model: “Implement this layout, screenshot the result, and compare it to the mock. Repeat until they match.”

(3) Watch it code → screenshot → diff → refine until the visual diff hits zero

Design-to-dev handoff shrinks from days to minutes. Frontend teams can A/B three layout variants before lunch, and PMs can solo build UI prototypes without opening Figma.

Try it: paste a screenshot/Figma design with the above prompt in your AI IDE of choice and let the loop run.

(4) Auto-generated MCP servers

One trick I often use and not many other AI builders know: you can now spin up an MCP server for almost any Mintlify-powered API with just two commands.

MCP (Model Context Protocol) servers are the missing layer that enables AI coding assistants like Cursor and Windsurf to generate reliable programs that interact with APIs through natural language, without needing manual integrations or risking hallucinated parameters and broken function calls.

Traditionally, setting up an MCP server meant custom-building one for each API. A tedious, manual process. Now, thanks to Mintlify’s new package, it’s effortless:

1. npm i mcp

2. npx mcp add [api subdomain] (for example, docs.bland.ai)

That’s it. The API you’re trying to work with becomes an MCP server. No custom code, no extra work.

You can test it with the Mintlify-powered Bland API docs to initiate phone calls programmatically.

(5) llms.txt

Most developers still use LLMs to write isolated functions. However, the real power of AI coding lies in making coding models interact reliably with APIs.

That’s where llms.txt comes in.

Inspired by robots.txt, llms.txt is a simple, standardized Markdown file that describes how a website or API works. It resides at the root of a site and distills documentation, endpoints, authentication flows, and usage examples into clear, predictable text. LLMs skip the cumbersome scraping, read the file, and instantly know how to call your API.

Why this matters:

1. Today, prompting an LLM to use an API is messy. Developers must manually write tool descriptions, guess parameter formats, and hope the model figures it out.

2. No HTML parsing or rate-limited from crawling websites

llms.txt provides a structured approach to eliminate guesswork and unlock a new era of API-driven AI coding.

To make it even easier, there’s now a full llms.txt directory, listing APIs and websites that already adopted the standard, including Anthropic, Eleven Labs, and Hugging Face https://directory.llmstxt.cloud

And if your website doesn’t have one yet, you can auto-generate it with Firecrawl's new llms.txt Generator API https://docs.firecrawl.dev/features/alpha/llmstxt

𝘞𝘢𝘪𝘵, 𝘪𝘴𝘯’𝘵 𝘵𝘩𝘢𝘵 𝘴𝘪𝘮𝘪𝘭𝘢𝘳 𝘵𝘰 𝘔𝘊𝘗𝘴? 𝘞𝘩𝘦𝘯 𝘵𝘰 𝘶𝘴𝘦 𝘵𝘩𝘢𝘵 𝘰𝘷𝘦𝘳 𝘔𝘊𝘗?

If you want AI agents to know how your API works and generate correct code, use llms.txt. If you want agents to use an API, set up an MCP server. Think of llms.txt as the knowledge base for AI coding, and MCP as the execution layer for live interactions.

(6) Test-Driven Development with AI

Read more

This one is an updated version of one of my recent popular posts, which outlined the most popular and useful open-source libraries for AI agent builders:

The landscape of AI agent tools is evolving rapidly. After publishing my previous post, I received suggestions for additional packages I hadn't encountered. Over the past few months, I've tested these tools and documented new, valuable agent-related libraries. This post shares my updated insights.

If you often wonder, "What tools are people actually using to build voice agents or understand documents?"—this post is for you. With new packages emerging almost daily, it can be challenging to determine what's state-of-the-art and truly usable. This list is deliberately selective, focusing on the libraries I've personally found most effective, or those recommended by colleagues I trust.

In this post, I'll provide a curated and updated overview of the open-source ecosystem for developers building AI agents. While there’s no shortage of AI agent market maps, most are geared toward non-builders who need actionable tools and frameworks to launch functional AI agents today.

Every package listed in this post allows commercial use and has a permissive open-source license.

Categories covered in this piece:
→ Building and Orchestrating Agents (10)
→ Computer Use (5)
→ Browser Automation (5)
→ Voice (12)
→ Document Processing (7)
→ Memory (3)
→ Testing, Evaluation, and Observability (6)
→ Vertical Agents (7)

Plus:

• Real-world agent stacks: Voice agent that answers phone calls + Browser agent that crawls LinkedIn URLs

• Curated guides and tutorials to get started building agents

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Building and Orchestrating Agents

To build agents that go beyond simple prompting, you need infrastructure for planning, memory, and tool use, and a way to hold it all together.

As more developers started shipping real-world agents, new frameworks popped up and older ones evolved to meet the actual challenges of agentic AI. This section covers the tools I’ve found most effective for building agents that can think, remember, and act with minimal hand-holding.

How to choose?

• For beginners or rapid prototyping, consider Langflow for its intuitive visual interface and OpenAI’s Agents SDK, or LangChain for their simplicity and flexibility.

• For enterprise applications, Portia and CrewAI offer robust features suitable for production environments requiring control and scalability.

• For multimodal or memory-intensive agents, Agno provides lightweight support for agents needing persistent memory and multimodal inputs.

• For complex simulations or data generation, Camel excels in creating customizable multi-agent systems for simulating real-world interactions.

• For autonomous task execution, AutoGPT is designed for agents that need to operate without continuous human input.

Computer Use

AI agents become far more useful when they can operate computers like humans: clicking, typing, browsing, and running programs. The libraries below make that possible, letting agents bridge the gap between language output and real-world action.

How to choose?

• For local code execution via natural language, go with Open Interpreter – it’s fast to set up and great for command-driven agents.

• For agents that need to see and control a computer screen like a human, Self-Operating Computer is your best bet.

• If your agent needs to run in a secure, fast, sandboxed environment, use CUA.

• For dynamic multi-step tasks on irregular interfaces, Agent-S offers the most flexibility with its planning and learning capabilities.

• If your agent relies on interpreting UIs from screenshots (e.g., grounding actions in visual layouts), OmniParser adds critical visual parsing capabilities.

Browser Automation

As AI agents move from passive reasoning to active execution, the browser becomes their gateway to the internet. Whether scraping data, submitting forms, or navigating complex workflows, browser automation tools let agents interact with web pages just like a human would—with clicks, scrolls, and typed input. These libraries differ in abstraction level, performance, and agent integration, so choosing the right one depends on your goals.

How to choose?

• For a low-code, declarative approach where the LLM plans the steps, try Stagehand.

• If you're building agents that need to deeply understand and extract content from websites, Firecrawl offers the cleanest pipeline.

• For LLM-friendly control over browser actions with integration hooks, I’d recommend the popular browser-use.

• Choose Playwright if you need more low-level control over browser actions across browsers.

• Use Puppeteer if you need fast, scriptable Chrome automation in a Node.js environment.

Voice

Voice is still the most intuitive interface for humans, and increasingly, for agents too. These tools let agents handle speech in and out: understanding spoken language, keeping track of conversations, and responding naturally.

How to choose?

I wrote a whole post covering open and commercial packages and APIs for voice agents, including my guide for choosing the right ones:

Dia compared to ElevenLabs and Sesame 👆

Document Processing

Modern AI agents must process and comprehend documents in various formats, from PDFs to images containing text. The following open-source tools empower agents to extract, interpret, and act upon information from unstructured documents, facilitating real-world business processes.

How to choose?

• Long-form PDFs such as contracts, research papers - use Qwen2.5-VL or mPLUG-DocOwl2 for efficient multi-page understanding without relying on OCR. And, as of a few months ago, you can also easily fine-tune a DocOwl2 model on your own data with ms-swift.

• Text + image docs such as medical reports, annotated diagrams - try Molmo for high-resolution multimodal inputs, visual QA, and GUI parsing.

• Layout analysis & table extraction - use Docling for JSON/Markdown conversion, or LayoutLMv3 for form understanding and layout-aware modeling.

• Lightweight multimodal with speech - Phi-4 handles text, vision, and speech in a compact model—great for on-device agents.

Memory

To feel truly intelligent, AI agents need memory. Without it, they’re stuck in single-turn loops, forgetting what just happened, what the user wants, or what they already did. The libraries below help agents remember, adapt, and personalize, enabling everything from contextual conversations to long-term planning.

How to choose?

• For long-term, personalized memory that improves with use, Mem0 is a solid choice—especially if you want minimal overhead.

• Use Letta when building agents that need persistent memory across sessions and integration with tools or APIs.

• To enable active memory management and knowledge sharing among agents, LangMem facilitates dynamic memory operations and shared knowledge bases.

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Testing, Evaluation, and Observability

As agents grow more complex, they need to be tested, measured, and monitored like any serious software system. These tools help you catch edge cases, debug behavior, and track performance, both during development and in production.

How to choose?

• To monitor and benchmark agent performance in production environments, AgentOps provides robust tracking and analysis tools.

• When comparing various agent configurations or conducting A/B tests, Agenta facilitates structured evaluations.

• To integrate observability into LLM applications, OpenLLMetry leverages OpenTelemetry for seamless monitoring.

• If detecting and addressing performance, bias, or security issues is a priority, Giskard offers automated scanning capabilities.

• For comprehensive LLM observability and debugging, Langfuse provides an open-source platform tailored for LLM applications.

• For voice agent evaluation across different models and prompts, VoiceLab offers a comprehensive testing framework.

Vertical Agents

The open-source world is full of vertical agents: purpose-built tools for coding, research, data analysis, and more. I’ve tested a bunch. These are the ones I’d actually reach for when building something real.

How to choose?

• Goose allows custom workflow integration to build extensible AI coding assistants.

• For comprehensive coding agents with GUI capabilities, OpenHands offers a full-stack solution inspired by Devin.

• If you prefer a Claude Code-like terminal-based pair programming, aider provides Git integration and multi-file editing.

• To convert UI designs from images to code, screenshot-to-code automates the prototyping process.

• For autonomous research tasks, GPT Researcher can scrape, summarize, and export findings efficiently.

• For conducting in-depth, privacy-focused research using local LLMs, Local Deep Research offers iterative analysis and comprehensive, cited reports.

• If your focus is on generating SQL queries from text, Vanna offers customizable and database-integrated solutions.

  

Lastly, here are real-world stacks and beginner-friendly tutorials to help you launch your first AI agent.

Real-world agent stacks

All the tools above are powerful on their own, but how do they actually fit together in practice? What does a real architecture look like when you're stitching these components into something usable, testable, and shippable? I’ve compiled a few concrete examples from recent open-source projects and builders in the space. If you're trying to move from “exploring tools” to “building real systems”, these will give you a head start.

Voice agent that answers phone calls

Read more

This post captures my takeaways from attending Google’s flagship event, I/O 2025. It’s not a comprehensive announcement round-up. Instead, I’ve focused on the launches that matter most to anyone building or working with AI. I also share my perspective on what these moves mean for the broader AI ecosystem and founders, developers, and researchers alike.

A NotebookLM-powered podcast episode discussing this post:

Since 2017, when Google unveiled the groundbreaking Duplex demo at its biggest event of the year, Google I/O, I've been captivated by the company's AI advancements. For me, it was the first truly practical, consumer-facing use of AI—a clear example of how AI could take over routine tasks like booking appointments. But more importantly, it marked a key step toward a future where AI helps people express themselves in ways that were previously out of reach.

In recent years, the AI community has often viewed Google as trailing behind leaders like OpenAI and Anthropic. However, this year's Google I/O conference felt different—everything finally clicked. Google moved from research to reality, capitalizing on its massive distribution channels and deep technological prowess. The perfect combination of state-of-the-art technology with access to real-world usage through Search, Google Workspace (Gmail, Sheets, Docs, etc.), and Android (smart TVs, glasses, phones).

And it wasn’t only me. The same sentiment echoed across the press tent at I/O last Tuesday, capturing an energy reminiscent of OpenAI’s inaugural DevDay.

The winning combination, as defined by Google in this week’s I/O, manifests across three principles:

• Powerful - deploying best-in-class models to support real-time, reliable experiences

• Personalize - tailoring AI to understand and cater to individual user preferences and needs

• Proactive - developing AI that anticipates user needs and acts accordingly without being too intrusive or eager

Out of these three, the one I found the most promising is Personalize.

Google's unparalleled access to user data gives it a powerful edge over competitors like OpenAI and Apple. It understands my interests through the searches I make (Search), the places I go (Maps), the music I listen to (YouTube), my payment habits (Google Pay), and even my work life (Gmail, Calendar, Docs). This breadth of insight uniquely positions Google to deliver truly personalized AI experiences.

Google didn’t just launch new products at I/O, it made deliberate moves into markets long held by OpenAI, Meta, Perplexity, Anthropic, and even Shopify and Stripe. Each announcement, from Jules to Gemini Live, stepped directly into competitive territory. If you’re working on dev tools, agent platforms, creative apps, e-commerce flows, or voice interfaces, these updates are worth reading. I’ve included a breakdown of the most directly affected companies and industries at the end of this post—worth reviewing if you want to stay ahead of what’s coming.

The real story isn’t about how many features Google shipped, though. It’s about the strategy taking shape. Google is doubling down on vertical integration and deeply contextual AI. That’s the new game. In Ben Thompson (Stratechery) terms, it’s Aggregation Theory with agency. Google owns the user interface, the distribution (Android, Chrome, Search), and now, increasingly, the intelligence layer.

In this post, I'll outline a selected subset of announcements I found most promising and share my 2¢ on why this event marks a turning point in AI's evolution.

Google AI Studio, Jules, and Stitch

Perhaps one of the most significant announcements at Google I/O was unveiling the upgraded Google AI Studio, with a whole new Build section—an integrated development environment explicitly designed for building AI-driven applications.

Positioned directly against IDEs like Cursor, Windsurf, Lovable, and Bolt, Google AI Studio unifies Google's flagship multimodal Gemini models into one streamlined interface. Developers now have the ability to build and deploy their creations using natural language and with a single click to Google Cloud, reinforcing Google's strategic advantage through infrastructure integration.

Jules, a particularly intriguing release, is Google's take on the autonomous coding agent, similar to the likes of Devin and Factory. Quietly entering public beta at jules.google, Jules represents Google's ambitions to dominate the software development lifecycle: from writing documentation and deploying applications to autonomously submitting pull requests. Though overshadowed by flashier announcements, Jules may well emerge as a sleeper hit among developers seeking highly efficient, AI-augmented development workflows.

Stitch, another groundbreaking tool revealed at I/O, could radically simplify UI design processes. Through natural language prompts, designers can describe interfaces, which Stitch then generates and exports directly into Figma.

Together, Google AI Studio, Jules, and Stitch exemplify Google's strategy of leveraging its state-of-the-art models and infrastructure to deliver highly integrated, practical, and transformative tools for developers and designers alike.

Powerful models

Gemini 2.5 took center stage at I/O, outperforming nearly every major AI benchmark: from coding and web development to complex reasoning and video understanding. Compared to leading commercial models, it stands out with a January 2025 knowledge cutoff, a 1 million-token context window, and operates at around a quarter of the cost of OpenAI’s GPT-4o.

Key improvements include:

• Deep Think - an advanced reasoning capability, achieving state-of-the-art results in complex mathematical and programming tasks in exchange to increased cost and latency.

• Enhanced function calling and Structured Outputs - until now, the real-time Gemini models haven’t been usable for anyone needing function calling or structured output. Now, it’s finally fixed.

• Gemini Diffusion - Google unveiled Gemini Diffusion, generating text 5x faster than the leading Flash Lite model. This advancement is powered by recent research utilizing diffusion models for text generation, marking a significant leap forward in efficiency and responsiveness.

Search & AI Mode

Google has been experimenting with a new way of search over the last few weeks, dubbed “AI Mode”. This new mode just got generally available in the US last Tuesday. Powered by the Gemini 2.5 Pro model, AI Mode allows users to engage in multi-turn dialogues, enabling more complex and nuanced information retrieval.

Highlights from the new search experience:

• Personal Context - for an even more customized experience, AI Mode will offer personalized suggestions based on your connected Google apps, starting with Gmail, to bring in more of your personal context. For example, if you’re searching for “things to do in Nashville this weekend with friends, we're big foodies who like music” ahead of an upcoming trip, AI Mode can show you restaurants with outdoor seating based on your past restaurant bookings and searches.

• Agentic Checkout - streamlining the purchasing process by allowing users to complete transactions directly within Search, bypassing the need to navigate to third-party websites. For example, when searching for concert tickets, AI Mode will find the best options and facilitate the purchase through Google Pay, all within the same interface. This seamless integration has the potential to disrupt traditional e-commerce models and reshape how users interact with online marketplaces. I wrote a whole series on the new agentic internet!

• Try It On - enhancing the virtual shopping experience, Google's "Try It On" feature utilizes Google’s strong image generation diffusion models to allow users to visualize clothing items on themselves. Users can upload their picture using Google Photos and see how different garments would look on their own bodies. Google’s generative AI capabilities meet distribution (Google Photos).

• Deep Search - by synthesizing information from multiple sources, AI Mode can provide comprehensive answers to multifaceted questions, making Google Search relevant again in the face of competing tools such as OpenAI’s and Perplexity’s Deep Research.

The revenue gamble

While Google's AI Mode represents a significant leap forward in search capabilities, it also reveals a fundamental tension at the heart of the company's strategy. Google is essentially betting against its own golden goose: the advertising-driven search model that has generated over 50% of its revenues for over two decades.

The math is straightforward but concerning: if AI Mode provides comprehensive answers directly within search results, users will click through to fewer websites. Independent studies already suggest this trend with AI Overviews, and AI Mode's conversational interface offers even fewer opportunities for traditional paid link placements. Google's executives at I/O spoke confidently about the technical capabilities of their new search experience, but when it came to discussing how this translates into sustainable revenue streams, the answers were notably vague.

This isn't just a minor product pivot, it's a fundamental reimagining of how Google makes money. The company appears to be racing toward a future where AI assistants and conversational interfaces replace link-based search, and while there are certainly ways to imagine business models around personalized AI assistants and agentic workflows, Google hasn't articulated what those might look like or how they'll replace the massive cash flows from traditional search advertising.

Project Mariner

Project Mariner is Google's step toward giving AI true agency across your devices. It’s their answer to OpenAI's Operator and Anthropic's Computer Use. An infrastructure-level system for teaching AI to interact with your digital environment just like a human would.

At its core, Mariner is about "teach and repeat". Show Gemini how to perform a task: filling out a form, generating a weekly status report, uploading data to a dashboard, and it can replicate that workflow again and again.

Mariner will be released as part of the Gemini API later this summer, which means developers can build agents that don’t just reason and plan, but act: navigating apps, automating browser actions, and manipulating on-screen interfaces.

Whether it’s booking a flight, copying events into a spreadsheet, or handling repetitive workflows across company tools, Mariner helps AI move beyond suggestions and into action.

Gemini app and Gemini Live

With the new Gemini app and its Live feature, Google is officially entering the race for the "everything AI assistant”, a direct challenger to ChatGPT, Meta AI, and Apple Intelligence.

The Gemini app is no longer just a chatbot. It’s a real-time, context-aware assistant that lives across your devices and ties directly into Google’s ecosystem: Gmail, Calendar, Keep, Docs, Maps, and even YouTube. Thanks to its tight OS-level integration (powered by Project Mariner), Gemini can also take actions on your phone.

But what really sets Gemini apart isn’t just input, it’s output:

• Search Live and Project Astra - building on the capabilities of AI Mode, Google introduced Search Live, a feature that combines real-time camera input with search functionality. Users can point their device's camera at an object or scene and receive immediate information (similar to OpenAI’s Advanced Voice Mode), effectively turning their environment into an interactive search field. This feature is powered by Project Astra, Google's multimodal AI assistant that integrates visual and auditory data to provide contextually relevant responses.

• Canvas is Google’s answer to tools like OpenAI’s Canvas and Anthropic’s Artifacts. Ask Gemini to summarize an article and it will build an interactive webpage, infographic, quiz, or even a lightweight app.

• Deep Research now supports uploaded personal files, synthesizing them into study guides, plans, or insights, connecting directly to your Drive and Gmail, offering context-rich reasoning grounded in your data.

• Agent Mode enables task automation across Gmail, Calendar, and partner services like Zillow. Unlike a basic plugin system, this builds on Mariner’s deeper Android-level control and Google's new MCP support, enabling multi-step reasoning and actions.

• Quiz and Video Generation taps into Veo (text2video) and Lyria (music generation model), turning documents into test prep material and short videos.

Generative models for creatives

Google’s generative media stack is finally starting to feel competitive.

• Veo 3 is their new text-to-video model - high-quality, photorealistic footage, now with native audio generation. Think Pika or Runway, but with better motion, longer clips, and built-in sound.

• Image 4 improved with sharper details, better text rendering, and is now integrated into Gemini.

• Lyria 2 is Google’s music generation model. Based on the demo, Lyria is still in its infancy and far from the quality of Suno and Udio.

• Flow is a new AI-powered video editor. Type a prompt, get an 8-second clip. Stitch clips together, tweak scenes with natural language. It’s Google’s answer to creative environments like Adobe Premiere, but for AI-native workflows.

Taken together, this is Google’s most serious push yet into generative video, music, and imagery, accessible via Google AI Studio and the Gemini API.

Google AI Glasses

Twelve years after the original Google Glass flop, Google’s trying again, and this time, it looks promising.

Google unveiled a new pair of smart glasses powered by Android XR and deeply integrated with the Gemini model family. They come equipped with microphones, speakers, a camera, and an in-lens display, offering a level of interactivity that goes beyond Meta’s Ray-Ban, which don’t have a display. Google is going a step further: your real world now comes with real-time captions, directions, translations, and a personal assistant whispering relevant information.

And that’s the key difference: Google has the phone and app distribution. Meta and OpenAI with its ChatGPT consumer app do not. That means Google can natively integrate with Gmail, Calendar, Maps, Docs, Translate, and YouTube—capabilities that come pre-installed on Android and are used by billions. Need to translate a live conversation? Snap a photo and auto-organize it? Navigate to a meeting while rescheduling the next one? All of that is now on your face.

To get there, Google partners with Gentle Monster and Warby Parker for manufacturing, echoing the Meta + Ray-Ban strategy.

If you’re thinking this sounds like something Ben Thompson would write a thousand-word piece about, you're not wrong. This is exactly the kind of vertical integration that makes Apple and others sweat: powerful native models, fused with real-time inputs (voice, vision), and paired with a ubiquitous OS.

The world was not ready for wearable AI in 2013. But in 2025, with AI-native operating systems and mainstream model adoption, and after Meta has proven market traction, Google may have found the perfect moment for a comeback.

Industry impact

So what does all this mean if you're not Google? Below is a breakdown of the major announcements from I/O and the companies most likely to feel the heat.

Read more

Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Computer-Use Agent

Running OpenAI’s computer-use model just got a whole lot easier. A new open-source package called Computer-Use Agent (Cua) lets you run OpenAI’s computer-use-preview model inside a full-featured macOS virtual machine.

Until now, using OpenAI’s computer-use model meant working with limited APIs or browser-based sandboxes (like ChatGPT Operator). With Cua, you can:

1. Interact with native apps like Finder, Terminal, and Final Cut Pro

2. Automate real desktop workflows, not just web tasks

3. Run everything locally for better privacy and control

4. Avoid the pain of wiring screenshots, actions, and VM interfaces manually

Under the hood, Cua uses Apple’s Virtualization framework to launch macOS VMs on Apple Silicon and runs an event loop that handles clicks, typing, scrolling, and more — all based on OpenAI’s structured responses.

With Cua, you can build an AI agent that files your expenses across desktop and browser apps — moving between Excel, Chrome, and system dialogs, or launch a self-healing QA bot that installs your macOS app, navigates its UI, and reports bugs automatically.

If you're experimenting with OS-level agents, GUI automation, or reinforcement learning on real UIs, Cua provides the missing infrastructure.

GitHub repo https://github.com/trycua/cua

Portia

Portia just open-sourced a powerful agent framework that solves the three biggest challenges of production AI agents.

The new open-source library addresses the core problems that plague most agentic frameworks with a refreshingly practical approach: separating agent planning from execution, allowing clear human oversight and structured collaboration at every step.

Package highlights:

1. Multi-agent planning - the SDK uses few-shot prompting to teach your agents what successful plans look like, significantly boosting reliability.

2. Stateful execution - agents track their own progress and proactively request human input whenever necessary. Think: authentication requests, missing data, or asking for missing context when task execution hits an unexpected scenario.

3. Streamlined security - just-in-time authentication handovers ensure your agents can securely interact with popular tools like Google Calendar, Zendesk, and Hubspot without compromising credentials

This architecture solves persistent roadblocks like unpredictable behavior, lack of human oversight, and cumbersome authentication processes, making production-ready agent deployment realistic and scalable.

It's open-source, production-ready, and works out-of-the-box with major LLM providers including OpenAI, Anthropic, Mistral, Gemini, and Azure.

GitHub repo https://github.com/portiaAI/portia-sdk-python

My recent post on coding with AI

Goose - LLM-powered Agents

Block recently launched Goose, an open-source framework for developers building AI agents, and I got to tinker with it the last few days.

Goose is a free open framework for LLM-powered agents, from DeepSeek to proprietary models from OpenAI, Google, or Anthropic.

Unlike other agent frameworks, Goose is designed for software development tasks. The framework has already proven valuable for tasks like:

1. Conducting code migrations - from Ember to React or Ruby to Kotlin

2. Navigating new projects in unfamiliar languages - eliminating steep learning curves

3. Generating unit tests - quickly increasing code coverage above specific thresholds

The neat thing about Goose is that it's extremely easy to extend its capabilities by leveraging MCP servers like Figma, Google Drive, and Asana.

Other notable frameworks for coding tasks include OpenHand and Cline.

GitHub repo https://github.com/block/goose

Andrej Karpathy leverging LLMs

Read more

In this post, I’ll share the principles and tactics that helped me go from chaotic vibe coding sessions to consistently building better and faster with AI. Subscribe to get notified when the next post is published.

A NotebookLM-powered podcast episode discussing this post:

In today’s “vibe-coding” world, where tools like Cursor, Windsurf, Cline, and Claude Code can autocomplete an entire feature before your coffee cools, speed is no longer the bottleneck. Clarity is. Early on, I mistook these agents for magical refactor engines: give them a prompt, lean back, wait for perfect code. Instead, I got files rearranged beyond recognition, subtle bugs, and a creeping sense that I was pair-programming with an eager intern running on double espresso shots. Not great.

Through experimentation, failures, and continuous learning, I discovered a critical insight: treating your AI coding agent like a highly capable human pair programmer dramatically improves performance. Just as clarity and thoroughness are crucial when collaborating with human teammates, providing explicit context and structured guidance to AI coding agents is equally essential.

In this post, I’ll walk you through the principles and tactics that transformed my chaotic vibe coding sessions into a consistent, faster, and higher-quality AI-assisted workflow.

The first three tips are foundational. They’ll show you why a successful vibe coding session is roughly 80% planning, 20% execution.

#1 Wear the Product Manager hat

The single biggest unlock I’ve found is to treat the project the way a seasoned PM would—before the first line of code is generated. A couple of hours of purposeful “spec-ing” gives the AI (and yourself!) all the needed context and saves you days of refactors later.

Create a Product Requirements Document (PRD)

Start every project with a prd.md file in the root directory explaining what you’re building, why, the user flows, in-scope / out-of-scope items, and a short tech-stack overview.

I have a PRD template I follow for my projects and you should have one, too. To save time generating it, I often use ChatGPT’s built-in Whisper to dictate my raw notes and then ask it to transform those notes into a PRD using my template. I also ask ChatGPT to surface any missing parts or context in my PRD before copy-pasting the output into the project’s prd.md file.

Then, add a rule to .cursor/rules or .windsurfrules (more on this below), encouraging the AI agent to “always read prd.md before writing any code”.

Break the knowledge base into bite-sized docs

For most side projects, one prd.md file is enough, but once it grows, split it:

• app_flow.md - wire-frame-level tour of every page/endpoint and transitions from one page to another

• db_schema.md - canonical source of truth for tables, relations, enums

• tech_stack.md - chosen libs, versions, style guides, links to API docs

• implementation_plan.md -step-by-step build checklist

Organize these files within a project-docs directory. Update your AI agent's rules to include:

Refer to all documents in the /project-docs directory for context before proceeding with code generation.

This modular approach provides your coding agent with the much-needed context to build your project correctly. I’ve consistently seen how a well-structured and documented project-docs directory leads to faster and better coding.

Feed the agent real artifacts

Drag-and-drop your Figma export, Swagger spec, or PDF PRD right into the chat (Cursor, Windsurf, etc. can ingest files). The richer the context graph, the fewer guesses the model makes.

Remember: the AI is a bright but literal teammate

I spend a whole day up front writing the product doc, test flows, and API contracts.

By adopting a product manager's mindset and providing comprehensive, organized documentation, you set your AI coding agent up for success. This structured approach minimizes misunderstandings and streamlines the development process, allowing you to build more effectively and efficiently.

Become a premium member to access the full LLM Builders series, $1k in free credits for leading AI tools and APIs (Claude, Hugging Face, Deepgram), and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

#2 Planning with frontier models

As models proliferate (Gemini 2.5 Pro, Claude 3.5, o3, o4, GPT-4.1), developing intuition for which model to employ when becomes crucial for efficiency. Think of your AI stack like a toolbox. Different tools for different jobs.

Most developers, including myself, use different models for various phases of the development cycle. For the planning phase, leverage advanced reasoning models like Claude 3.7 Sonnet or o3 to write the PRD and generate implementation plans. These models excel at complex reasoning and can produce comprehensive, well-structured documentation that serves as the foundation for your project.

Recently, I discovered Anthropic's "ultrathink" technique: appending this keyword to your prompt signals Claude to allocate more thinking budget to your request. This results in more thorough plans at the expense of higher latency, which is a worthwhile tradeoff for critical planning sessions.

For instance, when recently developing an evaluation framework for voice agents, I used Claude 3.7 to architect the entire system. I prompted:

Based on these project requirements and the existing evaluation metrics in my /project-docs folder, generate a comprehensive implementation plan for building a voice agent evaluation framework that simulates conversations and scores performance across multiple dimensions. Include conversation flow design, metrics calculation logic, and reporting structure. ultrathink

The resulting implementation plan was exceptionally thorough. It proposed a conversation simulator with configurable user personas, identified seventeen distinct evaluation metrics (including non-obvious ones like 'recovery from misunderstanding'), suggested a weighted scoring system that accounted for business priorities, and outlined a modular architecture allowing for easy addition of new evaluation criteria as voice agents evolve.

After reviewing and refining the initial plan, addressing any missing pieces or gaps the LLM overlooked, I transition to different models for implementation: Gemini Pro 2.5 for most code generation, with Claude 3.5 and GPT-4.1 reserved for coding tasks that are local, e.g. generating/fixing a small function. Claude 3.7 and similar advanced models tend to overextend, introducing extra suggestions that require additional cleanup, so I avoid them for scoped changes.

#3 The building block approach - break tasks into atomic components

Rather than overwhelming yourself and your AI pair programmer with an entire project at once, think of your development process as assembling LEGO blocks. Each component should be well-defined, independently testable, and have clear inputs and outputs.

Start a fresh chat with your AI agent (⌘ + I) for each component to maintain focus and prevent context contamination. This approach not only helps the AI generate more precise code but also makes debugging and integration significantly easier.

For relatively self-contained components, I've found tremendous success using a separate development environment:

1. Open a new Cursor instance in an empty project directory

2. Copy only my project-docs folder into this environment

3. Ask the AI to build just that new component I need

4. Test the component thoroughly in isolation

5. Integrate the polished component back into the main project

A recent example from my voice agent evaluation platform mentioned earlier: I needed a component to transform raw JSON evaluation results into an interactive HTML dashboard displaying performance metrics. Rather than complicating matters by building within the already complex codebase, I isolated this visualization task entirely by generating the code in a new isolated environment and copying it back once done.

This isolation technique works especially well for visualization components, data transformation utilities, API clients, custom algorithms, and reusable UI elements.

Remember: the smaller and more focused the task, the higher the quality of the AI-generated solution.

#4 Use git for version control as a safety net

Git serves as an indispensable safety net, allowing you to track changes, revert to previous states, and understand the evolution of your project.

Tools like Cursor and Windsurf are powerful, but their change tracking interfaces are somewhat clunky, and as models get more aggressive about editing larger parts of your codebase, the more overwhelming such comparison interfaces will become.

Frequent commits act as checkpoints in your development journey. By committing often, you create clean, incremental snapshots of your project, making it far easier to review changes, spot unintended edits, and quickly revert mistakes.

What makes committing easier is Cursor’s and Windsurf’s built-in “Generate Commit Message” button. This feature analyzes your changes and automatically drafts a descriptive commit message, transforming what used to be a chore into a one-click process.

#5 AI IDE rules (.cursor/rules or .windsurfrules)

One of the highest-leverage moves you can make when working with AI coding agents like Cursor or Windsurf is to define clear, project-specific rules.
Think of rules as your agent’s operating manual: the clearer it is, the better your results.

At the start of every project, create a concise rules file, .cursor/rules or .windsurfrules, that guides how your AI pair programmer should behave. These rules act as a standing context layer that supplements every prompt, saving you from endless re-explaining and firefighting.

Here’s what great rules include:

• Your project-specific guidelines such as “Always prefer strict types over 'any' in TypeScript”

• Tech stack details - tell your agent what you're using: Flask, FastAPI, Supabase, SQLite, etc. so it stops making incorrect assumptions

• Known pitfalls and fixes - add proactive corrections here if you notice recurring errors (e.g., Cursor defaulting to the wrong Python version)

• High-level project overview - summarize the purpose, major functionalities, and key files (you can link to your prd.md here, see previous section).

Use .md reference files alongside rules for better modularity. For instance, we once spent a full day as a team designing our testing philosophy, captured it in test-guidelines.md, and added a simple rule: “Refer to test-guidelines.md when writing tests.”

Another rule that dramatically boosted my high-stakes coding sessions:

For any complex or critical task, ask any and all clarification questions needed before proceeding.

Cursor recently introduced a /Generate Cursor Rules command, allowing you to instantly create new rules based on the current chat context, which is particularly useful after a significant architecture change or a project refactor.

Extra resources:

• Browse and reuse existing community rules: Cursor Directory and Playbooks Rules

• Build your own rules easily: Playbooks Rule Builder

#6 Generate an MCP server for any API in two commands

Read more

Each month, I'll share my five seven top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

This post covers groundbreaking developments in AI agents and document processing, from Anthropic's foundational patterns for building effective agents to LlamaIndex's new Agentic Document Workflows. You'll learn about DeepSeek's surprising findings about prompting reasoning models, cutting-edge tools for PDF processing and web automation, and explore how LLMs handle structured table data.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. GOT-OCR 2.0

I finally had the chance to explore a new document extraction technique introduced in a paper last September. Bonus: the code and model are free to use (Apache 2.0).

This new approach, called General OCR Theory (GOT-OCR2.0), suggests a unified end-to-end model that handles tasks traditional OCR systems struggle with.

Unlike legacy OCR, which relies on complex multi-modular pipelines, GOT uses a simple encoder-decoder architecture with only 580M parameters that outperforms models 10-100× larger.

Paper highlights:

1. Unified architecture - a high-compression encoder paired with a long-context decoder that handles everything from scene text to complex formulas

2. Stunning performance - delivers nearly perfect text accuracy on documents, surpassing Qwen-VL-Max (>72B) and other leading models

3. Versatility beyond text - processes math formulas, molecular structures, and even geometric shapes

4. Interactive capabilities - supports region-level recognition guided by coordinates or colors

I just tried it out and was blown away by how it handles complex documents with mixed content types. The ability to convert math formulas from Arxiv PDFs to Mathpix format alone is worth exploring this model.

What strikes me most about GOT is how it challenges the notion that only billion-parameter LLMs can tackle complex visual tasks.

Paper + code + model can be found in their GitHub repo https://github.com/Ucas-HaoranWei/GOT-OCR2.0

Last month’s LinkedIn Highlights

2. Six Levels of Agenic Behavior

I came across a new framework that brings clarity to the messy world of AI agents with a 6-level autonomy hierarchy.

While most definitions of AI agents are binary (it either is or isn't), a new framework from Vellum introduces a spectrum of agency that makes far more sense for the current AI landscape.

The six levels of agentic behavior provide a clear path from basic to advanced:

𝐋𝐞𝐯𝐞𝐥 0 - 𝐑𝐮𝐥𝐞-𝐁𝐚𝐬𝐞𝐝 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰 (𝐅𝐨𝐥𝐥𝐨𝐰𝐞𝐫)
No intelligence—just if-this-then-that logic with no decision-making or adaptation. Examples include Zapier workflows, pipeline schedulers, and scripted bots—useful but rigid systems that break when conditions change.

𝐋𝐞𝐯𝐞𝐥 1 - 𝐁𝐚𝐬𝐢𝐜 𝐑𝐞𝐬𝐩𝐨𝐧𝐝𝐞𝐫 (𝐄𝐱𝐞𝐜𝐮𝐭𝐨𝐫)
Shows minimal autonomy—processing inputs, retrieving data, and generating responses based on patterns. The key limitation: no control loop, memory, or iterative reasoning. It's purely reactive, like basic implementations of ChatGPT or Claude.

𝐋𝐞𝐯𝐞𝐥 2 - 𝐔𝐬𝐞 𝐨𝐟 𝐓𝐨𝐨𝐥𝐬 (𝐀𝐜𝐭𝐨𝐫)
Not just responding but executing—capable of deciding to call external tools, fetch data, and incorporate results. This is where most current AI applications live, including ChatGPT with plugins or Claude with Function Calling. Still fundamentally reactive without self-correction.

𝐋𝐞𝐯𝐞𝐥 3 - 𝐎𝐛𝐬𝐞𝐫𝐯𝐞, 𝐏𝐥𝐚𝐧, 𝐀𝐜𝐭 (𝐎𝐩𝐞𝐫𝐚𝐭𝐨𝐫)
Managing execution by mapping steps, evaluating outputs, and adjusting before moving forward. These systems detect state changes, plan multi-step workflows, and run internal evaluations. Examples like AutoGPT or LangChain agents attempt this, though they still shut down after task completion.

𝐋𝐞𝐯𝐞𝐥 4 - 𝐅𝐮𝐥𝐥𝐲 𝐀𝐮𝐭𝐨𝐧𝐨𝐦𝐨𝐮𝐬 (𝐄𝐱𝐩𝐥𝐨𝐫𝐞𝐫)
Behaving like stateful systems that maintain state, trigger actions autonomously, and refine execution in real-time. These agents "watch" multiple streams and execute without constant human intervention. Cognition Labs' Devin and Anthropic's Claude Code aspire to this level, but we're still in the early days, with reliable persistence being the key challenge.

𝐋𝐞𝐯𝐞𝐥 5 - 𝐅𝐮𝐥𝐥𝐲 𝐂𝐫𝐞𝐚𝐭𝐢𝐯𝐞 (𝐈𝐧𝐯𝐞𝐧𝐭𝐨𝐫)
Creating its own logic, building tools on the fly, and dynamically composing functions to solve novel problems. We're nowhere near this yet—even the most powerful models (o1, o3, Deepseek R1) still overfit and follow hardcoded heuristics rather than demonstrating true creativity.

The framework shows where we are now: production-grade solutions up to Level 2, with most innovation happening at Levels 2-3. This taxonomy helps builders understand what kind of agent they're creating and what capabilities correspond to each level.

Full report https://www.vellum.ai/blog/levels-of-agentic-behavior

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

3. Skyraven

Traditional web automation is dying as developers waste countless hours maintaining brittle XPath selectors. Skyvern, a new open-source package, revolutionizes browser automation by combining LLMs with computer vision.

Unlike traditional automation tools that break when websites change, Skyvern uses visual understanding and natural language processing to dynamically interpret and interact with web interfaces. This enables developers to:

→ Build website-agnostic automations - create workflows that work across multiple sites without custom code

→ Handle complex inference tasks - automatically reason through form responses like eligibility questions

→ Execute multi-step sequences - coordinate multiple agents for tasks like authentication, navigation, and data extraction

Packages like Skyvern signal the emergence of truly adaptable web agents. Instead of hard-coded rules, we see AI systems that can understand and navigate the web like humans do - reading content, making decisions, and handling edge cases autonomously. I wrote more about it in my latest AI Agents blog series.

GitHub repo https://github.com/Skyvern-AI/skyvern

4. Maestro

Read more

This post is about the unprecedented shift happening in voice AI interfaces and what it means for human interaction. As these new capabilities become accessible through simple APIs, a massive opportunity is emerging for founders to build products that reimagine how we communicate with technology and each other.

A NotebookLM-powered podcast episode discussing this post:

June 2025. Sarah paces in her living room, rehearsing an important client presentation. Her AI companion listens intently, chiming in when relevant to offer real-time feedback on her delivery and content. "I think you rushed through the ROI section," it suggests in a warm, natural voice. "Let's try that part again, but this time—" Sarah cuts in mid-sentence, "Actually, can we focus on the opening first? And don't be so nitpicky!" The AI smoothly adjusts, without awkward pauses or robotic transitions. What was once a frustrating experience of rigid, unnatural interactions with voice assistants has evolved into fluid, human-like conversation.

I've spent considerable time lately thinking about and building in the voice AI space, and something unprecedented is emerging: for the first time in history, we have real-time, affordable, and competent artificial voice that's just one API call away. In just a few months, we've seen significant leaps forward from the likes of OpenAI’s Advanced Voice Mode (AVM) and new speech models, Google’s real-time conversational Gemini Flash, and Sesame’s emotionally intelligent AI1.

This isn't just a technical milestone—it's a fundamental shift in how we interact with technology and, potentially, with each other. It will create numerous new opportunities for builders while redefining the very nature of human communication.

Gavin Purcell is arguing with Sesame’s realtime voice AI 👆

The dawn of natural voice AI

Remember the last time you called your bank's automated system? The familiar dance of repeated phrases, misunderstood words, and the desperate pressing of "0" to reach a human operator. That era is ending. OpenAI's release of Advanced Voice Mode (AVM) last September marked a pivotal moment when conversing with AI began to feel genuinely human.

This transformation stems from two key breakthroughs. First, the shift from cascading architectures (speech-to-text → text processing → text-to-speech) to direct speech-to-speech models eliminates intermediate processing stages that previously slowed conversational AI interactions. Second, the dramatic reduction in latency and cost. When OpenAI initially released its Realtime API, the price made it impractical for widespread adoption (18$/hour). But just four months later, Google's release of Gemini Flash 2.0 and OpenAI's 60% price reduction opened the floodgates for affordable and human-like voice AI applications that are one API call away.

Just last week, OpenAI unveiled its most human-like speech models yet, enabling developers to embed expressive cues like [WHISPERING] or [LAUGHING] directly into the text. Here's a quick demo from OpenAI.fm—a public tool launched alongside this release, showcasing what this new level of expressiveness sounds like in action:

Builders can now launch phone assistants that qualify sales leads, resolve customer support calls, automate insurance sales, or screen patients before their upcoming appointments. The necessary tools are already available and are just a single API call away.

The interruption problem

However, building truly natural voice interactions isn't just about faster processing and better voice synthesis. One of the most fascinating challenges lies in handling interruptions—a fundamental aspect of human conversation that AI still struggles with.

Current voice AI systems, including the ones mentioned like OpenAI’s AVM, face several key challenges:

1. Oversensitivity to background noise (I always mute myself when not speaking)

2. Inability to distinguish between relevant speakers and ambient conversation

3. Lack of visual cues that humans use to anticipate and manage interruptions

Unlike human phone conversations, where near-zero latency and natural turn-taking make interruptions manageable, AI interactions often feel clunky when users try to interject2. Interestingly, humans tend to interrupt AI more frequently and aggressively than they would other humans, creating a new challenge for voice AI developers while creating a new interaction paradigm for human-AI conversation.

The social impact

This voice revolution raises profound questions about human interaction and relationships:

• Could the instant gratification of interruptible AI conversations and the ability to be rude without consequences degrade our patience and interpersonal skills, similar to how ubiquitous access to pornography has distorted societal expectations around intimacy?

• The convenience of always-available AI consultation might reduce our reliance on human relationships. Consider how we once relied on reading maps and asking locals for directions—skills now largely abandoned as we defer to GPS. Could meaningful conversations be next?

• Could we soon have more conversational exchanges with AI agents than with human companions?

Think: Would you rather rehearse a high-stakes presentation in front of a potentially judgmental friend or instantly consult a non-judgmental AI companion available 24/7?

What does this mean for our interpersonal relationships?

Cultural nuances in AI conversation

One size doesn't fit all in human conversation, and the same is true for AI. OpenAI's recent update of GPT-4o to GPT-4.5 was mainly about moving away from its "corporate HR" tone, recognizing that natural conversation varies significantly across cultures and contexts.

Different cultures have distinct interruption patterns, politeness norms, and conversation styles. Today's systems largely fail to account for these cultural differences, creating a significant opportunity for AI builders to develop models that adapt to:

• Cultural background

• Individual user patterns

• Contextual cues

• Historical interactions

OpenAI already possesses such context through its Memory feature, and Google, of course, knows virtually everything about us already.

I imagine the best conversational AI systems of the future will incorporate nuances that we take for granted.

Rethinking communication

The holy grail for conversational AI might be achieving the natural flow of a phone call between humans, where interruptions feel natural and turn-taking is seamless. But perhaps we need to aim higher. As AI systems gain multimodal capabilities (vision, touch, etc.), they could potentially surpass human conversation by reading subtle cues we often miss.

Figure's household robots learn tasks on the fly 👆

What surprises me most is how slowly Advanced Voice Mode is being adopted. Despite its impressive capabilities, many of my friends still default to typing or using Whisper (OpenAI's speech-to-text model) rather than having natural conversations with it. Perhaps this hesitation reflects our collective uncertainty about speaking naturally to machines, or simply a lack of awareness—after all, it only became available to free users last month, and many may not yet know how to use it. Either way, it suggests we're in an awkward adolescent phase of voice AI adoption—the technology is capable, but our habits and expectations haven't quite caught up.

The voice AI revolution isn't just about making machines sound more human—it's about fundamentally changing how we think about conversation, relationships, and human interaction. While we'll certainly see a proliferation of phone AI agents and computer assistants in the short term, there's a more profound transformation taking shape beneath the surface.

As we build these systems, we need to consider not just what's technically possible, but what's socially desirable. For now, it's clear that we're entering an era where the line between human and AI conversation is increasingly blurry—for better or worse.

To end on a lighter note, here’s a fun video of ChatGPT’s Voice Mode reimagining an alternate ending to Titanic.

Each month, I'll share my five six top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes six posts instead of five—there were just too many good ones to leave out!

This post covers groundbreaking developments in AI agents and document processing, from Anthropic's foundational patterns for building effective agents to LlamaIndex's new Agentic Document Workflows. You'll learn about DeepSeek's surprising findings about prompting reasoning models, cutting-edge tools for PDF processing and web automation, and explore how LLMs handle structured table data.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. Long Context Prompting Tips

Anthropic shared a counterintuitive yet powerful tip that boosts Claude's accuracy by 30% when working with long documents/prompts.

The secret? Place your lengthy documents (~20K+ tokens) at the TOP of your prompt before your actual query. While this might seem counterintuitive, internal tests show this simple change significantly improves response quality across all Claude models.

This becomes crucial when dealing with multiple documents. For optimal results:

1. Documents first - place all your data inputs at the beginning

2. Structured organization - use XML tags to separate documents and metadata

3. Specific query - end with a clear, focused question

As language models' context window grows in size and companies increasingly rely on LLMs to process complex datasets, reports, and documentation, this technique ensures more reliable and accurate results.

P.S. For those working with multi-document analysis, I highly recommend structuring your content with XML tags - it provides additional clarity and helps the model better understand document relationships.

2. Agents for Computer Use Repository

While everyone's talking about AI agents, most developers and researchers are missing out on the most comprehensive collection of computer use frameworks, papers, and tools ever assembled.

Awesome Agents for Computer Use is a curated repository documenting the recent rapid progress of AI agents that can autonomously control computers through clicks, keystrokes, and API calls. From Anthropic's Claude Computer Use to Microsoft's OmniParser and Self-Operating Computer framework, it covers the entire landscape of computer control agents.

It features:

• Research papers - featuring 30+ recent publications on GUI agents, from foundational models to safety considerations

• Open-source frameworks - documenting practical implementations like AutoGen, Browser Use, and OpenInterpreter

• Commercial solutions - tracking industry developments from major players like Anthropic and emerging startups

The rise of computer-controlling AI agents marks a pivotal shift in human-computer interaction. As these systems mature, we're moving towards a future where AI assistants won't just give advice - they'll directly help us accomplish complex tasks across applications and platforms (I wrote more about this topic here).

Repo https://github.com/francedot/acu

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

3. GPT-4.5 Pro Tip

Pro tip from the research team @ OpenAI on how to improve GPT-4.5 performance: add the following text to the start of your system message.

You are a highly capable, thoughtful, and precise assistant. Your goal is to deeply understand the user's intent, ask clarifying questions when needed, think step-by-step through complex problems, provide clear and accurate answers, and proactively anticipate helpful follow-up information. Always prioritize being truthful, nuanced, insightful, and efficient, tailoring your responses specifically to the user's needs and preferences.

OpenAI internal evals show it results in better performance.

Try it out in the OpenAI Playground https://platform.openai.com/playground/chat?preset=7CywXwBqWRC5quhkU9LEFv6A

4. Chorus - Chat with Multiple AIs on Your Desktop

Read more

Each month, I'll share my five eight top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes eight posts instead of five—there were just too many good ones to leave out!

This post covers groundbreaking developments in AI agents and document processing, from Anthropic's foundational patterns for building effective agents to LlamaIndex's new Agentic Document Workflows. You'll learn about DeepSeek's surprising findings about prompting reasoning models, cutting-edge tools for PDF processing and web automation, and explore how LLMs handle structured table data.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. Building Effective AI Agents by Anthropic

Anthropic clarifies the fuzzy definition of AI Agents by introducing a critical architectural distinction: workflows are systems with predefined code paths, while agents dynamically direct their own processes.

After building agents for a year, they identified five fundamental patterns that drive successful agentic implementations:

1. Prompt chaining - breaking tasks into sequential steps, useful for complex operations like content generation and translation

2. Routing - directing inputs to specialized handlers, perfect for customer service and model optimization

3. Parallelization - running subtasks simultaneously through sectioning or voting, ideal for code review and content moderation

4. Orchestrator-workers - using a central LLM to coordinate task delegation, essential for complex coding projects

5. Evaluator-optimizer - implementing feedback loops for iterative refinement, perfect for improving search results

Success isn't about building the most sophisticated system - it's about choosing the right pattern for your specific needs. Start simple, measure performance, and only add complexity when simpler solutions fall short.

Anthropic’s post (highly recommend reading it) https://www.anthropic.com/research/building-effective-agents

2. Agentic Document Workflows

LlamaIndex just unveiled a new approach involving AI agents for reliable document processing, from processing invoices to insurance claims and contract reviews.

LlamaIndex’s new architecture, Agentic Document Workflows (ADW), goes beyond basic retrieval and extraction to orchestrate end-to-end document processing and decision-making. Imagine a contract review workflow: you don't just parse terms, you identify potential risks, cross-reference regulations, and recommend compliance actions.

This level of coordination requires an agentic framework that maintains context, applies business rules, and interacts with multiple system components.

Here’s how ADW works at a high level:

1. Document parsing and structuring – using robust tools like LlamaParse to extract relevant fields from contracts, invoices, or medical records.

2. Stateful agents – coordinating each step of the process, maintaining context across multiple documents, and applying logic to generate actionable outputs.

3. Retrieval and reference – tapping into knowledge bases via LlamaCloud to cross-check policies, regulations, or best practices in real-time.

4. Actionable recommendations – delivering insights that help professionals make informed decisions rather than just handing over raw text.

ADW provides a path to building truly “intelligent” document systems that augment rather than replace human expertise. From legal contract reviews to patient case summaries, invoice processing, and insurance claims management—ADW supports human decision-making with context-rich workflows rather than one-off extractions.

Ready to use notebooks https://github.com/run-llama/llamacloud-demo/tree/main/examples/document_workflows

More open-source tools for AI agent developers in my recent blog post https://www.aitidbits.ai/p/open-source-agents

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

3. Vanna

Turning questions into SQL statements has been an area of research and development for over a decade. An open-source package called Vanna uses a retrieval augmented generation approach to crack the code.

Vanna is a Python-based, MIT-licensed framework that allows non-data folks to interact and ask questions about their SQL databases.

At its core, Vanna employs a RAG model that leverages a large corpus of data, including a diverse range of SQL queries and their natural language descriptions. When a query is received, Vanna searches this corpus to find similar queries and their corresponding SQL translations. This step enables Vanna to understand the context and structure of the query better.

Using the insights gained from the retrieved examples, Vanna generates the SQL query that matches the user's natural language request. This involves structuring the select statements, where clauses, joins, and other SQL components are based on the intent and requirements identified in the initial query.

Pro tip: Vanna often hallucinates when it doesn't know the content of your table's columns. I therefore recommend providing a few examples through the train() method.

The GitHub repo already includes ready-to-use templates to deploy Vanna in Slack, Streamlit, or a Flask endpoint https://github.com/vanna-ai/vanna

4. DeepSeek & How to prompt reasoning models

DeepSeek's new model doesn't just beat GPT-4o - it proves 𝐰𝐞'𝐯𝐞 𝐛𝐞𝐞𝐧 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥𝐬 𝐰𝐫𝐨𝐧𝐠 𝐚𝐥𝐥 𝐚𝐥𝐨𝐧𝐠.

Read more

A NotebookLM-powered podcast episode discussing this post:

December 2028. Maria's AI agent is negotiating simultaneously with twelve different vendors for her daughter's upcoming birthday party. Within minutes, it secured the perfect cake from a local bakery (after verifying their nut-free certification), booked an entertainer with stellar safety ratings, and coordinated custom goodie bags filled with each child's favorite treats (after checking allergies and dietary restrictions with the other parents' agents)—all while staying 15% under budget. What would have taken Maria hours of calls, emails, and anxiety about vendor reliability now happens seamlessly through a web of agent-to-agent interactions powered by the new infrastructure we explored in our previous post.

The revolution in web infrastructure we discussed in previous posts isn't just theoretical—it's enabling fundamental changes in how commerce, marketing, and customer service function. As agent passports and trust protocols become standardized, we're witnessing the emergence of entirely new commercial paradigms.

With the recent release of Tasks by OpenAI, which equips ChatGPT—its consumer-facing AI—with the ability to perform tasks behind the scenes on behalf of users, it's now easier than ever to envision a future where ChatGPT seamlessly navigates the web and handles complex operations for us.

Today, we'll explore how an agent-first internet will reshape domains like payments, marketing, support, and localization.

Agentic payments

Remember when online shopping first emerged, and entering your credit card details on a website felt risky? Card networks like Visa and Mastercard and banks like Chase and Barclays had to rapidly adapt to the digital realm, introducing new protocols like CVV codes and secure payment gateways to protect consumers and merchants alike. This evolution was pivotal in building trust and facilitating the e-commerce boom of the early 2000s, giving birth to digital payment giants like Stripe and PayPal.

Similarly, the payment infrastructure that powers today's internet was built around a simple dichotomy: card-present versus card-not-present transactions and the assumption that all payments are human-initiated. The internet already faced a similar shift with the introduction of subscription payments in the early days of e-commerce, in which customers input their payment credentials once, allowing a merchant to charge them repeatedly in the future. Recurring payments, now a dominant commerce mechanism projected to surpass $2.4T in spending by 2028, underscore how critical adaptive payment systems are to supporting global e-commerce. Yet, even more than a decade later, there are still countries like India that impose strict controls on recurring payments, highlighting the complexities of adapting systems to new paradigms.

But what happens when the legitimate transacting party isn’t human at all?

Digital payment systems are built around human users, employing measures like CVV codes and billing address verification to prevent fraud. These methods assume a human is initiating the transaction, making them ill-suited for autonomous agent transactions.

One major challenge is fraud detection and resolution—an area I intimately understand from my time as a PM at Stripe, where I worked closely with card issuers to develop sophisticated fraud prevention systems. The current paradigm relies heavily on human behavioral patterns: typing speed, mouse movements, time spent reviewing checkout pages, and other signals that indicate legitimate human activity. But in an agent-driven world, these signals become obsolete.

During my time at Stripe, we saw how crucial these behavioral signals were for differentiating legitimate transactions from fraudulent ones. The shift to agent-driven commerce demands an entirely new approach. Rather than looking for signs of human behavior, we'll need systems that verify agent legitimacy, authorization scope, and decision-making logic.

This shift raises critical questions for fraud prevention across banks (Chase, Barclays), card networks (Visa, Mastercard), and payment processors (Stripe, PayPal). For instance, how do you establish a chain of trust when an agent purchases on behalf of a user? How do you verify that an agent hasn't been compromised or hijacked? These challenges require fundamentally rethinking our approach to transaction security and fraud prevention in an agent-driven ecosystem.

Future payment systems could introduce 'agent wallets' with granular spending controls, such as $100 limits for trusted merchants like Amazon and stricter caps for lesser-known websites. These wallets would integrate real-time fraud detection, submitting cryptographic evidence for disputes and maintaining transparent, auditable records of agent actions tied to human authorization.

Visa could introduce an agent-specific flag to the existing payment protocols, indicating this payment was initiated autonomously, along with a trial of reasoning and actions leading to this payment. Meanwhile, Stripe might expand its SDKs to enforce programmable payment rules, ensuring alignment with user-delegated instructions (Stripe has already made its foray into Agentic payments with its recent SDK release).

Beyond preventing fraud, agentic payments face fundamental economic and infrastructural challenges. The existing payment infrastructure wasn't architected for the high-frequency, low-latency transactions that characterize agent interactions. Consider the standard pricing model of payment processors like Stripe: a 2.9% fee plus 30¢ per transaction. While manageable for traditional e-commerce, this fee structure becomes prohibitively expensive when scaled to the myriad micro-transactions that agents might need to execute.

This pricing isn't arbitrary—it reflects the complex web of stakeholders in the traditional payment chain. Card networks like Visa and issuers like Chase have built their business models around these transaction fees. Interestingly, Stripe's recent acquisition of Bridge, a stablecoin payment infrastructure provider, hints at a potential solution. By leveraging blockchain-based payment rails, companies could facilitate agent-to-agent transactions without incurring the expensive overhead of traditional payment networks. This move suggests a growing recognition that the future of payments may require completely new infrastructure, optimized for the unique demands of autonomous agents.

Consider how a new payment protocol might work in practice. An "Agent Payment Protocol" (APP) could include:

• Delegation Chain Verification - a cryptographic proof chain showing the agent's authorization to make specific types of purchases

• Transaction Context Object - machine-readable metadata including:

  • Reasoning trail that led to the purchase decision

  • Reference to specific user preferences/rules that were satisfied

  • Confidence score for the decision

• Smart spending controls with programmable constraints like:

  • Category-specific limits (e.g., $200 for groceries, $50 for entertainment)

  • Merchant-specific trust scores

  • Required human confirmation above certain thresholds

  

Major payment providers could implement this through extensions to existing standards. For instance, Visa's existing 3D Secure protocol could add an agent verification layer, while Stripe's API could introduce new parameters for agent-specific transaction metadata.

While payment infrastructure provides the foundation for agent-driven commerce, the very nature of how we complete transactions must also evolve. The familiar checkout process—a hallmark of e-commerce for decades—is about to undergo its own transformation.

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Redefined checkout experience

In an agent-first environment, the concept of a traditional checkout—where a human user confirms their purchase by clicking a “Buy” button—fades into the background. Instead, agents operate with predefined goals and parameters, continuously evaluating whether a proposed transaction aligns with those objectives. Rather than halting everything at a payment prompt, agents could integrate a “stop and reflect” step into their workflows. For example, if a user’s agent is tasked with booking a flight seat that’s both a window seat and close to an exit, the agent pauses before completing the reservation. It double-checks that the seat assignment matches the user’s criteria and only then proceeds, ensuring flawless execution of the user’s intent and mitigating the probability of the agent going off the rails due to hallucinations.

This reflective process transforms the final authorization into a subtle verification loop rather than a jarring user interruption. The agent reviews the selected attributes—price, seat location, baggage allowance, and cancellation policy—and compares them against the user’s stored preferences and constraints. It confirms not only that the requested outcome has been met but also that it falls within acceptable spending limits and trust parameters. This transforms purchasing from a manual “Are you sure?” prompt into a nuanced, data-driven decision matrix.

As these agent-mediated transactions proliferate, payment providers and merchants might offer additional layers of context-aware validation. For instance, when an agent chooses a specific insurance add-on, the payment system could prompt the agent to confirm whether its logic correctly interpreted the user’s needs. This transparent chain of reasoning, visible to the agent and logged for future reference, ensures that each transaction stands up to scrutiny. Ultimately, the checkout step evolves from a user-facing choke point to an agent-managed quality control measure, minimizing errors and elevating the overall integrity of automated commerce.

As agents reshape how we complete purchases, they're also forcing us to rethink how businesses attract and engage customers in the first place. The era of human-centric marketing campaigns is giving way to something far more systematic and efficient.

Agent-driven marketing and promotions

Marketing campaigns and promotions will evolve radically in an AI agent-mediated economy.

Traditional email marketing and coupon distribution systems, designed around human attention and impulse, will give way to programmatic offer networks where consumers’ AI agents maintain persistent queries about their principals' needs and preferences. These agents subscribe to vendor APIs that broadcast real-time offers matching specific criteria, enabling hyper-personalized deal discovery that transcends the limitations of batch email campaigns.

Vendors might maintain agent-first promotional channels that communicate in structured data formats, allowing instant price comparison and benefit analysis. This ecosystem could enable “intent casting", where agents broadcast shopping goals to trusted vendor networks, receiving precisely targeted offers that align with the principal's timing, budget, and preferences—all without cluttering a human inbox or requiring manual coupon management.

Consider a practical example: A user instructs their agent to monitor high-end fashion retailers for specific items within their style preferences and budget constraints. Rather than the user repeatedly checking websites or subscribing to countless email lists, their agent maintains persistent monitoring across multiple vendors:

• Real-time inventory tracking across size, color, and style variations

• Dynamic price monitoring, including flash sales and member-exclusive discounts

• Evaluation of shipping times and costs to the user's location

When ideal conditions align—perhaps a preferred sweater hits the target price point during an end-of-season sale—the agent can either notify the user or execute the purchase automatically based on pre-authorized parameters. This transforms shopping from an attention-demanding activity into an efficient background process governed by clear rules and preferences.

Major retailers like Nordstrom or ASOS could expose agent-specific APIs that provide structured access to:

• Real-time inventory and pricing data

• Detailed product specifications and measurements

• Membership program benefits and restrictions

• Regional availability and shipping constraints

This evolution democratizes personal shopping, allowing everyone to benefit from persistent, intelligent monitoring of their fashion preferences—not just those who can afford human personal shoppers. It also enables retailers to better match inventory with actual customer intent, reducing overhead from unsold merchandise and improving supply chain efficiency.

Other companies like Honey may need to pivot to offer agent-optimized tools that integrate directly with vendor APIs, allowing agents to query real-time discounts and rewards. Similarly, Mailchimp and HubSpot could develop agent-oriented campaign frameworks that distribute offers as structured data streams rather than traditional email blasts, ensuring seamless integration with agent-driven workflows.

Agent-native customer support

Customer support today centers on human interactions or user-facing chatbots. In an agent-first paradigm, this shifts to agent-to-agent communication. Personalized AI agents will directly engage with business systems to resolve issues, retrieve shipping or refund policies, or autonomously initiate returns. This evolution will streamline processes, reduce human intervention, and enhance efficiency in support workflows.

In an agent-first paradigm, customer support is no longer solely about human users contacting businesses through chat widgets or call centers. Instead, autonomous agents interact directly with enterprise systems, pulling diagnostic information, requesting refunds, or escalating complex issues to a more constrained/expensive resource like a human or a superior model (e.g., o1 over GPT-4o). This shift encourages platforms like Intercom to develop agent-oriented communication layers—specialized APIs that allow autonomous agents to navigate support options, retrieve knowledge base articles, and submit detailed troubleshooting requests without human intervention.

These agent-facing APIs would streamline issue resolution, allowing routine queries, such as package tracking, account verification, or policy clarifications, to be handled agent-to-agent, drastically reducing response times. As soon as a problem arises, the user’s agent can pinpoint the issue and connect with the business’s support agent (be it a specialized LLM or a human representative), negotiating resolutions or applying discounts as needed. The result is a fluid, automated dialogue that bypasses human frustration and latency.

Over time, companies could implement reputation scoring systems that measure how efficiently their support agents (both human and AI) interact with consumer agents. Metrics like resolution speed, policy clarity, and refund accuracy become machine-readable signals, informing user agents which vendors offer superior support experiences. As more vendors embrace these standards, the entire support ecosystem evolves: prompt, well-structured responses become the norm, and agent-native customer support becomes a hallmark of high-quality digital services.

Imagine a complex warranty claim scenario: Your agent detects that your new laptop's battery is degrading unusually fast. It immediately:

1. Collects diagnostic data and usage patterns

2. Cross-references warranty terms with actual performance

3. Initiates a support interaction with the manufacturer's agent

4. Negotiates a resolution based on precedent cases

5. Arranges shipping for replacement parts or full device replacement

6. Schedules a technician visit if needed

This entire process happens without human intervention unless exceptional circumstances arise. The interaction generates a complete audit trail, including all diagnostic data, communication logs, and decision points—valuable data for improving both product quality and support processes.

One example in this space of agentic customer support is Sierra, a startup taking aim at the expansive market of customer support by embedding AI agents into business workflows. Their conversational agents handle complex queries with contextual precision, managing tasks such as processing returns or updating subscriptions. While their primary focus remains on serving human customers, the foundation they’ve built is clearly aligned with an AI agent-driven future. With access to company policies (e.g., refund and shipping rules) and robust conversational AI infrastructure (spanning LLMs and voice interfaces), Sierra is well-positioned to seamlessly transition to support agent-to-agent interactions as demand evolves.

The end of language-optimized interfaces

As AI agents seamlessly translate and interpret information on the fly, the need for painstakingly maintained multilingual websites diminishes. Instead of forcing businesses to host separate English, French, or Mandarin versions of their interfaces, agents handle language conversion dynamically. This capability allows brands to maintain a single, streamlined codebase while ensuring that users, regardless of location, receive content and instructions in their preferred language—instantly and accurately.

Website builders like Webflow and Wix could evolve into platforms that generate “universal templates” optimized for agent interpretation rather than human linguistic preferences. Instead of focusing on localized landing pages, these platforms would produce standardized, machine-readable structures enriched with metadata and semantic cues. Agents, armed with cutting-edge language models, would then adapt the presentation layer for each user, including local dialects, cultural nuances, and even personalization cues drawn from the user’s profile.

The transformation goes beyond simple translation. Agents will handle complex cultural adaptations across multiple dimensions simultaneously. They'll dynamically adjust pricing strategies for different markets while modifying product descriptions to reflect local preferences and purchasing patterns. These agents will intelligently adapt imagery and design elements to ensure cultural appropriateness, automatically managing regional compliance requirements such as privacy policies or consumer protection disclosures. They'll even personalize communication styles based on cultural norms, shifting between formal and casual tones and adapting messaging cadence to match local expectations. This comprehensive cultural intelligence transforms what was once a labor-intensive localization process into a fluid, automated system that maintains cultural authenticity across all customer touchpoints.

In this new reality, the value proposition of website builders shifts from localization to robustness, structure, and data integrity. Rather than wrestling with manual translations or commissioning multiple language variants, businesses can rely on well-defined data schemas and agent-ready manifests. As a result, the concept of “language-optimized” sites becomes obsolete, replaced by fluid, dynamic interfaces that transcend linguistic barriers.

Small vs. large business impact

This transition creates both opportunities and challenges across the business spectrum. Large enterprises can invest in building sophisticated agent interfaces and maintaining complex agent-ready APIs. However, small businesses might initially struggle with the technical requirements and infrastructure costs.

To bridge this gap, we will likely see the emergence of "agent-enablement platforms"—services that help small businesses become agent-ready without significant technical investment. Think of them as the Shopify of the agent era, providing standardized tools that level the playing field. These platforms would offer pre-built solutions for creating agent-readable product catalogs and managing automated pricing and inventory systems. They would include standardized support protocols that small businesses can easily implement, along with simplified integration paths to agent payment systems. By democratizing access to agent-ready commerce capabilities, these platforms will play a crucial role in preventing a digital divide between large and small businesses in the agent economy.

This democratization of agent-ready commerce will be crucial for preventing a digital divide between large and small businesses in the agent economy.

The Great Rewiring

The transition to an agent-first internet represents more than just a technological shift—it's a fundamental reimagining of how commerce functions in the digital age. We're moving from a web optimized for human attention and interaction to one built for efficient, automated decision-making. This transformation touches every aspect of online business:

• Payment systems evolve from human-verification models to agent-oriented protocols with built-in delegation and accountability

• Marketing shifts from attention-grabbing campaigns to structured, machine-readable offer networks

• Customer support transforms from human-to-human interaction to efficient agent-to-agent problem resolution

• Language barriers dissolve as agent-mediated communication enables seamless global commerce

Companies that quickly adapt to this new paradigm—implementing agent passports, embracing agent-to-agent protocols, and restructuring their services for machine readability—will shape the next era of online interaction. Just as the mobile revolution created trillion-dollar opportunities, the agent revolution opens new horizons for innovation and value creation. The businesses that thrive won't just be those with the best products or prices, but those that best enable and embrace agent-driven commerce.

This is the third essay in a five-part series exploring the future of AI agents and their impact on the internet. Having established the foundational shifts and their practical implications across multiple domains, our next post will examine early case studies and implementation challenges as businesses begin this transformation. Subscribe and follow as we continue to navigate the technical, social, and economic implications of this new paradigm.

Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. MindSearch

An open-source search engine is rivaling top-tier AI products like Perplexity.ai Pro and ChatGPT web search.

MindSearch is an innovative AI search engine framework that combines LLMs and a multi-agent system to tackle three critical issues that often limit LLM-powered search engines:

1. LLMs struggle to decompose complex queries into simpler, actionable requests

2. Search results often contain too much noise, making it hard to filter and extract relevant information

3. Iterative searches can quickly overload the LLM’s input length capacity

MindSearch utilizes two main components:

• WebPlanner - decomposes complex queries into sub-tasks and creates a dynamic graph structure for problem-solving

• WebSearcher - conducts fine-grained searches and delivers summarized information back to WebPlanner for further refinement

This approach allows MindSearch to handle massive web content (e.g., more than 300 pages) effectively, surpassing limitations faced by traditional LLM-based search systems.

Code https://github.com/InternLM/MindSearch

2. Gemini Multimodal Playground

Holiday coding project: Build voice agents that can see with Google's new Gemini 2.0 model and my new real-time Multimodal Playground repo.

The playground implements voice and video-based interactions with the new Gemini model, allowing natural conversations in real-time while solving the critical background noise challenge using Voice Activity Detection (VAD).

In the last few days, I added a full-stack web app to interact with Gemini (see video below) along with a standalone script for those eager to quickly dive into building real-time voice agents.

Google’s real-time Gemini model is a game-changer, enabling you to independently create production-ready voice agents for industries like customer service, education, and healthcare in a matter of days.

Happy holidays. Go build! https://github.com/saharmor/gemini-multimodal-playground

3. Meta Apollo

Video understanding has been lagging behind text, image, and audio modalities—until now.

Meta and Stanford researchers unveiled Apollo, a new family of state-of-the-art video-centric large multimodal models (video-LMMs) designed to close this gap. Unlike prior efforts, Apollo sets a new standard by efficiently analyzing hour-long videos and achieving breakthrough results on multiple benchmarks.

Paper highlights:

1. Scaling Consistency - design decisions made with smaller models transfer reliably to larger ones, drastically cutting computational costs

2. Advanced video sampling techniques - Apollo uses FPS sampling, outperforming traditional uniform sampling methods

3. Streamlined evaluation - the new ApolloBench benchmark evaluating video-LMMs efficiently, reducing evaluation time by 41x while maintaining accuracy

Apollo’s superior video comprehension capabilities pave the way for breakthroughs like real-time video summarization for content creators, better temporal reasoning for medical diagnostics, and enhanced video analytics for autonomous driving.

With Apollo, video understanding might finally catch up to its multimodal counterparts.

Project page https://apollo-lmms.github.io

4. Claude PDF API

Anthropic has introduced a powerful new PDF-processing feature in its Claude API, surpassing basic text extraction, and it has largely flown under the radar.

Historically, many LLMs stumble when documents include complex elements like images, charts, and LaTeX formulas. But Anthropic’s latest upgrade manages to parse both textual and visual content within a PDF—no extra coding wizardry needed.

Key capabilities include:

1. Automatically parsing PDF text, images, and tables for further analysis, from answering questions about the attached PDF to turning unstructured data into formatted JSONs

2. Providing insight on charts and diagrams by evaluating visual context, not just textual tags

3. Extracting and interpreting LaTeX for scientific or technical documentation

It works by splitting each PDF into two components: the text is extracted as normal, and the entire page is converted into an image. Claude then merges text and visual context for a more holistic understanding. It’s essentially combining LLM intelligence with basic computer vision techniques.

The API supports up to 32MB or 100 pages of PDF content and pricing is similar to the LLM pricing so there’s no premium cost for PDF analysis.

This API could dramatically streamline how we handle financial reports, legal docs, or any PDF requiring detailed interpretation.

Ready-to-run notebook analyzing Anthropic's constitutional AI paper here https://github.com/anthropics/anthropic-cookbook/blob/main/misc/pdf_upload_summarization.ipynb

5. LLaMa-Factory

When is it better to fine-tune a language model than using prompt engineering or RAG? Here’s a clear framework you can apply along with an open-source library I use for fine-tuning.

Good reasons to fine-tune:

1. Emphasizing knowledge that already exists in the model - for instance, in a text-to-SQL task, fine-tuning can be used to emphasize specific SQL dialects or to avoid error-prone edge cases, utilizing the comprehensive understanding of SQL syntax, dialects, and database functionality that the model already possesses.

2. Customizing the structure or tone of responses - fine-tuning can modify the structure or tone of a model's output, such as making the model output valid JSON, which is beneficial for programmatic interactions where handling invalid JSON could lead to many downstream error cases. This includes fine-tuning a model to your company’s writing style.

3. Teaching a model very complex instructions - fine-tuning allows for showing the model many more examples than can be included in a model's context window, which is helpful for complex instructions. This leads to cheaper and faster inference.

Wrong reasons to fine-tune:

1. Adding new knowledge to the base model - the knowledge in a large language model is established during the pre-training runs. New knowledge can't effectively be introduced during the limited scope of fine-tuning. RAG is better suited in such cases.

2. Quickly iterating on a new use-case - fine-tuning involves a slower feedback loop and requires substantial investment in creating the dataset and other aspects of the fine-tuning process. Therefore, it's not suitable for rapid iteration of new use cases.

My preferred tool for fine-tuning open language models is LLaMA-Factory. It features 100+ different large language models, including Meta’s Llama-2, Google’s Gemma, and Mistral’s Mixtral. It also supports advanced algorithms like LoRA, QLoRA, and GaLore for optimized performance.

GitHub repo https://github.com/hiyouga/LLaMA-Factory

Last month’s LinkedIn Highlights

A NotebookLM-powered podcast episode discussing this post:

Over the past few weeks, I've explored how AI agents will fundamentally reshape the internet. From introducing the concept of "agent-responsive design" - where websites optimize for AI interaction - to examining the infrastructure needed for agent-to-agent communication protocols, my previous posts laid the groundwork for understanding the agent-centric future of the web.

Now, it's time to tackle perhaps the most natural and transformative interface for these agents: voice.

While text-based agents have dominated the early wave of AI applications, voice represents the next frontier in human-AI interaction. It's not just another interface - it's the most intuitive and accessible way for humans to interact with AI systems. This intersection of voice technology and AI agents creates unprecedented opportunities for developers, so I'm excited to share this comprehensive guide to the voice AI ecosystem.

After decades of frustrating experiences with scoped voice assistants that cannot be interrupted and follow a narrow rule-based script, we're witnessing a fundamental shift in what's possible. Three key developments drive this shift:

1. Breakthrough in speech-native models - the release of OpenAI's Realtime API last October and Google's Gemini 2.0 Realtime Multimodal API last week mark a transition from traditional "cascading architectures" (where speech is converted to text, processed, and converted back) to speech-native models that can process audio directly with unprecedented quality. With OpenAI's recent 60% Realtime API price reduction and the hiring of WebRTC's founder, we're seeing a clear industry push toward making real-time voice interactions accessible and affordable.

2. Dramatic reduction in complexity - what previously required hundreds of data scientists can now be achieved by small teams of AI engineers. We're seeing companies reach substantial ARR with lean teams by building specialized voice agents for specific verticals - from restaurant order-taking to lead qualification for sales teams.

3. Infrastructure maturity - the emergence of robust developer platforms and middleware solutions has dramatically simplified voice agent development. These tools handle complex challenges like latency optimization, error handling, and conversation management, allowing developers to focus on building unique user experiences.

This convergence creates a unique opportunity for builders. For the first time in human history, we have a god-like AI systems that converse like humans. The era of capable voice AI has arrived, opening up vast opportunities for innovators and developers alike.

Unlike web or mobile app development, where patterns are well-established, voice AI is still in its formative stage. The winners in this space will be those who can combine technical capability with a deep understanding of specific industry needs.

In this post, I'll provide a well-curated overview of the open-source and commercial tools available for developers building voice agents. While VCs segment the market based on investment opportunities, I'll map the ecosystem based on what matters to developers: APIs, SDKs, and tools you can actually use today. What is the go-to model for speech-to-text? The API for synthesis speech? Which tools do other builders rely on to develop voice agents? With the holiday season upon us, there's no better time to build your voice agent, turn it into a company, or automate a personal workflow.

Categories covered in this piece, along with my recommended stack and tips for choosing the right architecture at the end:

Foundations
→ Speech-to-text
→ Text-to-speech

Voice AI Middleware
→ Speech-to-speech
→ Conversational Agents Frameworks
→ Evaluation

End-to-end Automation
→ Full Stack

Missing package or tool you think should have been part of this list? Comment on this post and I’ll take a look.

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Foundations

Speech-to-text (STT)

Automated transcription models have improved significantly over the past couple of years, featuring low latency and cost along with a decreasing WER (Word Error Rate). When selecting a speech-to-text model, consider these key factors:

• Latency requirements - for real-time conversations, you need models that can process speech with < 300ms latency. Cloud APIs like Assembly AI and Deepgram excel here, while Whisper variants are better suited for async transcription.

• Deployment constraints - open source models like Whisper.cpp offer flexibility for edge deployment and privacy-sensitive applications but require more engineering effort to optimize. Cloud APIs provide easier scaling but with higher operational costs.

• Language and accent support - while most models handle standard English well, performance varies significantly for other languages and accents. Whisper has broad language support, while specialized APIs like Assembly AI may offer better accuracy for specific use cases.

Open-source (mostly Whisper-based)

1. Whisper.cpp

2. WhisperX

3. RealtimeSTT

Commercial

1. Deepgram1

2. Assembly AI

3. OpenAI’s hosted Whisper

4. Cloud Providers: Azure, Google, AWS. There don’t seem to be significant differences between the cloud providers, so I usually opt for the one I already use for other services like storage and compute.

👆 RealtimeSTT featured low-latency transcription with wake word activation

Text-to-speech (TTS)

Text-to-speech, once dominated by ElevenLabs’ costly API as of June 2023, has evolved into a competitive market with numerous low-latency, cost-effective alternatives. Modern TTS systems have evolved far beyond simple voice synthesis, offering capabilities like:

• Voice cloning - creating custom voices from just a few minutes of audio samples, enabling personalized agent voices

• Emotional synthesis - adjusting tone, pace, and emphasis to convey different emotions and speaking styles

• Multi-speaker synthesis - seamlessly switching between different voices in a single conversation

The most significant recent advancement is the emergence of "speech-native" models that can generate highly natural speech without the traditional text-intermediate step. This enables more fluid conversations with lower latency and better preservation of emotional nuances.

Open-source

1. Coqui TTS

2. ChatTTS

3. OpenVoice

Commercial

1. Cartesia

2. OpenAI TTS

3. Deepgram Aura

4. ElevenLabs (just released Flash—a 75ms latency model in 32 languages)

Other popular APIs: Play, Amazon Poly, Azure AI Speech, Google

👆 Hello Patient’s voice agent is powered by Cartesia

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Voice AI Middleware

Speech-to-speech

OpenAI sparked everyone’s imagination when they demoed GPT-4o last April, showcasing real-time conversational AI that can be interrupted. Until a few months ago, this kind of technology wasn’t available to developers. Now, we have commercial APIs at a reasonable price (OpenAI dropped the price for its Realtime API by 60% last week) and open-source alternatives.

Speech-to-speech models are specialized AI systems that process and respond to voice input directly in the audio domain without converting to and from text. They are superior to cascading workflows that involve sequential speech-to-text, text-to-text, and text-to-speech models for several key reasons:

1. Ultra-low latency - by processing audio natively, these models achieve response times of ~300 milliseconds, matching natural human conversation speeds. Traditional cascading approaches often exceed 1000ms, creating noticeable delays.

2. Better contextual understanding - speech-to-speech models preserve important non-textual elements like tone, emotion, and conversation dynamics that are often lost in text conversion.

3. Natural interruptions - unlike cascading systems that require rigid turn-taking, speech-native models can listen and process input even while speaking, enabling natural interruptions and more fluid conversations.

4. Improved conversation quality - by maintaining the audio context throughout the interaction, these models better handle overlapping speech, background noise, and the natural rhythm of human dialogue.

Open-source

• Ultravox

• Moshi

Commercial

• OpenAI’s Realtime API

• Gemini 2.0 Multimodal Live API

👆 Conversing with Google’s new Gemini 2.0 (source: Gemini Playground)

Conversational Agents Frameworks

Conversational agent frameworks provide the essential scaffolding needed to build production-ready voice AI applications. These frameworks abstract away much of the complexity in managing real-time voice infrastructure, handling edge cases, and orchestrating the various components of a voice application. Pipecat offers an open-source approach for developers who want complete control over their stack and need flexibility to build custom multimodal experiences. Vocode provides a more opinionated toolkit that simplifies the process of creating voice-based LLM agents, handling much of the complexity around conversation management and error handling. Vapi takes this further by offering a full-featured platform that abstracts away infrastructure challenges while providing enterprise-grade reliability and scalability. The choice between these frameworks often comes down to the specific requirements of your project - whether you need the flexibility of open-source, the simplicity of a focused toolkit, or the robustness of a complete platform.

Open-source

• Pipecat

• Vocode

Commercial

• Vapi

Evaluation

One of the most significant challenges in voice AI development is accurately measuring agent performance. Unlike text-based interactions, where metrics like response accuracy and latency are relatively straightforward, voice agents require a more nuanced evaluation approach that considers elements like conversation flow, emotional appropriateness, and real-world task completion.

Technical performance metrics form the foundation of evaluation, starting with response latency - which should target under 300ms for natural conversation and be measured across different percentiles, while speech recognition accuracy is tracked through Word Error Rate (WER) with special attention to domain-specific terminology. The second dimension focuses on conversational quality metrics that measure the agent's ability to handle natural dialogue. This includes turn management metrics like interruption handling and end-of-speech detection, along with task completion metrics such as success rate and the number of turns needed to complete a task. User experience metrics round out the evaluation framework, tracking call abandonment rates, average call duration, user sentiment scores, and the frequency of repeat requests.

When done right, proper evaluation becomes a game-changing unlock - it enables developers to identify subtle issues that could frustrate users (like awkward pauses or mismatched emotional responses), optimize for natural conversation patterns, and ensure consistent performance across different accents and background noise conditions.

Open-source

• Voice Lab

Commercial

• Coval

End-to-end Automation

Full Stack

The Stripe for voice agents. From prompt to a call.

While building with individual components offers maximum flexibility, full-stack solutions make sense when you need to quickly deploy production-ready voice agents without managing infrastructure complexity. These platforms are particularly valuable for teams that need to focus on their core business logic and customer experience rather than wrestling with the intricacies of voice infrastructure, latency optimization, and scaling challenges.

These solutions typically provide end-to-end capabilities, including pre-built integrations with popular business systems (CRMs, scheduling tools, payment processors), sophisticated error handling, automated failover, and comprehensive analytics. They handle complex engineering challenges like maintaining low latency during peak loads, graceful degradation during API failures, and automated quality monitoring. While you trade some flexibility compared to building your own stack, these platforms can dramatically accelerate time-to-market and reduce engineering overhead.

• Bland

• Deepgram Voice Agents API

• Eleven Labs Conversational AI

👆 A Deepgram-powered voice agent handles a food order

Choosing the right architecture for your voice agent

When selecting a voice AI architecture, developers face a fundamental choice between using full-stack platforms and assembling custom solutions from individual components. This decision ultimately comes down to three key factors: latency requirements, cost constraints, and the need for fine-grained control over the conversation flow.

Read more

A NotebookLM-powered podcast episode discussing this post:

December 2028. Sarah's AI agent encounters an unusual situation while booking her family's holiday trip to Japan. The multi-leg journey requires coordinating with three different airlines, two hotels, and a local tour operator. As the agent begins negotiations, it presents its "agent passport"—a cryptographic attestation of its delegation rights and transaction history. The vendors' systems instantly verify the agent's authorization scope, spending limits, and exposable metadata like age and passport number. Within seconds, the agent has established secure payment channels and begun orchestrating the complex booking sequence. When one airline's system flags the rapid sequence of international bookings as suspicious, the agent smoothly provides additional verification, demonstrating its legitimate delegation chain back to Sarah. What would have triggered fraud alerts and CAPTCHA challenges in 2024 now flows seamlessly in an infrastructure built for autonomous AI agents.

—> The future, four years from now.

In my previous essay, we explored how websites and applications must evolve to accommodate AI agents. Now we turn to the deeper infrastructural shifts that make such agent interactions possible. The systems we've relied on for decades: CAPTCHAs, credit card verification, review platforms, and authentication protocols, were all built with human actors in mind. As AI agents transition from experimental curiosities to fully operational assistants, the mechanisms underpinning the digital world for decades are beginning to crack under the pressure of automation.

The transition to an agent-first internet won't just streamline existing processes—it will unlock entirely new possibilities that were impractical in a human-centric web. Tasks that humans find too tedious or time-consuming become effortless through automation. Instead of clicking 'Accept All' on cookie banners, agents can granularly optimize privacy preferences across thousands of sites. Rather than abandoning a cart due to complex shipping calculations, agents can simultaneously compare multiple courier services and customs implications. Even seemingly simple tasks like comparing prices across multiple vendors, which humans typically limit to 2-3 sites, can be executed across hundreds of retailers in seconds. Perhaps most importantly, agents can maintain persistent relationships with services, continuously monitoring for price drops, policy changes, or relevant updates that humans would miss. This shift from manual, limited interactions to automated, comprehensive engagement represents not just a change in speed, but a fundamental expansion of what's possible online.

Amid these sweeping changes, a new gold rush is emerging. Just as the shift to mobile created opportunities for companies like Uber and Instagram to reinvent existing services, the transition to agent-first infrastructure opens unprecedented possibilities for founders. From building next-generation authentication systems and trust protocols to creating agent-mediated data marketplaces, entrepreneurs have a chance to establish the foundational layers of this new paradigm. In many ways, we're returning to the internet's early days, where core infrastructure is being reimagined from the ground up—this time for an autonomous, agent-driven future.

In this second post of the AI Agents series, we’ll focus on the foundational infrastructure changes that underlie the agent-first internet: new authentication mechanisms, trust systems, novel security challenges, and agent-to-agent protocols, setting the stage for the more commerce-oriented transformations we’ll explore in the following post.

Proving you're a human an agent

Remember when "proving you're not a robot" meant deciphering distorted text or selecting crosswalk images? Those mechanisms become obsolete in a world where legitimate automated actors are the norm rather than the exception. Today’s CAPTCHAs, designed to block bots, have become increasingly complex due to advances in multimodal AI. Paradoxically, these mechanisms now hinder real humans while sophisticated bots often bypass them. As AI outpaces human problem-solving in these domains, CAPTCHAs risk becoming obsolete, reducing website conversions and frustrating legitimate users.

The challenge shifts from proving humanity to verifying the agent has been legitimately delegated and authorized by a human user.

Today’s rate-limiting mechanisms assume human-paced interactions, relying heavily on IP-based throttling to manage access. But in a world of AI agents, what constitutes "fair use" of digital services? In an agent-driven internet, automated browsing will become not just accepted but essential. Cloudflare, Akamai, and similar services will need to pivot from simplistic IP-based throttling to sophisticated agent-aware frameworks.

As businesses grapple with these challenges, a new solution is emerging—one that shifts the paradigm from blocking automated traffic to authenticating and managing it intelligently. Enter the Agent Passport.

Imagine a digital credential that encapsulates an agent's identity and permissions—cryptographically secured and universally recognized. Unlike simple API keys or OAuth tokens, these passports maintain a verifiable chain of trust from the agent back to its human principal. They carry rich metadata about permissions scope, spending limits, and authorized behaviors, allowing services to make nuanced decisions about agent access and capabilities.

By integrating Agent Passports, business websites like airlines can distinguish between legitimate, authorized agents and malicious actors. New metrics, such as agent reliability scores and behavioral analysis, could ensure fair access while mitigating abuse, balancing security with the need for allowing agent-driven traffic.

Authentication mechanisms, such as signing up and signing in, must also evolve for an agent-first internet. Websites will need to determine not just an agent's identity but also its authorized scope—what data the agent is authorized to access (‘read’) and what actions it is permitted to execute (‘write’).

Google Login revolutionized online authentication by centralizing access with a single credential, reducing friction and enhancing security. Similarly, agent passports could create a universal standard for agent authentication, simplifying multi-platform access while maintaining robust authorization controls.

Companies like Auth0 and Okta could adapt by offering agent-specific identity frameworks, enabling seamless integration of these passports into their authentication platforms. Meanwhile, consumer companies like Google and Apple could extend their authentication and wallet services to seamlessly support agent-mediated interactions, bridging the gap between human and agent use cases.

A new protocol for Agent-to-Agent communication

In the early days of the web, protocols like HTTP emerged to standardize how browsers and servers communicated. In much the same way, the rise of agent-mediated interactions demands a new foundational layer: an Agent-to-Agent Communication Protocol (AACP). This protocol would formalize how consumer agents and business agents discover each other’s capabilities, authenticate identities, negotiate trust parameters, and exchange actionable data—all while ensuring both parties operate within well-defined boundaries.

Just as Sarah's travel agent from the intro paragraph seamlessly coordinated with multiple airlines and hotels, AACP enables complex multi-party interactions that would be tedious or impossible for humans to manage manually.

Much like HTTPS introduced encryption and certificates to authenticate servers and protect user data, AACP would implement cryptographic attestation for agents. Trusted third-party authorities, similar to today’s certificate authorities, would issue digital “agent certificates” confirming an agent’s legitimacy, delegation chain, and operational scope. This ensures that when a consumer’s travel-planning agent communicates with an airline’s booking agent, both sides can instantly verify authenticity and adherence to agreed-upon standards.

Without such a protocol, a rogue agent might impersonate a trusted retailer to trick consumer agents into unauthorized transactions, or a malicious consumer agent could spoof credentials to overwhelm a merchant’s infrastructure. By mandating cryptographic proof, robust authentication handshakes, and behavior logs, AACP mitigates these threats before meaningful data or funds change hands.

The handshake phase in AACP would include mutual disclosure of the agents’ technical stacks—such as which LLM or language configuration they use—and their supported capabilities. Once established, the protocol would also govern “write-like operations” (e.g., initiating a payment or updating account details) by enforcing strict sign-offs with auditable cryptographic signatures. Every action would leave a verifiable trail of authorization that can be reviewed and validated after the fact.

Finally, AACP would incorporate locale and language negotiation at the protocol level. Although agents can translate and interpret content dynamically, specifying a preferred language or locale upfront helps streamline interactions. This new protocol weaves together trust, authentication, and contextual awareness, forging a resilient substrate on which the agent-first internet can reliably function.

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Trust and reputation reimagined

When we navigate the internet, our judgment of a website's credibility hinges on a blend of visual and social cues. We look for secure HTTPS connections, professional design, and familiar branding to assure us that a site is trustworthy. No one wants to input their credit card information on a site that looks like it was built in the early 2000s. User reviews and star ratings on platforms like Trustpilot and G2 further influence our decisions, offering insights drawn from shared human experiences.

Perhaps no aspect of online commerce requires more fundamental reimagining than trust and reputation systems. In an agent-mediated economy, traditional cues for reliability fall short. AI agents can't interpret visual aesthetics or branding elements–they operate on data, protocols, and cryptographic proofs. Trust mechanisms must pivot from human perception to machine-readable verifications. For instance, an agent might verify a seller's identity through cryptographic attestations and assess service quality via automated compliance records, ensuring decisions are based on objective, tamper-proof data. Traditional review platforms like Trustpilot and G2, built around subjective human experiences and star ratings, will also become increasingly obsolete.

The emerging alternative is a new trust infrastructure built on quantifiable, machine-readable metrics. Instead of relying on potentially AI-generated reviews, a problem that has already undermined traditional review systems, agents could assess services using benchmarks like delivery time reliability, system uptime, or refund processing speed—measurable metrics that ensure objective evaluations rather than subjective human reviews. This could involve decentralized reputation networks where trust is established through cryptographically verified interaction histories and smart contract execution records. Such systems would offer objective assessments of service quality, enabling agents to make informed decisions without relying on potentially biased or manipulated human reviews.

Moreover, the feedback loop between consumers and businesses will evolve dramatically. Instead of sending generic emails requesting reviews—a method often resulting in low response rates—commerce websites can engage directly with your AI agent to collect timely feedback about specific topics like shipping or product quality. They might offer incentives like future store credit to encourage participation. The human user could provide a brief impression, such as "The cordless vacuum cleaner works well, but the battery life is short." The agent then takes this input, contextualizes it with additional product data, and generates a comprehensive review that highlights key features and areas for improvement. This process not only saves time for the user but also provides businesses with richer, more actionable insights.

Trustpilot and G2 could pivot by introducing agent-oriented verification systems, such as machine-readable trust scores derived from operational metrics like service accuracy, delivery consistency, and customer support responsiveness, enabling agents to evaluate businesses programmatically.

The new data-sharing economy

Information sharing in the age of AI agents demands a fundamental reinvention of the current consent and data access model. Rather than blunt instruments like cookie banners and privacy policies, websites will implement structured data requirement protocols—machine-readable manifests that explicitly declare what information is needed and why.

This granular control would operate at multiple levels of specificity. For example, an agent could share your shirt size (L) with a retailer while withholding your exact measurements. It might grant 24-hour access to your travel dates but permanent access to your seating preferences. When a service requests location data, your agent could share your city for shipping purposes but withhold your exact address until purchase confirmation. These permissions wouldn't be just binary yes/no choices—they could include sophisticated rules like "share my phone number only during business hours" or "allow access to purchase history solely for personalization, not marketing." Such granular controls, impossible to manage manually at scale, become feasible when delegated to AI agents operating under precise constraints.

AI agents would also act as sophisticated information gatekeepers, maintaining encrypted personal data vaults and negotiating data access in real time.

These mechanisms will fundamentally shift the balance of power in data-sharing dynamics. GDPR-like frameworks may evolve to include provisions for dynamic, agent-mediated consent, allowing for more granular data-sharing agreements tailored to specific tasks. Websites might implement real-time negotiation protocols, where agents can evaluate and respond to data requests based on their principal's preferences, preserving privacy while optimizing functionality.

New attack vectors

The shift to agent-mediated interaction introduces novel security challenges. Agent impersonation and jailbreaking agents are two examples.

Jailbreaking AI agents pose significant risks, as manipulated agents could act outside their intended scope, leading to unintended purchases or other errors. Techniques like instruction-tuning poisoning or adversarial suffix manipulation could alter an agent’s behavior during critical tasks. For example, adversarial instructions embedded in websites’ HTML might influence an agent’s purchasing logic, bypassing its human-defined constraints. Robust safeguards and continuous monitoring will be essential to prevent these vulnerabilities.

Agent impersonation adds a complex layer to cybersecurity challenges. Malicious actors could spoof an agent's credentials to access sensitive data or execute fraudulent transactions. Addressing this threat demands robust multi-layered verification protocols, such as cryptographic identity verification paired with continuous behavioral monitoring, to ensure authenticity and safeguard sensitive interactions.

Building the new web - opportunities for founders

The web’s agent-first future has no established playbook—and that’s exactly where founders thrive. Entirely new product categories are waiting to be defined: agent-to-agent compliance dashboards, cryptographic attestation services that replace outdated CAPTCHAs, and dynamic data-sharing frameworks that make “privacy by design” a reality. Platforms that offer standardized “agent passports,” identity brokerages that verify delegation rights, agent-native payment gateways, and trust ecosystems driven by machine-readable performance metrics—each of these represents a greenfield opportunity to set the standards of tomorrow’s internet. Startups anticipating these shifts can position themselves as foundational players in an agent-driven economy, opening new channels of value creation and establishing a competitive edge before the rest of the market catches up.

Some concrete areas include:

• Trustpilot for agents - creating machine-readable trust metrics and reputation systems that help agents evaluate services and vendors

• Okta for AI agents - building the identity and authentication layer that manages agent credentials, permissions, and delegation chains

• OneTrust for agents - creating the new standard for privacy preference management, turning today's basic cookie banners into sophisticated data-sharing frameworks where agents can negotiate and manage granular permissions across thousands of services

• Cloudflare for agent traffic - developing intelligent rate-limiting and traffic management systems designed for agent-scale operations

• LastPass for agent permissions - building secure vaults that manage agent credentials and access rights across services

• AWS CloudFront for agent data - creating CDN-like infrastructure optimized for agent-readable formats and rapid agent-to-agent communication

• McAfee security for agents - developing security platforms that protect against agent impersonation and novel attack vectors

Go build.

This is the second essay in a five-part series exploring the future of AI agents and their impact on the internet. In Part 3, we'll move from infrastructure to commerce, exploring critical questions that emerge in an agent-first economy: How will payment systems evolve to support autonomous purchasing decisions? What happens to marketing and customer engagement when AI agents become the primary decision-makers? And perhaps most intriguingly, how will businesses adapt their revenue models when agents can negotiate prices and terms at superhuman speed and scale? Subscribe and follow as we continue to examine the technical, social, and economic implications of this transformation.

Welcome to a new post in the AI Agents Series - helping AI developers and researchers deploy and make sense of the next step in AI.

My last post explored how the internet will trans

form for an agent-first future - from websites optimizing for AI interaction through "agent-responsive design" to the emergence of Agent Engine Optimization (AEO) as the next SEO. We saw how tech giants like Google, Apple, OpenAI, and Anthropic are racing to define this next evolution of digital interaction, with Gartner projecting that by 2028, 33% of enterprise software applications will include agentic AI.

In this post, I'll outline a curated, though non-exhaustive, overview of the open-source ecosystem for developers creating these AI agents. While numerous market maps exist for AI agents, they often cater more to venture capitalists than builders. Developers need actionable tools and frameworks to launch functional AI agents today.

Which tools do other builders rely on to develop voice agents? What’s the leading open model for document understanding? With new packages emerging almost daily, I’ll focus solely on the libraries I’ve personally found most effective. This list is, therefore, intentionally selective rather than exhaustive.

Every package included here supports commercial use and has a permissive open-source license.

With the holiday season coming, there's no better time to dive into these tools and start building.

Categories covered in this piece:
→ Frameworks for Building and Orchestrating Agents
→ Computer and Browser Use
→ Voice
→ Document Understanding
→ Memory
→ Testing and Evaluation
→ Monitoring and Observability
→ Simulation
→ Vertical Agents

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Frameworks for Building and Orchestrating Agents

Building AI agents requires robust frameworks that can handle complex workflows, memory management, and tool integration. These foundational frameworks serve as the backbone for creating agents that can understand, plan, and execute tasks autonomously.

• CrewAI - a framework for orchestrating role-playing, autonomous AI agents

• Phidata - build AI assistants with memory, knowledge, and tools

• Camel - build customized multi-agent systems to generate data, complete tasks, or simulate real-world interactions

• AutoGPT - create, deploy, and manage continuous AI agents that automate complex workflows

• AutoGen - develop LLM applications using multiple agents that can converse with each other

• SuperAGI - build, manage, and run autonomous AI agents quickly and reliably

• Superagent - an open framework for building AI assistants

• LangChain & LlamaIndex - the usual suspects, facilitating AI Agents through composability

Computer and Browser Use

For AI agents to be truly useful, they need to interact with computers and browsers just like humans do. These tools enable agents to navigate websites, control applications, and execute commands programmatically, bridging the gap between AI reasoning and real-world actions.

• Open Interpreter - turn natural language commands into code that runs on your local machine

• Self-Operating Computer - enables multimodal models to operate a computer

• Agent-S - an open agentic framework that uses computers like a human

• LaVague - create web agents that take actions on websites using LLMs as their reasoning engines

• Playwright - a framework for web testing and automation

• Puppeteer - a JavaScript library that provides a high-level API to control Chrome or Firefox

Voice

Voice interfaces represent the most natural way for humans to interact with AI agents. These tools enable the creation of agents that can understand spoken language, maintain context in conversations, and respond with natural-sounding speech, making AI interaction more accessible and intuitive.

Speech2speech

• Ultravox - a speech2speech model for real-time voice interaction, superior to Moshi for now

• Moshi - a speech2speech model for real-time voice interaction

• Pipecat - a framework for voice and multimodal conversational AI, supporting speech2text, text2speech, video, etc.

Speech2text

• Whisper - OpenAI's speech2text model

• Stable-ts - a lightweight Whisper wrapper with timestamps and more

• Speaker diarization 3.1 - pyannote’s flagship model for speaker detection

Text2speech

The only decent open model I came across was ChatTTS, which is satisfactory for production. I, therefore, default to ElevenLabs or Cartesia.

Misc

• Vocode - a toolkit for building voice-based LLM agents

• Voice Lab - a comprehensive testing and evaluation framework for voice agents across language models, prompts, and agent personas

Document Understanding

Modern AI agents need to process and understand documents in various formats, from PDFs to images with text. These tools provide the crucial ability to extract, comprehend, and act on information from unstructured documents, enabling agents to handle real-world business processes.

• Qwen2-VL - vision language model from Alibaba outperforming GPT-4o and Claude 3.5 Sonnet

• DocOwl2 - an efficient multimodal LLM for OCR-free document understanding

Memory

Without memory, AI agents are limited to single-turn interactions. These memory tools enable agents to maintain context over long conversations, remember user preferences, and learn from past interactions, making them truly personal assistants rather than just query responders.

• Mem0 - provides an efficient, self-improving memory layer for LLMs, enabling personalized AI experiences

• Letta (fka MemGPT) - create LLM agents with long-term memory and custom tools

• LangChain - offers memory components to manage conversation history and context

Become a premium AI Tidbits subscriber and get over $1k in free credits to build AI agents with Vapi, Claude, and other leading AI tools (Hugging Face, Deepgram, etc.), along with exclusive access to the LLM Builders series and in-depth explorations of crucial topics, such as the future of the internet in an era driven by AI agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Testing and Evaluation

As AI agents become more complex, robust testing becomes critical. These tools help developers evaluate agent performance, identify failure modes, and ensure reliability across different scenarios and environments.

• Voice Lab - a comprehensive testing and evaluation framework for voice agents

• AgentOps - tools for monitoring and benchmarking agent performance

• AgentBench - a benchmark to evaluate LLMs as agents across various environments (Web, Minecraft, Visual Design, etc.)

Monitoring and Observability

Understanding how AI agents perform in production is crucial for maintaining reliability and optimizing costs. These tools provide insights into agent behavior, resource usage, and performance metrics essential for running agents at scale.

• openllmetry - an open-source, OpenTelemetry-based end-to-end observability tool for LLM applications

• AgentOps - agent monitoring, LLM cost tracking, benchmarking, and more

Simulation

Before deploying agents to real-world scenarios, testing them in controlled environments is crucial. These simulation tools allow developers to validate agent behavior, test edge cases, and refine decision-making capabilities in safe, reproducible environments.

• AgentVerse - facilitates the deployment of multiple LLM-based agents in various applications, including simulations

• Tau-Bench - a benchmark and testing code for agent-user interactions in real-world domains like retail and airline

• ChatArena - multi-agent language game environments for research on autonomous LLM agents

• AI Town - A virtual town where AI characters live, chat, and socialize

• Generative Agents - Stanford’s Interactive simulacra of human behavior

Vertical Agents

There are dozens of open vertical agents out there, so here are just a few select ones I’ve tinkered with and found the most useful:

• OpenHands (Coding) - a platform for software development agents powered by AI

• aider (Coding) - pair programming in your terminal

• GPT Engineer (Low code) - build applications using natural language. Specify what you want to build, and the AI will ask for clarification before building it.

• screenshot-to-code - convert screenshots into a functioning website using HTML/Tailwind/React/Vue

• GPT Researcher (Research) - an autonomous agent that performs comprehensive research on any given topic

• Vanna (SQL) - chat with your SQL database

Looking Ahead

While this post focused on open-source packages with permissive licenses, I plan to publish another comprehensive list specifically for engineers building voice agents. This upcoming guide will include both open-source and commercial tools, covering solutions like OpenAI's Realtime API (speech2speech) and ElevenLabs (text2speech), along with detailed comparisons of their capabilities, pricing models, and ideal use cases.

Stay tuned for more deep dives in the AI Agents Series.

Comprehensive list of open-source packages for AI engineers (last update: Aug ‘23)