This post captures my takeaways from attending Google’s flagship event, I/O 2025. It’s not a comprehensive announcement round-up. Instead, I’ve focused on the launches that matter most to anyone building or working with AI. I also share my perspective on what these moves mean for the broader AI ecosystem and founders, developers, and researchers alike.

A NotebookLM-powered podcast episode discussing this post:

Since 2017, when Google unveiled the groundbreaking Duplex demo at its biggest event of the year, Google I/O, I've been captivated by the company's AI advancements. For me, it was the first truly practical, consumer-facing use of AI—a clear example of how AI could take over routine tasks like booking appointments. But more importantly, it marked a key step toward a future where AI helps people express themselves in ways that were previously out of reach.

In recent years, the AI community has often viewed Google as trailing behind leaders like OpenAI and Anthropic. However, this year's Google I/O conference felt different—everything finally clicked. Google moved from research to reality, capitalizing on its massive distribution channels and deep technological prowess. The perfect combination of state-of-the-art technology with access to real-world usage through Search, Google Workspace (Gmail, Sheets, Docs, etc.), and Android (smart TVs, glasses, phones).

And it wasn’t only me. The same sentiment echoed across the press tent at I/O last Tuesday, capturing an energy reminiscent of OpenAI’s inaugural DevDay.

The winning combination, as defined by Google in this week’s I/O, manifests across three principles:

• Powerful - deploying best-in-class models to support real-time, reliable experiences

• Personalize - tailoring AI to understand and cater to individual user preferences and needs

• Proactive - developing AI that anticipates user needs and acts accordingly without being too intrusive or eager

Out of these three, the one I found the most promising is Personalize.

Google's unparalleled access to user data gives it a powerful edge over competitors like OpenAI and Apple. It understands my interests through the searches I make (Search), the places I go (Maps), the music I listen to (YouTube), my payment habits (Google Pay), and even my work life (Gmail, Calendar, Docs). This breadth of insight uniquely positions Google to deliver truly personalized AI experiences.

Google didn’t just launch new products at I/O, it made deliberate moves into markets long held by OpenAI, Meta, Perplexity, Anthropic, and even Shopify and Stripe. Each announcement, from Jules to Gemini Live, stepped directly into competitive territory. If you’re working on dev tools, agent platforms, creative apps, e-commerce flows, or voice interfaces, these updates are worth reading. I’ve included a breakdown of the most directly affected companies and industries at the end of this post—worth reviewing if you want to stay ahead of what’s coming.

The real story isn’t about how many features Google shipped, though. It’s about the strategy taking shape. Google is doubling down on vertical integration and deeply contextual AI. That’s the new game. In Ben Thompson (Stratechery) terms, it’s Aggregation Theory with agency. Google owns the user interface, the distribution (Android, Chrome, Search), and now, increasingly, the intelligence layer.

In this post, I'll outline a selected subset of announcements I found most promising and share my 2¢ on why this event marks a turning point in AI's evolution.

Google AI Studio, Jules, and Stitch

Perhaps one of the most significant announcements at Google I/O was unveiling the upgraded Google AI Studio, with a whole new Build section—an integrated development environment explicitly designed for building AI-driven applications.

Positioned directly against IDEs like Cursor, Windsurf, Lovable, and Bolt, Google AI Studio unifies Google's flagship multimodal Gemini models into one streamlined interface. Developers now have the ability to build and deploy their creations using natural language and with a single click to Google Cloud, reinforcing Google's strategic advantage through infrastructure integration.

Jules, a particularly intriguing release, is Google's take on the autonomous coding agent, similar to the likes of Devin and Factory. Quietly entering public beta at jules.google, Jules represents Google's ambitions to dominate the software development lifecycle: from writing documentation and deploying applications to autonomously submitting pull requests. Though overshadowed by flashier announcements, Jules may well emerge as a sleeper hit among developers seeking highly efficient, AI-augmented development workflows.

Stitch, another groundbreaking tool revealed at I/O, could radically simplify UI design processes. Through natural language prompts, designers can describe interfaces, which Stitch then generates and exports directly into Figma.

Together, Google AI Studio, Jules, and Stitch exemplify Google's strategy of leveraging its state-of-the-art models and infrastructure to deliver highly integrated, practical, and transformative tools for developers and designers alike.

Powerful models

Gemini 2.5 took center stage at I/O, outperforming nearly every major AI benchmark: from coding and web development to complex reasoning and video understanding. Compared to leading commercial models, it stands out with a January 2025 knowledge cutoff, a 1 million-token context window, and operates at around a quarter of the cost of OpenAI’s GPT-4o.

Key improvements include:

• Deep Think - an advanced reasoning capability, achieving state-of-the-art results in complex mathematical and programming tasks in exchange to increased cost and latency.

• Enhanced function calling and Structured Outputs - until now, the real-time Gemini models haven’t been usable for anyone needing function calling or structured output. Now, it’s finally fixed.

• Gemini Diffusion - Google unveiled Gemini Diffusion, generating text 5x faster than the leading Flash Lite model. This advancement is powered by recent research utilizing diffusion models for text generation, marking a significant leap forward in efficiency and responsiveness.

Search & AI Mode

Google has been experimenting with a new way of search over the last few weeks, dubbed “AI Mode”. This new mode just got generally available in the US last Tuesday. Powered by the Gemini 2.5 Pro model, AI Mode allows users to engage in multi-turn dialogues, enabling more complex and nuanced information retrieval.

Highlights from the new search experience:

• Personal Context - for an even more customized experience, AI Mode will offer personalized suggestions based on your connected Google apps, starting with Gmail, to bring in more of your personal context. For example, if you’re searching for “things to do in Nashville this weekend with friends, we're big foodies who like music” ahead of an upcoming trip, AI Mode can show you restaurants with outdoor seating based on your past restaurant bookings and searches.

• Agentic Checkout - streamlining the purchasing process by allowing users to complete transactions directly within Search, bypassing the need to navigate to third-party websites. For example, when searching for concert tickets, AI Mode will find the best options and facilitate the purchase through Google Pay, all within the same interface. This seamless integration has the potential to disrupt traditional e-commerce models and reshape how users interact with online marketplaces. I wrote a whole series on the new agentic internet!

• Try It On - enhancing the virtual shopping experience, Google's "Try It On" feature utilizes Google’s strong image generation diffusion models to allow users to visualize clothing items on themselves. Users can upload their picture using Google Photos and see how different garments would look on their own bodies. Google’s generative AI capabilities meet distribution (Google Photos).

• Deep Search - by synthesizing information from multiple sources, AI Mode can provide comprehensive answers to multifaceted questions, making Google Search relevant again in the face of competing tools such as OpenAI’s and Perplexity’s Deep Research.

The revenue gamble

While Google's AI Mode represents a significant leap forward in search capabilities, it also reveals a fundamental tension at the heart of the company's strategy. Google is essentially betting against its own golden goose: the advertising-driven search model that has generated over 50% of its revenues for over two decades.

The math is straightforward but concerning: if AI Mode provides comprehensive answers directly within search results, users will click through to fewer websites. Independent studies already suggest this trend with AI Overviews, and AI Mode's conversational interface offers even fewer opportunities for traditional paid link placements. Google's executives at I/O spoke confidently about the technical capabilities of their new search experience, but when it came to discussing how this translates into sustainable revenue streams, the answers were notably vague.

This isn't just a minor product pivot, it's a fundamental reimagining of how Google makes money. The company appears to be racing toward a future where AI assistants and conversational interfaces replace link-based search, and while there are certainly ways to imagine business models around personalized AI assistants and agentic workflows, Google hasn't articulated what those might look like or how they'll replace the massive cash flows from traditional search advertising.

Project Mariner

Project Mariner is Google's step toward giving AI true agency across your devices. It’s their answer to OpenAI's Operator and Anthropic's Computer Use. An infrastructure-level system for teaching AI to interact with your digital environment just like a human would.

At its core, Mariner is about "teach and repeat". Show Gemini how to perform a task: filling out a form, generating a weekly status report, uploading data to a dashboard, and it can replicate that workflow again and again.

Mariner will be released as part of the Gemini API later this summer, which means developers can build agents that don’t just reason and plan, but act: navigating apps, automating browser actions, and manipulating on-screen interfaces.

Whether it’s booking a flight, copying events into a spreadsheet, or handling repetitive workflows across company tools, Mariner helps AI move beyond suggestions and into action.

Gemini app and Gemini Live

With the new Gemini app and its Live feature, Google is officially entering the race for the "everything AI assistant”, a direct challenger to ChatGPT, Meta AI, and Apple Intelligence.

The Gemini app is no longer just a chatbot. It’s a real-time, context-aware assistant that lives across your devices and ties directly into Google’s ecosystem: Gmail, Calendar, Keep, Docs, Maps, and even YouTube. Thanks to its tight OS-level integration (powered by Project Mariner), Gemini can also take actions on your phone.

But what really sets Gemini apart isn’t just input, it’s output:

• Search Live and Project Astra - building on the capabilities of AI Mode, Google introduced Search Live, a feature that combines real-time camera input with search functionality. Users can point their device's camera at an object or scene and receive immediate information (similar to OpenAI’s Advanced Voice Mode), effectively turning their environment into an interactive search field. This feature is powered by Project Astra, Google's multimodal AI assistant that integrates visual and auditory data to provide contextually relevant responses.

• Canvas is Google’s answer to tools like OpenAI’s Canvas and Anthropic’s Artifacts. Ask Gemini to summarize an article and it will build an interactive webpage, infographic, quiz, or even a lightweight app.

• Deep Research now supports uploaded personal files, synthesizing them into study guides, plans, or insights, connecting directly to your Drive and Gmail, offering context-rich reasoning grounded in your data.

• Agent Mode enables task automation across Gmail, Calendar, and partner services like Zillow. Unlike a basic plugin system, this builds on Mariner’s deeper Android-level control and Google's new MCP support, enabling multi-step reasoning and actions.

• Quiz and Video Generation taps into Veo (text2video) and Lyria (music generation model), turning documents into test prep material and short videos.

Generative models for creatives

Google’s generative media stack is finally starting to feel competitive.

• Veo 3 is their new text-to-video model - high-quality, photorealistic footage, now with native audio generation. Think Pika or Runway, but with better motion, longer clips, and built-in sound.

• Image 4 improved with sharper details, better text rendering, and is now integrated into Gemini.

• Lyria 2 is Google’s music generation model. Based on the demo, Lyria is still in its infancy and far from the quality of Suno and Udio.

• Flow is a new AI-powered video editor. Type a prompt, get an 8-second clip. Stitch clips together, tweak scenes with natural language. It’s Google’s answer to creative environments like Adobe Premiere, but for AI-native workflows.

Taken together, this is Google’s most serious push yet into generative video, music, and imagery, accessible via Google AI Studio and the Gemini API.

Google AI Glasses

Twelve years after the original Google Glass flop, Google’s trying again, and this time, it looks promising.

Google unveiled a new pair of smart glasses powered by Android XR and deeply integrated with the Gemini model family. They come equipped with microphones, speakers, a camera, and an in-lens display, offering a level of interactivity that goes beyond Meta’s Ray-Ban, which don’t have a display. Google is going a step further: your real world now comes with real-time captions, directions, translations, and a personal assistant whispering relevant information.

And that’s the key difference: Google has the phone and app distribution. Meta and OpenAI with its ChatGPT consumer app do not. That means Google can natively integrate with Gmail, Calendar, Maps, Docs, Translate, and YouTube—capabilities that come pre-installed on Android and are used by billions. Need to translate a live conversation? Snap a photo and auto-organize it? Navigate to a meeting while rescheduling the next one? All of that is now on your face.

To get there, Google partners with Gentle Monster and Warby Parker for manufacturing, echoing the Meta + Ray-Ban strategy.

If you’re thinking this sounds like something Ben Thompson would write a thousand-word piece about, you're not wrong. This is exactly the kind of vertical integration that makes Apple and others sweat: powerful native models, fused with real-time inputs (voice, vision), and paired with a ubiquitous OS.

The world was not ready for wearable AI in 2013. But in 2025, with AI-native operating systems and mainstream model adoption, and after Meta has proven market traction, Google may have found the perfect moment for a comeback.

Industry impact

So what does all this mean if you're not Google? Below is a breakdown of the major announcements from I/O and the companies most likely to feel the heat.

Read more

This post is about the unprecedented shift happening in voice AI interfaces and what it means for human interaction. As these new capabilities become accessible through simple APIs, a massive opportunity is emerging for founders to build products that reimagine how we communicate with technology and each other.

A NotebookLM-powered podcast episode discussing this post:

June 2025. Sarah paces in her living room, rehearsing an important client presentation. Her AI companion listens intently, chiming in when relevant to offer real-time feedback on her delivery and content. "I think you rushed through the ROI section," it suggests in a warm, natural voice. "Let's try that part again, but this time—" Sarah cuts in mid-sentence, "Actually, can we focus on the opening first? And don't be so nitpicky!" The AI smoothly adjusts, without awkward pauses or robotic transitions. What was once a frustrating experience of rigid, unnatural interactions with voice assistants has evolved into fluid, human-like conversation.

I've spent considerable time lately thinking about and building in the voice AI space, and something unprecedented is emerging: for the first time in history, we have real-time, affordable, and competent artificial voice that's just one API call away. In just a few months, we've seen significant leaps forward from the likes of OpenAI’s Advanced Voice Mode (AVM) and new speech models, Google’s real-time conversational Gemini Flash, and Sesame’s emotionally intelligent AI1.

This isn't just a technical milestone—it's a fundamental shift in how we interact with technology and, potentially, with each other. It will create numerous new opportunities for builders while redefining the very nature of human communication.

Gavin Purcell is arguing with Sesame’s realtime voice AI 👆

The dawn of natural voice AI

Remember the last time you called your bank's automated system? The familiar dance of repeated phrases, misunderstood words, and the desperate pressing of "0" to reach a human operator. That era is ending. OpenAI's release of Advanced Voice Mode (AVM) last September marked a pivotal moment when conversing with AI began to feel genuinely human.

This transformation stems from two key breakthroughs. First, the shift from cascading architectures (speech-to-text → text processing → text-to-speech) to direct speech-to-speech models eliminates intermediate processing stages that previously slowed conversational AI interactions. Second, the dramatic reduction in latency and cost. When OpenAI initially released its Realtime API, the price made it impractical for widespread adoption (18$/hour). But just four months later, Google's release of Gemini Flash 2.0 and OpenAI's 60% price reduction opened the floodgates for affordable and human-like voice AI applications that are one API call away.

Just last week, OpenAI unveiled its most human-like speech models yet, enabling developers to embed expressive cues like [WHISPERING] or [LAUGHING] directly into the text. Here's a quick demo from OpenAI.fm—a public tool launched alongside this release, showcasing what this new level of expressiveness sounds like in action:

Builders can now launch phone assistants that qualify sales leads, resolve customer support calls, automate insurance sales, or screen patients before their upcoming appointments. The necessary tools are already available and are just a single API call away.

The interruption problem

However, building truly natural voice interactions isn't just about faster processing and better voice synthesis. One of the most fascinating challenges lies in handling interruptions—a fundamental aspect of human conversation that AI still struggles with.

Current voice AI systems, including the ones mentioned like OpenAI’s AVM, face several key challenges:

1. Oversensitivity to background noise (I always mute myself when not speaking)

2. Inability to distinguish between relevant speakers and ambient conversation

3. Lack of visual cues that humans use to anticipate and manage interruptions

Unlike human phone conversations, where near-zero latency and natural turn-taking make interruptions manageable, AI interactions often feel clunky when users try to interject2. Interestingly, humans tend to interrupt AI more frequently and aggressively than they would other humans, creating a new challenge for voice AI developers while creating a new interaction paradigm for human-AI conversation.

The social impact

This voice revolution raises profound questions about human interaction and relationships:

• Could the instant gratification of interruptible AI conversations and the ability to be rude without consequences degrade our patience and interpersonal skills, similar to how ubiquitous access to pornography has distorted societal expectations around intimacy?

• The convenience of always-available AI consultation might reduce our reliance on human relationships. Consider how we once relied on reading maps and asking locals for directions—skills now largely abandoned as we defer to GPS. Could meaningful conversations be next?

• Could we soon have more conversational exchanges with AI agents than with human companions?

Think: Would you rather rehearse a high-stakes presentation in front of a potentially judgmental friend or instantly consult a non-judgmental AI companion available 24/7?

What does this mean for our interpersonal relationships?

Cultural nuances in AI conversation

One size doesn't fit all in human conversation, and the same is true for AI. OpenAI's recent update of GPT-4o to GPT-4.5 was mainly about moving away from its "corporate HR" tone, recognizing that natural conversation varies significantly across cultures and contexts.

Different cultures have distinct interruption patterns, politeness norms, and conversation styles. Today's systems largely fail to account for these cultural differences, creating a significant opportunity for AI builders to develop models that adapt to:

• Cultural background

• Individual user patterns

• Contextual cues

• Historical interactions

OpenAI already possesses such context through its Memory feature, and Google, of course, knows virtually everything about us already.

I imagine the best conversational AI systems of the future will incorporate nuances that we take for granted.

Rethinking communication

The holy grail for conversational AI might be achieving the natural flow of a phone call between humans, where interruptions feel natural and turn-taking is seamless. But perhaps we need to aim higher. As AI systems gain multimodal capabilities (vision, touch, etc.), they could potentially surpass human conversation by reading subtle cues we often miss.

Figure's household robots learn tasks on the fly 👆

What surprises me most is how slowly Advanced Voice Mode is being adopted. Despite its impressive capabilities, many of my friends still default to typing or using Whisper (OpenAI's speech-to-text model) rather than having natural conversations with it. Perhaps this hesitation reflects our collective uncertainty about speaking naturally to machines, or simply a lack of awareness—after all, it only became available to free users last month, and many may not yet know how to use it. Either way, it suggests we're in an awkward adolescent phase of voice AI adoption—the technology is capable, but our habits and expectations haven't quite caught up.

The voice AI revolution isn't just about making machines sound more human—it's about fundamentally changing how we think about conversation, relationships, and human interaction. While we'll certainly see a proliferation of phone AI agents and computer assistants in the short term, there's a more profound transformation taking shape beneath the surface.

As we build these systems, we need to consider not just what's technically possible, but what's socially desirable. For now, it's clear that we're entering an era where the line between human and AI conversation is increasingly blurry—for better or worse.

To end on a lighter note, here’s a fun video of ChatGPT’s Voice Mode reimagining an alternate ending to Titanic.

A NotebookLM-powered podcast episode discussing this post:

After listening to a recent podcast featuring Andrej Karpathy1, I've pondered the concept of economies of scale in AI systems, specifically when it comes to data acquisition. The idea of economies of scale originated in the manufacturing world, where increased production led to lower per-unit costs. In the tech world, this concept was adapted by SaaS startups, where the marginal cost of serving an additional customer approached zero. A prime example is Uber, which leveraged its platform to achieve massive scale: as more drivers joined, more riders were attracted, creating a virtuous cycle that dramatically reduced per-ride costs while improving service quality.

Hungry generative AI models drive major tech companies to pursue high-stakes data partnerships, exemplified by OpenAI's agreements with TIME magazine and Reddit and Meta's strategic alliance with Reuters to secure premium training content.

This post explores how economies of scale in the context of data apply to AI and generative models, focusing on three key areas: software vs. hardware, humanoid robots, and large language models.

Scaling AI software vs. hardware - autonomous vehicles

Tesla and Waymo represent two distinct approaches to achieving autonomous driving capabilities. Tesla, under Elon Musk's leadership, has pursued a vision of making self-driving technology accessible to the mass market through its consumer vehicles. Their strategy revolves around deploying a large fleet of cars equipped with cameras and neural networks that learn from real-world driving data. Waymo, originally Google's self-driving car project, has taken a more cautious approach, focusing on developing a robust autonomous driving system using high-end sensors and detailed mapping technology. While both companies aim to revolutionize transportation, their contrasting strategies highlight fundamental differences in how AI systems can be scaled.

Software-driven AI solutions, such as Tesla's AI systems, can scale more efficiently than hardware-based systems like those deployed by Google’s Waymo.

Waymo started deploying its autonomous fleet using Jaguar cars, featuring expensive hardware such as LiDAR, radar, and high-precision GPS to capture and interpret real-time data. Penetrating a market with a high-end offering is a familiar strategy: Uber started with Uber Black before offering the more affordable option, Uber X.

Tesla is less hardware-dependent and mostly relies on cameras, which are substantially cheaper: a fully deployed Waymo costs $200k compared to Tesla's Model 3 starting price of ~$39k.

Software has a distribution advantage—once built, it can be deployed and iterated across millions of devices, i.e. vehicles, with minimal additional cost. In contrast, hardware solutions are constrained by the need for physical components like sensors, processors, and maintenance, which scale more slowly and are harder to replicate.

In this case, Waymo is limited by its deployed hardware because its models are tightly coupled with their underlying sensor data.

Also, unlike Tesla, Waymo's pace of data acquisition is a function of the number of rides it provides, i.e., utilization rate. This is why partnership deals like the one with Uber make sense.

While many view Waymo’s partnership with Uber as primarily a commercial move to increase revenues, the real strategic value lies in the diverse data it gathers from Uber’s wide geographic spread of drivers and riders, both across the U.S. and, potentially, globally. The Uber partnership could also serve as a stepping stone for Waymo to expand into broader delivery services. Imagine collaborations with major retailers like Walmart or Target, or fast-food giants such as McDonald’s—directly challenging Uber and logistics providers like UPS in the last-mile delivery race, generating even more data to improve its underlying self-driving technology.

But beating Tesla on data quantity isn’t a simple feat. Each day, Tesla drivers drive 137 million miles, generating and sending data that includes human overrides, external cameras' video, location and trip logs, and, in some cases, footage from in-cabin cameras—an endless stream of real-world human-labeled data. Recognizing the value of detailed data, Tesla made it extremely easy for drivers to contribute richer feedback by allowing them to provide voice input immediately after disengaging the autopilot system.

It's not only about the quantity. Tesla benefits from the diversity of data its drivers generate—different driving styles, varied terrains, and ever-changing weather conditions.

More data * More diverse data == faster iteration and deployment of better AI models.

So, even though Waymo is objectively ahead with real-world autonomous rides across California, Arizona, and Texas, Tesla could catch up quickly with a software update across its vehicle fleet. Waymo, on the other hand, would likely require a change in its sensor hardware, leading to scale issues.

Become a premium to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like OpenAI's DevDay and autonomous agents.

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

Humanoid robots and teleoperations - Better Labeled Data

Another key insight involves how humanoid robots tackle real-world environments.

I once thought the primary value of humanoid robots was their relatable, user-friendly design. Picture a square robot on wheels navigating your home versus a humanoid—it’s clear why human-like robots feel more intuitive and approachable.

But, human-like robots play another significant role - they allow humans to operate them remotely, also known as teleoperations. For example, in a manufacturing setting, a skilled technician can wear motion-tracking equipment to guide a humanoid robot through complex assembly tasks, like connecting delicate electronic components or threading wires through tight spaces. The robot mirrors the technician's precise hand movements and finger positions in real-time.

This approach is crucial for gathering high-quality labeled data in real-world conditions. Through teleoperations, companies like Figure can collect diverse, precisely labeled data that mirrors human decision-making in complex environments. Such data is critical for training AI systems to perform effectively in real-world scenarios.

As Karpathy notes in the podcast, there's a significant transfer from automotive AI to humanoid robotics. Tesla's Optimus robot initially used the same computer and cameras as Tesla cars, showcasing how foundational AI systems can be adapted across different applications. This cross-pollination of technology and data between automotive and humanoid robotics accelerates development and scaling in both domains.

Economies of scale for LLMs

Language models also benefit from scale:

1. Inference becomes cheaper as server utilization is more predicable at scale, allowing tailored hardware and software optimization

2. Broader distribution means more users, which then generate more data. Data is the #1 blocker for generative AI companies, making scale crucial for continued improvement.

Every user interaction with ChatGPT's feedback system, from rating responses to choosing between alternatives, becomes valuable training data. For example, when ChatGPT users click the thumbs-down icon or select their preferred generation out of two options, this feedback is stored in an internal database. Later, it can be used to evaluate future models or apply Reinforcement Learning from Human Feedback (RLHF), helping to better align the model for ChatGPT users or, even better, personalizing responses based on individual user preferences and interaction patterns

Theoretically, OpenAI and other model providers can go as far as clustering users according to demographics like age and political views to better align ChatGPT’s response. When a model is better aligned with users, it becomes more engaging, increasing both usage and data generation, continuously fueling the improvement cycle in a positive feedback loop.

OpenAI started collecting more detailed usage feedback for its new Advance Voice Mode

Meta's launch of its AI chatbot is a prime example of this strategy. By integrating AI assistants into widely used platforms like Facebook and Instagram, Meta can collect vast amounts of real-world interaction data. Such data is critical for improving its models’ performance and adaptability across diverse contexts.

Such a strategy goes beyond language models, expanding to image, video, and audio. By contributing to the open-source AI ecosystem with the multimodal Llama and the state-of-the-art image segmentation model Segment Anything 2, Meta leverages both users and AI developers to improve the same underlying technology the powers Instagram, WhatsApp, and its recent blockbuster, the Meta Ray-Ban.

The Future of AI Systems

The next frontier in AI development involves:

1. Balancing software scalability and hardware constraints

2. Building data collection devices (e.g. robots) that resemble the real world to benefit from transfer learning and easier human labeling

3. Maximizing user distribution for (a) data collection (see Meta’s example above) and (b) brand recognition

Tech giants like Google and Microsoft already position themselves for this AI-dominated future, recognizing the critical role of economies of scale in their race to tomorrow's AI landscape.

From autonomous vehicles to humanoid robots and LLMs, the ability to efficiently scale both software and data collection is becoming a key differentiator. As AI continues to evolve, companies that can effectively leverage these economies of scale will likely lead the way in innovation and market dominance.

Deep Dives are short posts offering a perspective on AI-related topics. Some of my previous ones covered autonomous agents, document extraction with LLMs, and a review of 2023’s state-of-the-art AI.

A NotebookLM-powered podcast episode discussing this post:

The AI industry is entering a consolidation phase as major tech giants like Microsoft and Amazon acquire or hire talent from smaller AI startups, securing exclusive rights to cutting-edge technologies. Meanwhile, key players like Stability AI and Aleph Alpha, up until recently Germany’s sole language model provider, are pivoting away from ambitious projects, citing commercialization challenges, and the future of remaining model providers like Mistral and Cohere remains a work in progress as they strive to differentiate themselves and establish their positioning in the fast-paced generative AI landscape.

But is this merely a phase in the AI industry's evolution, or are we witnessing the start of an AI oligopoly? A brief observation of the recent acquisitions and pivots of leading AI labs, including Adept, Inflection, Character AI, and Aleph Alpha, and a humble forecast for the companies still in the race.

The rise of Big Tech acquisitions

AI startups have reached unprecedented valuations and garnered massive customer bases. Character AI, a company that develops advanced conversational AI models allowing users to interact with customizable AI characters like Elon Musk and a couples therapist, had over 5 million monthly active users a year ago and was ranked as the 3rd top generative AI web app by a16z. Founded by Noam Shazeer and Daniel De Freitas, two highly respected AI researchers from Google, the company recently reached a valuation of $1 billion.

Last August, Google acquired its founders and most of its research team for $2.7B.

Adept, a company building AI-powered agents to automate digital tasks, was well-positioned to capitalize on the recent autonomous agents trend. Its novel ACT-1 model, which debuted in September 2022, was celebrated in the AI community long before the concept of AI agents became popular. Adept raised $415 million and was valued at ~$1 billion.

Last June, Amazon acquired its founders to join its growing AI division, reporting to Rohit Prasad, the former Alexa head who’s leading a new AGI team focused on building LLMs.

Microsoft has also joined the wave of AI consolidation, paying a staggering $650 million to AI startup Inflection for a non-exclusive licensing deal and, crucially, to hire the bulk of its top talent, including its co-founders.

What makes these acquisitions striking is that they often don’t involve outright purchases of entire companies. Instead, Big Tech firms opt for reverse “acquihires,” focusing on poaching talent and securing non-exclusive licensing deals, thereby sidestepping antitrust scrutiny. This tactic allows tech giants to assimilate the best minds and technology without raising the ire of regulators.

This all makes sense in an era when AI researchers are hard to come by, and Google’s co-founder calls researchers to prevent them from leaving for competing AI labs.

Big AI dreams, bigger bills

Building world-class AI models is becoming prohibitively expensive for startups. Developing competitive AI products, especially large language models, requires enormous computational resources and access to vast datasets. For instance, Adept AI’s ambitious plans to create models that translate natural language into machine actions would have required not only technical innovation but also continuous fundraising.

For a startup, surviving in a space where you must lease computational power from the same giants you’re trying to compete with is unsustainable. Startups often rely on cloud services from AWS, Azure, or Google Cloud, directly paying their rivals for the means to develop their own products. Add to this the exorbitant cost of AI research talent, which can command salaries into the millions, and it is easy to see why many startups struggle to stay independent.

With these acquisitions, not only are startups losing their autonomy, but Big Tech’s growing control raises concerns about a potential oligopoly in AI.

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs, including Claude, Hugging Face, Deepgram. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

AI oligopoly

The ongoing trend of Big Tech’s dominance in AI raises a critical question: Will all major AI research and applications ultimately end up in the hands of a few giants? If so, what does that mean for competition and innovation?

OpenAI, once an independent startup, is now heavily funded by Microsoft. Anthropic relies on investment from Google and Amazon.

One might argue that AI consolidation is a natural phase, much like in other industries where larger companies eventually absorb smaller players. However, AI is different. The technology is poised to influence everything from personal assistants to healthcare diagnostics, and control over its development could have significant societal implications. As we can see from Microsoft's acquisition of Inflection AI’s talent and technology, these moves are not just about enhancing corporate capabilities but also about positioning Big Tech as the gatekeeper of the future of AI.

With companies like Microsoft, Amazon, and Alphabet investing heavily in their AI ecosystems, startups increasingly face a difficult choice: either scale up dramatically to remain competitive or sell out to a tech giant.

Europe exiting the LLM arena, or not?

The pressure to transition from research to commercial success has not spared Europe's AI sector.

Aleph Alpha, founded in 2019 by Jonas Andrulis, was once hailed as Germany's most promising AI startup. The company aimed to develop powerful language models that could compete with those created by American tech giants while adhering to European values and data protection standards.

Raising a $500 million Series B less than a year ago, the company's efforts were crucial for maintaining European technological sovereignty in AI, especially given the dominance of U.S. and Chinese firms in the sector.

However, Aleph Alpha recently announced a pivot away from developing LLMs. Instead, the company will focus on creating AI systems for specific enterprise applications. This shift marks a significant change in strategy and raises questions about Europe's ability to compete in the global AI race.

With another well-funded AI company out of the game, Europe's AI ambitions now lie with the 18-month-old Paris-based Mistral. Founded by former Google and Meta AI researchers, Mistral has been making waves with its open-source approach and impressive technical achievements.

Explore previous deep dives:

Further consolidation

What lies ahead for the teams still building foundational models?

Mistral, known for its cutting-edge generative models like Mistral 7B and the multimodal Pixtral 12B, now faces increasing pressure from both sides of the AI model development market. On the open-source front, Meta continues to release powerful models such as Llama 3.1 405B and 3.2 90B, which have outperformed Mistral’s proprietary Large 2 on benchmarks and the Chatbot Arena leaderboard. Commercially, Mistral’s hosted Large 2 model also lags behind competitors like Claude 3.5 Sonnet and GPT-4o, raising questions about the company’s ability to maintain its competitive edge in both open-source innovation and proprietary offerings.

The Toronto/SF-based AI company Cohere is another long-time player in this space. It recently raised $500 million at a $5.5 billion valuation, although reportedly only having $35M in annual recurring revenue.

Mistral and Cohere represent two distinct strategies in the AI market, both attempting to carve out a niche in the face of Big Tech dominance. Mistral, for instance, has embraced an open-source approach, releasing models like Mistral 7B to foster community adoption and differentiate itself from the proprietary systems of larger players like Microsoft and OpenAI. This strategy not only aligns with the growing demand for transparency and on-premise deployment but also positions Mistral as an alternative for developers seeking more accessible and customizable tools.

Despite limited commercial traction, both Mistral and Cohere have raised significant capital (Mistral raised $645 million at a $6 billion valuation)—largely due to investor confidence in their cutting-edge technology and the potential for future breakthroughs. Investors are betting that, even with modest current revenues, these companies can scale by offering specialized models for enterprise applications or monetizing their technological advancements in the long term.

This ability to secure funding in the absence of high recurring revenue highlights the immense value placed on innovation and technical leadership in the AI sector, where future returns can outweigh immediate profitability. Nonetheless, both companies will likely face a similar outcome as their previous AI peers unless they generate the cash flow to sustain further model development and keep investors happy.

The tale of a founding team of researchers

This current AI wave brought a new trait to the most funded and lucrative AI startups—a trait Adept, Cohere, Aleph Alpha, Mistral, and Character all share—the founders come from a deep research background.

All these companies launched groundbreaking models within months of incubation, but they all faced a common challenge: commercialization and leadership. Adept lost two co-founders early on, while Character AI raised $150M without generating any revenue.

The lack of go-to-market experience may be a significant factor in these companies' openness to acquisitions. Generating recurring revenue is a new and difficult challenge for researchers who are used to focusing more on innovation and technical breakthroughs than on commercial strategy.

Conclusion: A Consolidated AI Future?

While the recent flurry of acquisitions suggests that we are indeed entering the consolidation phase of AI investments, it’s not necessarily the end of innovation in the space. Startups still play a crucial role in advancing AI research, but as development costs rise, they may increasingly rely on Big Tech for survival.

However, the real question isn’t whether Big Tech will dominate AI—it already does—but whether that dominance will stifle innovation and investment.

What’s clear, however, is that the age of wild, independent AI startups might be coming to an end.

If you find AI Tidbits valuable, share it with a friend and consider showing your support.

Subscribe now

AI Supremacy

Most Impactful Generative AI Papers of 2023

☝ Image Created: Sahar Mor, January, 2023. Hey Everyone, I’ve really enjoyed reading AI papers in 2023 more than ever, and I hope you have as well! One of the best people to follow on breaking news in Generative A.I. is actually Sahar Mor , his LinkedIn posts…

Read more

2 years ago · 52 likes · 9 comments · Michael Spencer and Sahar Mor

Over 1,100 curated announcements and papers were featured in AI Tidbits in 2023. Only a handful of them changed the trajectory of AI research in the years to come, powering the products we use daily.

From Meta’s Llama to Stanford’s ControlNet and Microsoft phi—listing the 48 papers just as 2024 ushers in a fresh wave of groundbreaking discoveries.

Defining "most impactful" is a nuanced task, so I adopted a blend of objective and subjective criteria for selection:

• Objective - the paper’s number of citations and GitHub repository stars

• Subjective - papers I identified as having a significant influence across various modalities and applications

January

• BLIP-2, Salesforce - a novel pre-training approach for vision-language tasks, outperforming what was back then the state-of-the-art like Flamingo 80B in efficiency and zero-shot performance with significantly fewer parameters and ushering a new wave of vision language models throughout 2023

• InstructPix2Pix, Berkely - a conversational UI allowing image editing via textual prompts, enabling further research in image, and later video, editing using natural language

• MusicLM, Google - a transformer-based text-to-audio model capable of producing tracks of varying genres, instruments, and concepts with superior audio quality, piquing both professional and amateur musicians’ imagination and serving as a bedrock for further generative audio research

February

• LLaMA, Meta - an open 65B-parameter LLM trained on 1.4 trillion tokens, outperforming larger state-of-the-art LLMs like GPT-3 and PaLM-540B on most benchmarks, enabling the likes of Alpaca and Vicuna and sending the open-source LLM community off to the races

• ControlNet, Stanford - a groundbreaking architecture that robustly integrates spatial conditioning into text-to-image diffusion models, offering enhanced controllability and wide-ranging applicability

• Toolformer, Meta - a language model capable of teaching itself when and which external tools such as calculators and Wikipedia to use to generate accurate answers

• KOSMOS-1, Microsoft - a transformer-based multimodal LLM supporting a wide range of perception tasks such as image captioning, visual question answering, and zero-shot image classification

March

• Alpaca, Stanford - the first paper that mined instructions from large proprietary language models like ChatGPT to power smaller instruction-following language models

• PaLM-E, Google - a 562B-parameter multimodal AI that uses visual data to enhance its language processing capabilities, achieving exceptional performance in both robotic applications and visual-language tasks

• GPTs are GPTs, OpenAI - detailing the influence of LLMs on the American workforce, revealing its potential to affect 80% of American workers

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

April

• Segment Anything, Meta - the Segment Anything project, which includes both the Segment Anything Model (SAM) and SA-1B dataset, revolutionized image segmentation and was the bedrock to further novel papers like EdgeSam, MobileSAM, and EfficientSAM

• Generative Agents, Stanford - using language models to power autonomous agents and simulate realistic human behavior, leading to the creation of AutoGPT, BabyAGI, and many agent-powered applications

• Pythia, EleutherAI - one of the first fully open-source family of language models, Pythia is a suite of 16 LLMs from 70M to 12B parameters, facilitating research into model training dynamics, bias reduction, and performance

• Vicuna, LMSYS - Vicuna-13B is a LLaMA fine-tuned model that achieved outstanding performance, reportedly competitive with ChatGPT and Bard, at a training cost of just $300

• LLaVA, Microsoft - one of the first capable large multimodal models and the seed for further innovative research like LLaVA-1.5 and Qwen

May

• QLoRA, University of Washington - QLoRA fine-tunes large language models with reduced memory usage, resulting in high-performing models that are extremely efficient

• DragGAN, Max Planck Institute - a new model that took the internet by storm with its ability to manipulate the pose, shape, expression, and layout of any image

• Direct Preference Optimization (DPO), Stanford - a new algorithm for fine-tuning language models to align with human preferences, surpassing traditional complex RLHF methods

• Gorilla, Berkeley - a fine-tuned LLaMA model explicitly designed for API calls, surpassing the performance of GPT-4 in writing API calls and powering further research such as ToolLLM and AutoGen

• Voyager, Nvidia - an LLM-powered Minecraft agent that made the rounds thanks to its innovative approach of learning skills by generating code routines, later storing them in a database and retrieving them when needed

• Med-PaLM 2, Google - Med-PaLM 2 was the first LLM to perform at an expert test-taker level on the MedQA, reaching 85%+ accuracy. It was also the first AI system to reach a passing score on the MedMCQA dataset, scoring 72.3%

June

• MusicGen, Meta - a simple and controllable model for music generation using text prompts and input melodies. Also, one of the first audio models to be widely used by musicians for producing music.

• Textbooks are All You Need (phi-1), Microsoft - this paper proved (again) that high-quality data can enable smaller models to punch above their weight and outperform larger LMs. Since phi-1, Microsoft released phi-1.5 and phi-2 - a 2.7B model rivaling models up to 25x larger.

• WizardLM, Microsoft - one of the first papers to utilize LLM-generated instruction over human-generated ones, resulting in multiple powerful models: WizardLM, WizardCoder, and Wizard Math

• InstructBLIP, Salesforce - one of the first papers to explore vision-language instruction tuning, achieving SOTA zero-shot performance across diverse tasks

• LLaMA-Adapter, Shanghai AI Laboratory - the first efficient fine-tuning method for LLaMA-based models, adding minimal parameters to create high-quality instruction-following models in under an hour and the stem to further innovative papers like LLaMA2-Accessory, LLaMA-Adapter V2 multimodal, and ImageBind-LLM

July

• Llama 2, Meta - Meta’s first commercially permissive LLM enabled a host of new SOTA models from Code Llama to OpenHathi (the first Hindi LLM)

• Robotic Transformer 2 (RT-2), Google - a groundbreaking vision-language-action (VLA) model that combines web and robotics data to provide generalized instructions for robotic control, cited by subsequent novel papers like Robogen and JARVIS-1

August

• Code Llama, Meta - a commercially permissible SOTA model built on top of Llama 2, fine-tuned for generating and discussing code

• 3D Gaussian Splatting, Inria - Radiance Field methods have revolutionized novel-view synthesis of scenes in 2023 and this paper from Inria was the first one achieving state-of-the-art visual quality while maintaining competitive training times and allow real-time generation at 1080p resolution

• ModelScope, Alibaba - a powerful and commercially permissible text-to-video model that outperformed existing models with only 1.7B parameters

• OpenFlamingo, University of Washington - an open-source alternative to DeepMind's multimodal Flamingo, achieving up to 89% of its vision-language performance

September

• Mistral 7B, Mistral - a fully open-source model that outperformed all available open-source models up to 13B parameters. Mistral 7B’s edge is its efficiency, delivering strong performance with less computational demand than larger LMs, powering a suite of SOTA LLMs such as Solar 10.7B, Zephyr, OpenHermes, and even the multimodal Nous-Hermes-2-Vision

• Language Modeling Is Compression, DeepMind - discovering that LLMs are powerful lossless compressors, outperforming domain-specific counterparts (PNG, gzip, FLAC) in compressing data

• LongLoRA, MIT - leveraging LoRA to extend LLMs' context window, demonstrating significant computational savings with strong performance on various tasks and models

• Qwen, Alibaba - this suite of powerful multilingual LLMs, including versatile base models and specialized chat models in coding and mathematics, substantially outperformed Llama 2

• Large Language Models as Optimizers (OPRO), DeepMind - OPRO took the field of prompt engineering to the next level by leveraging LLMs to create prompts, outperforming human-designed prompts

October

• SteerLM, Nvidia - a technique that enables real-time customization of LLMs during inference, addressing the two major limitations of RLHF: (1) complex training setup and (2) static values that end users cannot control at run-time

• Zephyr, Hugging Face - a series of Mistral-based chat models with comparable performance to Anthropic's Claude 2 on AlpacaEval. The key innovation is distilled supervised fine-tuning, a method involving using the output from a larger, more capable ‘teacher’ model to train a smaller ‘student’ model

• MiniGPT-v2, KAUST - a unified interface for diverse vision-language tasks, enhancing image description, visual question answering, and visual grounding

• Set-of-Mark (SoM), Microsoft - a technique that boosts the performance of multimodal models like GPT-4V by segmenting objects in an image before passing it to the model

November

• Orca + Orca 2, Microsoft - Orca was one of the first to create tailored and high-quality synthetic data to equip smaller LMs with enhanced reasoning abilities, typically found only in much larger models. Orca progressively learns from complex explanation traces of GPT-4, with Orca-2 teaching small language models how to reason by employing advanced training methods and achieving reasoning levels comparable to models 5-10 times its size

• CogVLM, Tsinghua University - a novel model combining vision and language features, incorporating a trainable visual expert module, and achieving state-of-the-art performance across various multimodal benchmarks, rivaling larger models like PaLI-X 55B

• Medprompt, Microsoft - demonstrating how novel prompt engineering strategies can enable generalized models like GPT-4 to outperform specialist models like Med-PaLM 2 with significant efficiency gains

• Latent Consistency Models-LoRA, Tsinghua University - a distillation method that enables real-time image generation with Stable Diffusion

December

• LLM in a flash, Apple - on-device LLMs hold a big promise for AI and personal agents. Apple achieved up to 25x faster LLM inference thanks to dynamic storage and memory management, windowing for efficient data transfer, and a unique memory allocation algorithm.

• VideoPoet, Google - an innovative LLM that excels in diverse video generation tasks supporting multimodal inputs, diverse motion, and style generation, various video orientations, audio generation, and integration with other modalities

• FunSearch, DeepMind - in the pursuit of an AI capable of scientific breakthroughs, FunSearch represents a pioneering stride. It suggested a correct and previously unknown solution to the cap set problem and showcased the power of LLMs for making discoveries in mathematical and computer sciences.

• MagicAnimate, National University of Singapore - MagicAnimate is considered a step change in human image animation thanks to its improved quality over previous models. It is still far from an indistinguishable generated video, but its diffusion framework lays the foundation for progress in this space in 2024

Recent Deep Dives

If you find AI Tidbits valuable, share it with a friend and consider showing your support.

Subscribe now

Exactly a year ago, ChatGPT was one month old, Anthropic just released Claude, and Microsoft unveiled the first zero-shot model to clone someone’s voice. Long before Google Bard’s debut, Stanford’s inaugural autonomous agents paper, and the incorporation of the video-generating startup Pika Labs.

In 2023 alone, more than 1,100 curated announcements and papers were featured in AI Tidbits. It is hard to keep up with such mind-boggling progress when hundreds of innovative papers are published every single week, yet it is easy to forget the leapfrog progress the AI community has realized in just one year.

Just before we yell at ChatGPT once again as it got one detail wrong, let’s review the state-of-the-art today compared to December 2022 across different generative AI verticals.

Language Models

Open-source models

2023 was the year open-source language models started catching up, with Yi's 200k context window and Mistral's Mixture of Experts outperforming GPT-3.5, which was the SOTA just earlier this year.

Commercially permissive models with groundbreaking architectures such as Llama 2 gave birth to advanced models like Code Llama and whole new model families such as Vicuna. Small language models also had their moment, yielding dozens of tokens per second on consumer-grade devices.

Despite the significant advancements in open-source models in 2023, the evaluation of language models has yet to achieve consensus, and the AI community’s trust in researchers’ published benchmarks is at an all-time low.

Our report therefore only includes community-vetted LLMs or ones I’ve tinkered with and can vouch for. You can find the full list of almost 1,000 open-source language models on Hugging Face’s Open LLM Leaderboard.

As for the benchmarks I’ve considered as the performance proxy, I chose MMLU for its wide-ranging evaluation of language understanding across various domains and TruthfulQA to assess the AI's ability to provide factual and reliable information, ensuring a comprehensive analysis of both depth and accuracy.

SOTA models <=7B parameters

SOTA models 7B → 40B parameters

SOTA models >40B parameters

Mistral’s recently released Mixtral 8x7B model deserves a special shout-out. It is a Mixture of Experts (MoE) model that outperforms Llama 2 70B and GPT-3.5 on most benchmarks with 6x faster inference. It allows commercial use and judging on its predecessor, Mistral 7B, we can expect a host of Mixtral-based state-of-the-art language models coming in the next few months.

While the language models mentioned above are the state-of-the-art ones today, there are further noteworthy models released in 2023 that paved the way for these models to emerge: Flan-UL2, Llama 2, MosaicML MPT, Falcon, Vicuna, Dolly, phi 1.5, Orca 2, and WizardLM.

Just listing these models made me realize what a monumental year 2023 was for open-source language models, and we are scratching the surface of what's possible–most of these models are base models, i.e. they can be further fine-tuned for specific tasks to boost accuracy and achieve a smaller model size.

Commercial models

Surprisingly enough, GPT remained SOTA throughout 2023, with Gemini Ultra emerging as the only potential contender a few weeks ago, claiming to outperform GPT-4 across several benchmarks. However, given the divide between Gemini Pro's proclaimed and actual capabilities, such supremacy remains doubtful.

Anthropic also made great progress with its release of Claude 2.1, featuring the largest context window for proprietary LLMs (200k) at a substantially lower price than GPT-4.

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Multimodal AI

Multimodal AI refers to AI systems that can process and interpret multiple forms of data input, such as text, images, and audio, simultaneously or in an integrated manner, very much like humans.

Models specialized for a single modality, such as images or text, are limited in their capabilities and require much more training data. This contrasts with human learning, in which people learn much more efficiently thanks to different kinds of sensory inputs.

Progress in this space is paramount because it enables more sophisticated, efficient, and accurate AI applications and powers the next generation of wearables and embodied robotics.

2022 was relatively quiet for multimodal AI. The only major release was Meta’s data2vec, a model operating across multiple modalities, including speech, images, and text.

2023 saw several breakthrough models and frameworks, with CogVLM crowned as the SOTA open-source model.

CogVLM outperformed all previous models, such as LLaVA 1.5, an open-source chatbot alternative to ChatGPT that can converse using images and text, and Adept’s Fuyu, capable of understanding charts, documents, and interfaces. Meta also made significant contributions through its ImageBind and AnyMAL papers.

On the proprietary front, GPT-4V(ision) is the current industry leader, showcasing strong performance. Gemini Ultra seems like a close runner-up, though hard to tell as Google hasn’t opened developers access yet.

Autonomous Agents

An autonomous agent is an AI program capable of planning and executing tasks based on a given objective. Imagine asking an AI to “book a flight” or “create a website for people interested in renting their apartments when they’re away” - and the agent goes to work.

It does so by repeatedly asking “What should be the next steps to achieve the task at hand”, utilizing an LLM to answer this question and devise a plan to execute.

AI Agents that can do work for us is a great promise. In 2022, the main player was Adept with its ACT-1 model. Although mainly a demo, Adept showcased an agent that can find apartments on Redfin, take actions on Google Sheets, and input information into Salesforce using natural language.

On the contrary, 2023 was filled with open-source and commercial progress. Stanford’s Simulacra paper ignited everyone’s imagination and created innovative frameworks for agent builders (AutoGPT, BabyAGI) and agent-based applications (GPTEngineer, MetaGPT).

To date, the open-source SOTA model is CogAgent (demo), a powerful 18B visual language model capable of navigating mobile applications and websites, surpassing existing LLMs in both text and general VQA benchmarks.

There is still no clear winner on the commercial front. The leading startups operating in this space include Adept, Embra, Lindy, Induced, and HyperWrite AI. As the language models powering such agents become increasingly cheaper and more competent, 2024 might be the year of the first widely-used agent, our first AI companion.

Image generation

Image generation is the verticle having the most notable progress across modalities. Over the past two years, the transformation has been astounding, evolving from artificial-looking creations to artistry so professional it is virtually indistinguishable from human work.

2022 was the year in which image generation steered away from GANs and onto Diffusion models. In one buzzing summer, DALL-E 2, Stable Diffusion, and Midjourney were released, sending the image synthesis space off to the races. In the eighteen months since these releases, both the open-source and commercial models have undergone massive improvements.

Given the subjective nature of art, pinpointing a definitive 2023 SOTA is challenging. However, OpenAI's DALL-E 3 and Midjourney are the reigning champions, distinguished by their widespread usage and consistently high-quality outputs. In 2023, these image generation models overcame a significant hurdle that plagued image diffusion models in 2022: the accurate rendering of faces, hands, and text.

DALL-E 3 has introduced several advances over its 2022 predecessor with improved caption fidelity, overall image quality, and better steerability to reduce the generation of harmful images, demographic biases, and public figures.

In 2023, Midjourney graduated from Discord onto a web app and jumped from v5 to v6, showcasing notable improvements in detail and text generation capabilities.

Other noteworthy step changes include Stability AI’s SDXL Turbo, supporting high-resolution image generation at a fraction of the compute and cost, and Ideagram, which made the rounds with their text generation capabilities, a differentiating feature before DALL-E 3 was released.

On the open-source front, we got Fooocus, a package that allows everyone to generate Midjourney-quality images on their own computer without needing the advanced prompting techniques employed by Midjourney experts.

Generation latency has also substantially reduced thanks to novel techniques such as Latent Consistency Models (LCM), supporting real-time image generation at typing speed.

Video generation

Generative models in video weren’t really a thing in 2022. The space was practically dormant, with merely a teaser from Runway about its upcoming text-to-video model Gen-1 and a few closed releases such as Google’s Imagen Video and Meta’s Make-A-Video.

This changed in 2023, with numerous open-source packages emerging and notable advancements in commercial products, including an extended maximum video duration of 18 seconds (up from 4 seconds in 2022), alongside substantial enhancements in video quality and consistency.

On the commercial front, Pika Labs and Runway lead the pack, both developing their own foundation text-to-video models. This year, they added video inpainting and outpainting capabilities and the ability to render styles like anime and cinematic.

Another notable contender is HeyGen, which uses AI to translate videos into almost 30 languages by cloning the speaker’s voice and adjusting lip movements to the target language.

On the open-source front, VideoCrafter1 and ModelScope are the current state-of-the-art models. VideoCrafter1 can generate realistic and cinematic-quality videos from text with a resolution of 1024 × 576, outperforming previous open-source models in terms of quality. It features another image-to-video model designed to produce videos that strictly adhere to the content of the provided reference image. ModelScope can generate videos up to 25 seconds long at a reasonable quality. 

Also this year, Stability AI released Stable Video Difussion, turning images into short video clips, though permitting research use only.

Lastly, just before year-end, Meta released Emu Video, capable of efficiently generating high-quality, 512px, 4-second long videos, conditioned on text prompts and initial generated images, outperforming other state-of-the-art text-to-video models like Runway and Pika in terms of video quality and faithfulness to prompts.

Speech understanding

Transcribing sound and audio has been predominantly human-driven up until recently.

OpenAI’s release of Whisper in 2022, back then the SOTA open source transcription model, reignited the space and fostered a wave of speech-powered applications.

Since then, two more Whisper versions have been released in 2023, supporting 58 languages with reduced hallucinations and a 10%-20% lower Word Error Rate (WER). Thanks to packages such as Insanely Fast Whisper, inference latency has also been reduced to near real-time, enabling live conversational use cases.

The current commercial SOTA speech-to-text model is Nova-2 from Deepgram. It outperforms all alternatives in terms of accuracy, speed, and cost and achieves an 8.4 WER, 30% less than its 2022 predecessor.

Speech generation

The space of generative voice AI didn’t see much progress until this year, with incumbents like Amazon and Google providing ok-ish solutions.

In 2023, ElevenLabs wowed the industry with its remarkably fast text-to-speech model, outperforming all other models on latency and quality. AI-generated voice is no longer distinguishable from human voice. As competition heats up in this space with contenders such as OpenAI’s TTS, we can expect better and more affordable models in 2024.

Joe Biden’s voice cloned via ElevenLabs (source)

The open-source community is not far behind. Meta open-sourced Seamless–a suite of AI translation models that can generate and translate speech in real time.

Microsoft also introduced VALL-E X (open-source implementation), capable of synthesizing high-quality speech with only a 3-second enrolled recording of an unseen speaker as input, and Coqui released XTTS - a voice generation model capable of cloning voices into 17 different languages by using a 6-second audio clip.

Music generation

2022 ended with the release of Riffusion, an app that generates music from text using visual sonograms.

Like other generative AI vertices, 2023 was different. On the open-source front, Meta’s MusicGen demonstrated remarkable performance, turning text and melodies into music. MusicGen’s release was complemented with Amphion - an open-source toolkit for audio, music, and speech generation released a few weeks ago.

On the commercial side, Suno AI is the prominent industry leader, with Stable Audio and Google’s MusicFX as runner-ups. Suno generates original songs with lyrics and melody, all from text prompts and within a few minutes. To fully appreciate how far we have come, I strongly recommend experimenting with Suno through its web application or by using Microsoft Copilot.

In eight months, Suno extended song duration into full-length songs and added lyrics generation capabilities. April ‘23 (top) compared to Dec ‘23 (bottom):

2023 Deep Dives

If you find AI Tidbits valuable, share it with a friend and consider showing your support.

Subscribe now

Subscribe now

OpenAI's latest announcements have sent shockwaves through the tech world, signaling a new era for artificial intelligence. The company's ground-breaking announcements and bold moves would eliminate thousands of companies, from GPT-wrappers to deep tech ones, and pose a substantial threat to big tech incumbents.

These developments aren't just about shaking up the industry—they create a goldmine of possibilities for innovators and businesses in AI, turning previously unfeasible ideas into profitable ventures.

In this post, I'll unpack each of these pivotal announcements and outline the impact they have on those at the cutting edge of AI.

Announcements covered:

• GPT-4 Turbo with a 128k context length

• Substantial price reductions

• New text-to-speech (TTS) model and API

• Whisper v3

• Assistants API and Retrieval

• GPTs and the GPT Store, à la OpenAI’s App Store

• Other announcements

A new model: GPT-4 Turbo having a 128k context length

GPT-4 Turbo is a cheaper and faster version of GPT-4. It has an updated knowledge cutoff of April 2023, with OpenAI stating they will keep it up-to-date. It has a 128k context window so it can fit the equivalent of more than 300 pages of text in a single prompt. 

Availability

Available for all paying developers

Why it matters?

• Cheaper - having GPT-4 level capabilities at a 2.75x cheaper cost enables multiple applications that up until now didn’t make sense from a margins perspective.

• Faster - latency is a main consideration for LLM builders, especially for those building user-facing apps. A faster yet capable model unlocks applications for which latency is a core part of the user experience.

• Longer context window - context extends the language model’s knowledge, e.g. by augmenting it with your company’s Slack data. Up until today, the GPT-4 context window was limited to 32k tokens. A 4x longer context window means more data can fit in one prompt, reducing the need for retrieval augmented generation and the frequency of hallucinations by grounding GPT’s response in your data.

Examples of affected companies

LLM API companies such as AWS Bedrock, Google PaLM 2, Anthropic (Claude Instant), Hugging Face, and AI21

Substantial price reductions

The new GPT-4 Turbo API will be 2.75x cheaper than GPT-4. The same applies to the GPT-3.5 Turbo 16k model.

Developers using the 4k context version of GPT-3.5 would also benefit from a 33% reduction.

Fine-tuned GPT-3.5 Turbo 4K model input tokens are reduced by 3x as well.

Availability

Available for all paying developers

Why it matters?

GPT-powered app builders constantly monitor the cost of serving their apps. Cheaper generations translate to better margins, and better margins enable more innovation.

Substantially cheaper fine-tuned GPT models reduce the strain on long context windows and RAG applications.

Examples of affected companies

Anthropic, AWS Bedrock, Hugging Face, Google PaLM 2, AI21 Labs

New text-to-speech (TTS) model and API

Developers can now generate human-quality speech from text via a text-to-speech API. The current TTS model offers six preset voices to choose from and two model variants, tts-1 and tts-1-hd.

tts-1 is optimized for real-time use cases and tts-1-hd is optimized for quality, i.e. more human-like speech in exchange of latency.

OpenAI’s TTS supports real-time audio streaming and pricing starts at $0.015 per input 1,000 characters. For comparison, ElevenLabs, which is considered to be the current best TTS service, starts at $0.165 per 1k characters. That’s >10x the cost.

OpenAI's preset voice Nova in action:

Availability

Available for all developers

Why it matters?

Generating voice was the missing modality in OpenAI’s ecosystem. An OpenAI-grade TTS engine, that is cheaper and faster thanks to OpenAI’s economies of scale, will enable more business use cases and increase competition in the space.

Combined with OpenAI’s new GPTs vision (see below), a future of OpenAI-powered voice assistants is imminent.

Examples of affected companies

ElevanLabs, PlayHT, Coqui, Resemble AI, and cloud providers (AWS Polly, Google Text-to-Speech, Azure TTS)

Whisper v3

Whisper is OpenAI’s cutting-edge Automatic Speech Recognition model (ASR) model. Whisper’s open-source release in Sep 2022 had a profound impact on the speech2text industry and enabled many speech-powered applications since then.

Whisper large-v3 is OpenAI’s next-generation ASR which features improved performance across languages.

Availability

Immediately via the Whisper package on GitHub. API access will arrive in the “near future”.

Why it matters?

Natural language is how humans interact with one another. Incorporating Whisper v3 into the OpenAI API ecosystem will make this technology more accessible to developers, enabling them to integrate sophisticated speech-to-text features into their applications. This move would also drive the commoditization of the currently expensive speech2text market, allowing for a broader spectrum of uses and users.

Examples of affected companies

Deepgram, Azure TTS, Google Text-to-Speech AI, Amazon Transcribe

Assistants API and Retrieval

Using the Assistants API, developers can create agent-like AI within their applications, equipped with specialized functions like Code Interpreter, Retrieval, and function calling for efficient task execution. No more fancy Retrieval Augmented Generation (RAG) pipelines. Users just need to upload files to extend GPT’s knowledge. This alone eliminates GPT wrappers like ChatPDF and the need for smaller LangChain apps for conversing over your data.

The Assistants API also features persistent threads, which are designed to allow developers to manage long-running conversations and complex tasks without the limitations of a short-term memory context, enabling more coherent and contextually aware interactions over time.

Assistants can run Python code, manage diverse data, create visual content, and tap into external knowledge sources, eliminating the need for developers to embed or search through large datasets. Developers can also define and call custom functions through the API, e.g. calculating shipping costs based on weight, dimensions, and destination provided by the customer in a chat message.

Availability

Available for all developers under a beta program

Why it matters?

The Assistants API and interface is a no-code builder for intelligence AI agents. Non-engineers can build small pieces of software powered by OpenAI’s models and then share them with others. They can even monetize those agents through OpenAI’s new GPT Store (see below).

Users can also expand their assistant’s knowledge by uploading documents such as PDFs, Excel files, etc., removing the need for fancy RAG frameworks or customized LangChain scripts. OpenAI takes care of this all!

Examples of affected companies

AutoGPT, Characther.AI, ChatPDF, LangChain, Adept, Hugging Face

GPTs and the GPT Store, à la OpenAI’s App Store

Users can now create customized ChatGPT versions, dubbed GPTs, enabling users to craft personalized AI for specific uses like learning, work, or leisure, and to share these with others.

GPTs offer task-oriented assistance, such as explaining board game rules, teaching math, or designing graphics. They require zero coding skills and come with the capability to perform web searches, create images, and analyze data. 

GPTs would also have access to custom actions, connecting them to external APIs and enabling real-world interactions. This functionality can transform GPTs into versatile tools capable of interacting with databases, managing emails, or assisting with shopping. Building on the Plugins beta experience, the update gives developers more control and simplifies the transition for those with existing plugins, allowing them to seamlessly integrate these capabilities into their GPTs.

Lastly, OpenAI enables users to create and share custom GPTs, with a monetizable GPT Store launching soon.

Availability

Immediately for ChatGPT Plus and Enterprise users

Why it matters?

Customizable GPTs mark a pivotal shift towards more personalized and specific AI utility. Those without coding expertise will be able to craft AI tools for a range of tasks, thereby broadening the technology's accessibility and application.

For developers, it facilitates the integration of AI with other services, fostering more dynamic and practical uses in real-world scenarios.

For enterprises, custom GPTs offer a new frontier in customization, enabling the creation of AI tailored to specific corporate needs and proprietary data. Companies can now streamline operations, creating AI solutions for internal tasks like marketing, customer support, and employee onboarding while ensuring data privacy. Such integration makes many startups that raised lofty funding rounds to monetize generative AI for the enterprise almost obsolete.

For consumers, having specialized GPTs erodes the unique selling proposition of companies such as Character AI, which just last month saw almost 5M monthly active users.

Examples of affected companies

AutoGPT, Adept, Character AI, LangChain, Contextual AI, Hugging Face

Other notable announcements

• DALL-E 3 API access, with pricing starting at $0.04 per image

• Custom models - a new program from OpenAI that offers selected organizations the chance to collaborate with researchers to create bespoke GPT-4 models that are highly specialized to their domains, ensuring exclusive access and privacy for their proprietary data

• GPT-4 fine-tuning

• Function calling is now more efficient and accurate, with updates that enable the calling of multiple functions in a single message and enhancements that increase the likelihood of returning the correct function parameters.

• Higher rate limits - doubling the current tokens per minute limit and allowing users to request rate increases.

• JSON mode - ensuring GPT returns valid JSON outputs

• A new seed parameter for consistent completions, facilitating debugging, comprehensive unit testing, and enhanced control over model behavior.

• Copyright Shield - OpenAI will step in to defend and pay the costs incurred of any legal claims around copyright infringement for its users.

A new era

It is clear that the horizon for AI is not just broadening—it's being redefined.

With the advent of more efficient, cost-effective, and powerful models like GPT-4 Turbo, new text-to-speech capabilities, and an innovative GPT Store, OpenAI is empowering creators and businesses with tools that were once out of reach.

The implications are profound: barriers to entry are crumbling, enabling a democratization of technology that accelerates innovation at a breakneck pace. The sheer volume of possibilities for personalization, integration, and expansion in AI applications is staggering.

As developers, entrepreneurs, and technologists harness these breakthroughs, the next wave of AI utility and business models is upon us. To all those at the forefront of this change: the future is not just knocking, it has already stepped through the door.

In June 2020, OpenAI unveiled GPT-3. As a veteran in the document processing domain, I had long recognized the limitations of prevailing document extraction technologies, which largely relied on rigid, rule-based logic. I wondered if language models could be the answer to intelligent data extraction. And indeed, they were.

What started as a side project turned into a venture called AirPaper. Back then, GPT-3 was the cutting-edge language model, and it was only one API call away. The main challenges were that GPT-3 was expensive, 55x compared to today’s GPT-3.5 Turbo, and had a tiny context window of 2,048 tokens, compared to today’s 32k.

Another challenge was that language models, even if performant, only play along with text. This necessitated an extensive preprocessing phase to prepare documents for GPT: extracting text, à la OCR, structuring it in a way that would fit GPT’s limited context window, and intelligently mapping GPT's output to the relevant fields, such as invoice numbers or sales tax amounts on an invoice.

That was in 2020.

Enter Multimodal AI

The space of document intelligence has undergone massive shifts in recent years, with the underlying technology getting gradually commoditized. More and more state-of-the-art libraries were released, most of them with a commercially permissible license:

• Donut 🍩

• PaddleOCR

• layoutlm-document-qa

• Deepdoctection

• And the impressive LayoutLMv3, which is the only one not allowing commercial use

But then powerful multimodal AI in the form of LLaVA and GPT-4V arrived. Easily accessible for anyone with ChatGPT access.

I again wondered, can GPT-4V turn mere images of documents into structured data? The results were mindblowing.

Let's dive deeper into some of the use cases I've explored and their results.

Read more

Three years ago, I started a company that turns PDF and image documents into structured data. My twist? Using language models. Two years later, I decided to refocus my energy elsewhere with the main reason being commoditization. The OCR and document intelligence market were a race to the bottom, and even though I could raise VC money - I realized it was a lost war.

Since then, the same commoditization conclusion has spread to many more applications and fields. We are in an era in which AI can finally construct plausible essays. Actually, not only that - it can outperform doctors in summarizing clinical text, writing poems, and being your therapist. It can turn text into songs, images, and videos in mere seconds and at a fraction of the cost of what used to be cutting-edge technology just a few years ago.

Those with great aspirations would conclude such breakthroughs prompt starting a venture. Their mission? Democratize [enter profession here] and make it more accessible to all. The route? Raising money from venture capital funds.

That makes sense. A whole generation from 2008-2023, myself included, grew up in an environment that worships growth at all costs. The caliber of talent you can access will depend on the VCs on your About page. An era where VCs ask: “Well, what if Google builds the same thing?”, followed by a room-wide laugh, acknowledging the fact that incumbents don’t build, they buy.

Things have changed. Incumbents are no longer sleeping giants. The latest technology makes building new products too easy for companies to spend time integrating with that new YC startup, and disruptive progress is outpacing distribution, preventing companies from building their distribution moat before getting commoditized (see Jasper).

The tricky part? We are far from reaching a standstill. With the releases of multimodal GPT and Google’s Gemini around the corner, things are only going to become less predictable. The times of VC-backed ventures as the default is over, and the faster we adjust, the better.

P.S. It is not all gloomy. I also explore a few areas where I still see opportunities for VC-backed startups to operate in the outro section below.

New Truths

#1 The energized giants

Read more

I was somewhat new to the Payments space when I joined Stripe. I remember being dazzled by the sheer amount of complexities taking place when one hits the “Book a Ride” button to catch an Uber. It was at Stripe where I got to learn about a new concept - Multiprocessor. 

Imagine you are the Head of Payments at Amazon or Uber—you operate across dozens of countries, serve multiple types of customers, and employ different pricing structures, such as subscription-based and one-time payments. How can you maximize your approval rate, i.e. ensure every legitimate payment is approved? How do you avoid losing millions of dollars for every minute your partnering payment processor is down?

Enter Multiprocessor.

The likes of Amazon partner with more than one payment processor (Stripe is one). That way, they ensure redundancy by falling back to another processor if their main one is down. They also achieve maximal approval rates by routing payments to regional processors based on their geography, e.g., using a European processor for a Norway-based cardholder. Beyond better payment performance, they get to leverage this multiprocessor setup to drive down partners’ costs and push them to deliver better results. Otherwise, they will churn some of their payment traffic.

Now let’s talk about language models. Today, there are five major proprietary LLM providers: OpenAI (GPT), Google (PaLM), Anthropic (Claude), Inflection (Pi), and Cohere. There are dozens more open-source language models, small and large, with the best-performing ones including Meta’s Llama 2, Stability AI’s StableBeluga2, and the recently-introduced Platypus.

Each one of these models has its own traits and quirks: faster, cheaper, easier to experiment with, fits on consumer hardware, or is better suited for specific language tasks. Numerous guides teach how to choose the right LLM, with everyone stating the obvious - start with OpenAI and move from there. The truth is everyone anyway defaults to GPT-4. “We will take care of costs and latency later” is usually what I hear.

There is merit to it, but why choose one?

A unified LLM API layer, à la LLM Multiprocessor

Read more

Read more

“What do you mean there is an open-source library for that? We built the entire thing ourselves” is a quote I often hear from builders in the LLM space. I’ve been building with LLMs for the last year and turned my personal list of >60 useful packages into a public table so others won’t experience the same frustration.

Each one of these packages will save you hours and enable use cases you were interested in building yet didn't have the technical chops to do so.

Highlighted packages (link to full list below)

1. LlamaIndex - a framework to augment LLM applications with private data

2. GPT Engineer - let AI code full apps for you

3. ResearchGPT - an LLM-based research assistant that allows you to have a conversation with a research paper

4. Autodoc - a toolkit for auto-generating codebase documentation using LLMs

5. web-llm - run LLMs in web browsers without the need for a backend server

6. Text generation web UI - web UI for running LLMs like Llama 2, llama.cpp, and Vicuna

7. AnythingLLM - turn any document into an intelligent chatbot

8. Vocode - build voice-based LLM agents

9. EasyLM - a one-stop solution for pre-training, finetuning, evaluating, and serving LLMs in JAX/Flax.

10. Motorhead - a memory and information retrieval server for LLMs

Full table with >60 packages (I keep it up to date):

Read more