Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Computer-Use Agent

Running OpenAI’s computer-use model just got a whole lot easier. A new open-source package called Computer-Use Agent (Cua) lets you run OpenAI’s computer-use-preview model inside a full-featured macOS virtual machine.

Until now, using OpenAI’s computer-use model meant working with limited APIs or browser-based sandboxes (like ChatGPT Operator). With Cua, you can:

1. Interact with native apps like Finder, Terminal, and Final Cut Pro

2. Automate real desktop workflows, not just web tasks

3. Run everything locally for better privacy and control

4. Avoid the pain of wiring screenshots, actions, and VM interfaces manually

Under the hood, Cua uses Apple’s Virtualization framework to launch macOS VMs on Apple Silicon and runs an event loop that handles clicks, typing, scrolling, and more — all based on OpenAI’s structured responses.

With Cua, you can build an AI agent that files your expenses across desktop and browser apps — moving between Excel, Chrome, and system dialogs, or launch a self-healing QA bot that installs your macOS app, navigates its UI, and reports bugs automatically.

If you're experimenting with OS-level agents, GUI automation, or reinforcement learning on real UIs, Cua provides the missing infrastructure.

GitHub repo https://github.com/trycua/cua

Portia

Portia just open-sourced a powerful agent framework that solves the three biggest challenges of production AI agents.

The new open-source library addresses the core problems that plague most agentic frameworks with a refreshingly practical approach: separating agent planning from execution, allowing clear human oversight and structured collaboration at every step.

Package highlights:

1. Multi-agent planning - the SDK uses few-shot prompting to teach your agents what successful plans look like, significantly boosting reliability.

2. Stateful execution - agents track their own progress and proactively request human input whenever necessary. Think: authentication requests, missing data, or asking for missing context when task execution hits an unexpected scenario.

3. Streamlined security - just-in-time authentication handovers ensure your agents can securely interact with popular tools like Google Calendar, Zendesk, and Hubspot without compromising credentials

This architecture solves persistent roadblocks like unpredictable behavior, lack of human oversight, and cumbersome authentication processes, making production-ready agent deployment realistic and scalable.

It's open-source, production-ready, and works out-of-the-box with major LLM providers including OpenAI, Anthropic, Mistral, Gemini, and Azure.

GitHub repo https://github.com/portiaAI/portia-sdk-python

My recent post on coding with AI

Goose - LLM-powered Agents

Block recently launched Goose, an open-source framework for developers building AI agents, and I got to tinker with it the last few days.

Goose is a free open framework for LLM-powered agents, from DeepSeek to proprietary models from OpenAI, Google, or Anthropic.

Unlike other agent frameworks, Goose is designed for software development tasks. The framework has already proven valuable for tasks like:

1. Conducting code migrations - from Ember to React or Ruby to Kotlin

2. Navigating new projects in unfamiliar languages - eliminating steep learning curves

3. Generating unit tests - quickly increasing code coverage above specific thresholds

The neat thing about Goose is that it's extremely easy to extend its capabilities by leveraging MCP servers like Figma, Google Drive, and Asana.

Other notable frameworks for coding tasks include OpenHand and Cline.

GitHub repo https://github.com/block/goose

Andrej Karpathy leverging LLMs

Read more

Each month, I'll share my five seven top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes seven posts instead of five—there were just too many good ones to leave out!

This post covers groundbreaking developments in AI agents and document processing, from Anthropic's foundational patterns for building effective agents to LlamaIndex's new Agentic Document Workflows. You'll learn about DeepSeek's surprising findings about prompting reasoning models, cutting-edge tools for PDF processing and web automation, and explore how LLMs handle structured table data.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. GOT-OCR 2.0

I finally had the chance to explore a new document extraction technique introduced in a paper last September. Bonus: the code and model are free to use (Apache 2.0).

This new approach, called General OCR Theory (GOT-OCR2.0), suggests a unified end-to-end model that handles tasks traditional OCR systems struggle with.

Unlike legacy OCR, which relies on complex multi-modular pipelines, GOT uses a simple encoder-decoder architecture with only 580M parameters that outperforms models 10-100× larger.

Paper highlights:

1. Unified architecture - a high-compression encoder paired with a long-context decoder that handles everything from scene text to complex formulas

2. Stunning performance - delivers nearly perfect text accuracy on documents, surpassing Qwen-VL-Max (>72B) and other leading models

3. Versatility beyond text - processes math formulas, molecular structures, and even geometric shapes

4. Interactive capabilities - supports region-level recognition guided by coordinates or colors

I just tried it out and was blown away by how it handles complex documents with mixed content types. The ability to convert math formulas from Arxiv PDFs to Mathpix format alone is worth exploring this model.

What strikes me most about GOT is how it challenges the notion that only billion-parameter LLMs can tackle complex visual tasks.

Paper + code + model can be found in their GitHub repo https://github.com/Ucas-HaoranWei/GOT-OCR2.0

Last month’s LinkedIn Highlights

2. Six Levels of Agenic Behavior

I came across a new framework that brings clarity to the messy world of AI agents with a 6-level autonomy hierarchy.

While most definitions of AI agents are binary (it either is or isn't), a new framework from Vellum introduces a spectrum of agency that makes far more sense for the current AI landscape.

The six levels of agentic behavior provide a clear path from basic to advanced:

𝐋𝐞𝐯𝐞𝐥 0 - 𝐑𝐮𝐥𝐞-𝐁𝐚𝐬𝐞𝐝 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰 (𝐅𝐨𝐥𝐥𝐨𝐰𝐞𝐫)
No intelligence—just if-this-then-that logic with no decision-making or adaptation. Examples include Zapier workflows, pipeline schedulers, and scripted bots—useful but rigid systems that break when conditions change.

𝐋𝐞𝐯𝐞𝐥 1 - 𝐁𝐚𝐬𝐢𝐜 𝐑𝐞𝐬𝐩𝐨𝐧𝐝𝐞𝐫 (𝐄𝐱𝐞𝐜𝐮𝐭𝐨𝐫)
Shows minimal autonomy—processing inputs, retrieving data, and generating responses based on patterns. The key limitation: no control loop, memory, or iterative reasoning. It's purely reactive, like basic implementations of ChatGPT or Claude.

𝐋𝐞𝐯𝐞𝐥 2 - 𝐔𝐬𝐞 𝐨𝐟 𝐓𝐨𝐨𝐥𝐬 (𝐀𝐜𝐭𝐨𝐫)
Not just responding but executing—capable of deciding to call external tools, fetch data, and incorporate results. This is where most current AI applications live, including ChatGPT with plugins or Claude with Function Calling. Still fundamentally reactive without self-correction.

𝐋𝐞𝐯𝐞𝐥 3 - 𝐎𝐛𝐬𝐞𝐫𝐯𝐞, 𝐏𝐥𝐚𝐧, 𝐀𝐜𝐭 (𝐎𝐩𝐞𝐫𝐚𝐭𝐨𝐫)
Managing execution by mapping steps, evaluating outputs, and adjusting before moving forward. These systems detect state changes, plan multi-step workflows, and run internal evaluations. Examples like AutoGPT or LangChain agents attempt this, though they still shut down after task completion.

𝐋𝐞𝐯𝐞𝐥 4 - 𝐅𝐮𝐥𝐥𝐲 𝐀𝐮𝐭𝐨𝐧𝐨𝐦𝐨𝐮𝐬 (𝐄𝐱𝐩𝐥𝐨𝐫𝐞𝐫)
Behaving like stateful systems that maintain state, trigger actions autonomously, and refine execution in real-time. These agents "watch" multiple streams and execute without constant human intervention. Cognition Labs' Devin and Anthropic's Claude Code aspire to this level, but we're still in the early days, with reliable persistence being the key challenge.

𝐋𝐞𝐯𝐞𝐥 5 - 𝐅𝐮𝐥𝐥𝐲 𝐂𝐫𝐞𝐚𝐭𝐢𝐯𝐞 (𝐈𝐧𝐯𝐞𝐧𝐭𝐨𝐫)
Creating its own logic, building tools on the fly, and dynamically composing functions to solve novel problems. We're nowhere near this yet—even the most powerful models (o1, o3, Deepseek R1) still overfit and follow hardcoded heuristics rather than demonstrating true creativity.

The framework shows where we are now: production-grade solutions up to Level 2, with most innovation happening at Levels 2-3. This taxonomy helps builders understand what kind of agent they're creating and what capabilities correspond to each level.

Full report https://www.vellum.ai/blog/levels-of-agentic-behavior

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

3. Skyraven

Traditional web automation is dying as developers waste countless hours maintaining brittle XPath selectors. Skyvern, a new open-source package, revolutionizes browser automation by combining LLMs with computer vision.

Unlike traditional automation tools that break when websites change, Skyvern uses visual understanding and natural language processing to dynamically interpret and interact with web interfaces. This enables developers to:

→ Build website-agnostic automations - create workflows that work across multiple sites without custom code

→ Handle complex inference tasks - automatically reason through form responses like eligibility questions

→ Execute multi-step sequences - coordinate multiple agents for tasks like authentication, navigation, and data extraction

Packages like Skyvern signal the emergence of truly adaptable web agents. Instead of hard-coded rules, we see AI systems that can understand and navigate the web like humans do - reading content, making decisions, and handling edge cases autonomously. I wrote more about it in my latest AI Agents blog series.

GitHub repo https://github.com/Skyvern-AI/skyvern

4. Maestro

Read more

Each month, I'll share my five six top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes six posts instead of five—there were just too many good ones to leave out!

This post covers groundbreaking developments in AI agents and document processing, from Anthropic's foundational patterns for building effective agents to LlamaIndex's new Agentic Document Workflows. You'll learn about DeepSeek's surprising findings about prompting reasoning models, cutting-edge tools for PDF processing and web automation, and explore how LLMs handle structured table data.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. Long Context Prompting Tips

Anthropic shared a counterintuitive yet powerful tip that boosts Claude's accuracy by 30% when working with long documents/prompts.

The secret? Place your lengthy documents (~20K+ tokens) at the TOP of your prompt before your actual query. While this might seem counterintuitive, internal tests show this simple change significantly improves response quality across all Claude models.

This becomes crucial when dealing with multiple documents. For optimal results:

1. Documents first - place all your data inputs at the beginning

2. Structured organization - use XML tags to separate documents and metadata

3. Specific query - end with a clear, focused question

As language models' context window grows in size and companies increasingly rely on LLMs to process complex datasets, reports, and documentation, this technique ensures more reliable and accurate results.

P.S. For those working with multi-document analysis, I highly recommend structuring your content with XML tags - it provides additional clarity and helps the model better understand document relationships.

2. Agents for Computer Use Repository

While everyone's talking about AI agents, most developers and researchers are missing out on the most comprehensive collection of computer use frameworks, papers, and tools ever assembled.

Awesome Agents for Computer Use is a curated repository documenting the recent rapid progress of AI agents that can autonomously control computers through clicks, keystrokes, and API calls. From Anthropic's Claude Computer Use to Microsoft's OmniParser and Self-Operating Computer framework, it covers the entire landscape of computer control agents.

It features:

• Research papers - featuring 30+ recent publications on GUI agents, from foundational models to safety considerations

• Open-source frameworks - documenting practical implementations like AutoGen, Browser Use, and OpenInterpreter

• Commercial solutions - tracking industry developments from major players like Anthropic and emerging startups

The rise of computer-controlling AI agents marks a pivotal shift in human-computer interaction. As these systems mature, we're moving towards a future where AI assistants won't just give advice - they'll directly help us accomplish complex tasks across applications and platforms (I wrote more about this topic here).

Repo https://github.com/francedot/acu

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

3. GPT-4.5 Pro Tip

Pro tip from the research team @ OpenAI on how to improve GPT-4.5 performance: add the following text to the start of your system message.

You are a highly capable, thoughtful, and precise assistant. Your goal is to deeply understand the user's intent, ask clarifying questions when needed, think step-by-step through complex problems, provide clear and accurate answers, and proactively anticipate helpful follow-up information. Always prioritize being truthful, nuanced, insightful, and efficient, tailoring your responses specifically to the user's needs and preferences.

OpenAI internal evals show it results in better performance.

Try it out in the OpenAI Playground https://platform.openai.com/playground/chat?preset=7CywXwBqWRC5quhkU9LEFv6A

4. Chorus - Chat with Multiple AIs on Your Desktop

Read more

Each month, I'll share my five eight top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry. This edition includes eight posts instead of five—there were just too many good ones to leave out!

This post covers groundbreaking developments in AI agents and document processing, from Anthropic's foundational patterns for building effective agents to LlamaIndex's new Agentic Document Workflows. You'll learn about DeepSeek's surprising findings about prompting reasoning models, cutting-edge tools for PDF processing and web automation, and explore how LLMs handle structured table data.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. Building Effective AI Agents by Anthropic

Anthropic clarifies the fuzzy definition of AI Agents by introducing a critical architectural distinction: workflows are systems with predefined code paths, while agents dynamically direct their own processes.

After building agents for a year, they identified five fundamental patterns that drive successful agentic implementations:

1. Prompt chaining - breaking tasks into sequential steps, useful for complex operations like content generation and translation

2. Routing - directing inputs to specialized handlers, perfect for customer service and model optimization

3. Parallelization - running subtasks simultaneously through sectioning or voting, ideal for code review and content moderation

4. Orchestrator-workers - using a central LLM to coordinate task delegation, essential for complex coding projects

5. Evaluator-optimizer - implementing feedback loops for iterative refinement, perfect for improving search results

Success isn't about building the most sophisticated system - it's about choosing the right pattern for your specific needs. Start simple, measure performance, and only add complexity when simpler solutions fall short.

Anthropic’s post (highly recommend reading it) https://www.anthropic.com/research/building-effective-agents

2. Agentic Document Workflows

LlamaIndex just unveiled a new approach involving AI agents for reliable document processing, from processing invoices to insurance claims and contract reviews.

LlamaIndex’s new architecture, Agentic Document Workflows (ADW), goes beyond basic retrieval and extraction to orchestrate end-to-end document processing and decision-making. Imagine a contract review workflow: you don't just parse terms, you identify potential risks, cross-reference regulations, and recommend compliance actions.

This level of coordination requires an agentic framework that maintains context, applies business rules, and interacts with multiple system components.

Here’s how ADW works at a high level:

1. Document parsing and structuring – using robust tools like LlamaParse to extract relevant fields from contracts, invoices, or medical records.

2. Stateful agents – coordinating each step of the process, maintaining context across multiple documents, and applying logic to generate actionable outputs.

3. Retrieval and reference – tapping into knowledge bases via LlamaCloud to cross-check policies, regulations, or best practices in real-time.

4. Actionable recommendations – delivering insights that help professionals make informed decisions rather than just handing over raw text.

ADW provides a path to building truly “intelligent” document systems that augment rather than replace human expertise. From legal contract reviews to patient case summaries, invoice processing, and insurance claims management—ADW supports human decision-making with context-rich workflows rather than one-off extractions.

Ready to use notebooks https://github.com/run-llama/llamacloud-demo/tree/main/examples/document_workflows

More open-source tools for AI agent developers in my recent blog post https://www.aitidbits.ai/p/open-source-agents

Become a premium member to access the LLM Builders series, $1k in free credits for leading AI tools and APIs, and editorial deep dives into key topics like AI Voice Agents. It's also a great way to show your support :)

Many readers expense the paid membership from their learning and development education stipend.

Upgrade to Premium

3. Vanna

Turning questions into SQL statements has been an area of research and development for over a decade. An open-source package called Vanna uses a retrieval augmented generation approach to crack the code.

Vanna is a Python-based, MIT-licensed framework that allows non-data folks to interact and ask questions about their SQL databases.

At its core, Vanna employs a RAG model that leverages a large corpus of data, including a diverse range of SQL queries and their natural language descriptions. When a query is received, Vanna searches this corpus to find similar queries and their corresponding SQL translations. This step enables Vanna to understand the context and structure of the query better.

Using the insights gained from the retrieved examples, Vanna generates the SQL query that matches the user's natural language request. This involves structuring the select statements, where clauses, joins, and other SQL components are based on the intent and requirements identified in the initial query.

Pro tip: Vanna often hallucinates when it doesn't know the content of your table's columns. I therefore recommend providing a few examples through the train() method.

The GitHub repo already includes ready-to-use templates to deploy Vanna in Slack, Streamlit, or a Flask endpoint https://github.com/vanna-ai/vanna

4. DeepSeek & How to prompt reasoning models

DeepSeek's new model doesn't just beat GPT-4o - it proves 𝐰𝐞'𝐯𝐞 𝐛𝐞𝐞𝐧 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥𝐬 𝐰𝐫𝐨𝐧𝐠 𝐚𝐥𝐥 𝐚𝐥𝐨𝐧𝐠.

Read more

Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. MindSearch

An open-source search engine is rivaling top-tier AI products like Perplexity.ai Pro and ChatGPT web search.

MindSearch is an innovative AI search engine framework that combines LLMs and a multi-agent system to tackle three critical issues that often limit LLM-powered search engines:

1. LLMs struggle to decompose complex queries into simpler, actionable requests

2. Search results often contain too much noise, making it hard to filter and extract relevant information

3. Iterative searches can quickly overload the LLM’s input length capacity

MindSearch utilizes two main components:

• WebPlanner - decomposes complex queries into sub-tasks and creates a dynamic graph structure for problem-solving

• WebSearcher - conducts fine-grained searches and delivers summarized information back to WebPlanner for further refinement

This approach allows MindSearch to handle massive web content (e.g., more than 300 pages) effectively, surpassing limitations faced by traditional LLM-based search systems.

Code https://github.com/InternLM/MindSearch

2. Gemini Multimodal Playground

Holiday coding project: Build voice agents that can see with Google's new Gemini 2.0 model and my new real-time Multimodal Playground repo.

The playground implements voice and video-based interactions with the new Gemini model, allowing natural conversations in real-time while solving the critical background noise challenge using Voice Activity Detection (VAD).

In the last few days, I added a full-stack web app to interact with Gemini (see video below) along with a standalone script for those eager to quickly dive into building real-time voice agents.

Google’s real-time Gemini model is a game-changer, enabling you to independently create production-ready voice agents for industries like customer service, education, and healthcare in a matter of days.

Happy holidays. Go build! https://github.com/saharmor/gemini-multimodal-playground

3. Meta Apollo

Video understanding has been lagging behind text, image, and audio modalities—until now.

Meta and Stanford researchers unveiled Apollo, a new family of state-of-the-art video-centric large multimodal models (video-LMMs) designed to close this gap. Unlike prior efforts, Apollo sets a new standard by efficiently analyzing hour-long videos and achieving breakthrough results on multiple benchmarks.

Paper highlights:

1. Scaling Consistency - design decisions made with smaller models transfer reliably to larger ones, drastically cutting computational costs

2. Advanced video sampling techniques - Apollo uses FPS sampling, outperforming traditional uniform sampling methods

3. Streamlined evaluation - the new ApolloBench benchmark evaluating video-LMMs efficiently, reducing evaluation time by 41x while maintaining accuracy

Apollo’s superior video comprehension capabilities pave the way for breakthroughs like real-time video summarization for content creators, better temporal reasoning for medical diagnostics, and enhanced video analytics for autonomous driving.

With Apollo, video understanding might finally catch up to its multimodal counterparts.

Project page https://apollo-lmms.github.io

4. Claude PDF API

Anthropic has introduced a powerful new PDF-processing feature in its Claude API, surpassing basic text extraction, and it has largely flown under the radar.

Historically, many LLMs stumble when documents include complex elements like images, charts, and LaTeX formulas. But Anthropic’s latest upgrade manages to parse both textual and visual content within a PDF—no extra coding wizardry needed.

Key capabilities include:

1. Automatically parsing PDF text, images, and tables for further analysis, from answering questions about the attached PDF to turning unstructured data into formatted JSONs

2. Providing insight on charts and diagrams by evaluating visual context, not just textual tags

3. Extracting and interpreting LaTeX for scientific or technical documentation

It works by splitting each PDF into two components: the text is extracted as normal, and the entire page is converted into an image. Claude then merges text and visual context for a more holistic understanding. It’s essentially combining LLM intelligence with basic computer vision techniques.

The API supports up to 32MB or 100 pages of PDF content and pricing is similar to the LLM pricing so there’s no premium cost for PDF analysis.

This API could dramatically streamline how we handle financial reports, legal docs, or any PDF requiring detailed interpretation.

Ready-to-run notebook analyzing Anthropic's constitutional AI paper here https://github.com/anthropics/anthropic-cookbook/blob/main/misc/pdf_upload_summarization.ipynb

5. LLaMa-Factory

When is it better to fine-tune a language model than using prompt engineering or RAG? Here’s a clear framework you can apply along with an open-source library I use for fine-tuning.

Good reasons to fine-tune:

1. Emphasizing knowledge that already exists in the model - for instance, in a text-to-SQL task, fine-tuning can be used to emphasize specific SQL dialects or to avoid error-prone edge cases, utilizing the comprehensive understanding of SQL syntax, dialects, and database functionality that the model already possesses.

2. Customizing the structure or tone of responses - fine-tuning can modify the structure or tone of a model's output, such as making the model output valid JSON, which is beneficial for programmatic interactions where handling invalid JSON could lead to many downstream error cases. This includes fine-tuning a model to your company’s writing style.

3. Teaching a model very complex instructions - fine-tuning allows for showing the model many more examples than can be included in a model's context window, which is helpful for complex instructions. This leads to cheaper and faster inference.

Wrong reasons to fine-tune:

1. Adding new knowledge to the base model - the knowledge in a large language model is established during the pre-training runs. New knowledge can't effectively be introduced during the limited scope of fine-tuning. RAG is better suited in such cases.

2. Quickly iterating on a new use-case - fine-tuning involves a slower feedback loop and requires substantial investment in creating the dataset and other aspects of the fine-tuning process. Therefore, it's not suitable for rapid iteration of new use cases.

My preferred tool for fine-tuning open language models is LLaMA-Factory. It features 100+ different large language models, including Meta’s Llama-2, Google’s Gemma, and Mistral’s Mixtral. It also supports advanced algorithms like LoRA, QLoRA, and GaLore for optimized performance.

GitHub repo https://github.com/hiyouga/LLaMA-Factory

Last month’s LinkedIn Highlights

October marked a pivotal moment for agentic AI, as interest from industry and academia reached new heights, alongside groundbreaking video generation releases that democratized previously exclusive capabilities.

Genmo's release of Mochi 1 and Rhymes AI's Allegro led the charge, bringing commercial-grade text-to-video generation into the open-source domain. Meanwhile, industry giants made significant moves - Anthropic released Claude 3.5 Sonnet and Haiku with unprecedented computer use capabilities, GitHub expanded Copilot with Claude 3.5 and Gemini 1.5 integration, and OpenAI introduced Canvas for real-time writing and coding assistance.

The open-source community continued its remarkable momentum, with Nvidia's Llama-3.1-nemotron surpassing GPT-4 and Claude 3.5 on key benchmarks, while Apple made waves by open-sourcing Depth Pro and Ferret-UI 2, pushing the boundaries of on-device AI capabilities.

Multimodal AI saw further developments with Rhymes AI's Aria, the first open-source mixture-of-experts multimodal model, and Meta's innovative Spirit LM, combining text and speech capabilities.

These breakthroughs and numerous advances in autonomous agents, audio generation, and AI tools paint a picture of rapid democratization across AI domains.

Let's dive in!

Overview

• ✨ Special Feature: Open Video Generation

• Industry announcements (10 entries)

• Large Language Models

  • Open-source (9 entries)

  • Research (8 entries)

• Autonomous Agents (8 entries)

• Multimodal (7 entries)

• Image and Video (9 entries)

• Audio (3 entries)

• AI Tools (3 entries)

• Open-source Packages (7 entries)

  ---

✨ Special Feature: Open-source Video Generation

1. Genmo openly releases Mochi 1 - a text-to-video model delivering smooth 30fps videos with precise motion and accurate prompt adherence, with downloadable weights on Hugging Face and a commercially permissive license

2. Rhymes AI releases Allegro - a 2.8B open-source model capable of generating cinematic 6-second videos from text prompts at 15 FPS and 720p resolution

3. Meta AI presents MovieGen - a next-gen model family that generates HD personalized videos and synchronized audio from text prompts, enabling users to create and edit videos featuring their own faces

4. Peking University develops a new method for efficient video generation, producing smooth, high-quality 10-second videos in 768p resolution at 24 FPS

Recent Deep Dives

Industry announcements

1. Anthropic releases new versions of its LLMs Claude 3.5 Sonnet and Haiku, boasting top-tier performance in coding and problem-solving, surpassing OpenAI o1-preview, along with new human-like software interaction capabilities, enabling the model to click, type, and automate tasks directly through GUIs

2. OpenAI introduces Canvas - a visual interface that simplifies real-time edits on writing and coding tasks within ChatGPT

3. OpenAI releases Chat Completions API with support for text and audio, enabling both asynchronous audio experiences and real-time interactions

4. Google elevates its podcast-generating NotebookLM, introducing customizable Audio Overviews and giving users the ability to fine-tune AI summaries with specific instructions 

5. GitHub unveils multi-model Copilot in its annual developer conference, adding Claude 3.5 and Gemini 1.5, as well as launches GitHub Spark, an AI-native tool that builds micro web apps entirely through natural language with no coding required

6. Runway releases Act-One - a cutting-edge tool for transforming simple video and voice inputs into expressive character performances

7. ElevenLabs unveils Voice Design, allowing users to create unique voices from a text prompt alone

8. Ideogram releases Canvas - an AI-powered image editor offering inpainting and outpainting capabilities, outperforming competitors like Midjourney

9. Sequoia Capital publishes a report on the evolution of generative AI, highlighting the shift from fast, pattern-based responses ("System 1 thinking") to deliberate reasoning at inference time ("System 2 thinking")

10. Anthropic's CEO presents a hopeful vision for AI, predicting breakthroughs in health, economics, and governance if AI’s potential is harnessed correctly

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Claude, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Large Language Models

Open-source

1. Nvidia releases Llama-3.1-nemotron-70b - a language model that outperforms GPT-4o and Claude 3.5 Sonnet on instruction following benchmarks like AlpacaEval and MT-Bench, allowing commercial use 

2. Mistral releases Ministral 3B and 8B models for edge computing, pushing new limits in reasoning and function-calling within the sub-10B range, outperforming Llama 3 8B and Mistral 7B on instruction-following benchmarks 

3. Meta releases quantized Llama 3.2 models, delivering faster on-device AI processing with reduced size and memory use for mobile deployment

4. Meta releases an open-source replica of Google’s NotebookLM called NotebookLlama, offering an open-source framework using Llama models and text-to-speech tools to generate podcast-style audio from PDFs

Read more

This deep dive series will probably launch next week, so I'm fast-tracking the LinkedIn Highlights post.

Welcome to LinkedIn Highlights!

Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. Sparrow

A new open-source project called Sparrow simplifies the challenging task of extracting structured data from unstructured documents like forms, invoices, and images using machine learning and LLM pipelines.

Its modular and pluggable architecture lets you seamlessly integrate tools like LlamaIndex, Haystack, and Unstructured for customizable data processing workflows. Whether you're processing PDFs or extracting content from images, Sparrow provides independent agents for each task.

Sparrow's standout feature is its ability to let users build and deploy LLM agents through a simple API, making integration into your systems seamless and efficient. It even supports local LLM execution using Ollama or Apple MLX.

Key agents include:

• llamaindex - PDF processing with LlamaIndex

• vprocessor - OCR + LlamaIndex for image processing

• haystack - PDF processing with Haystack

• unstructured-light - PDF and image processing with Unstructured and LangChain

GitHub repo https://github.com/katanaml/sparrow

2. Windows Agent Arena

Are AI Agents coming to Windows? Microsoft just released an open-source project for developers to build autonomous agents for its Windows operating system.

As part of the release, Microsoft open-sourced Omniparser, the current top-performing screen understanding model in their benchmark.

A ready Windows OS environment ensures agents perform optimally in real-world conditions. Microsoft also integrated it with Azure ML so multiple agents can run in parallel and complete their tasks in minutes rather than days, thanks to cloud scaling.

Code https://github.com/microsoft/WindowsAgentArena

Become a premium member to get full access to my content and $1k+ in free credits for leading AI tools and APIs, including Claude, Hugging Face, Deepgram. It’s common to expense the paid membership from your company’s learning and development education stipend.

Support AI Tidbits as a premium member

3. ChatTTS

A new breakthrough in text-to-speech technology is here: ChatTTS.
Explicitly designed for dialogue-based scenarios like LLM assistants, ChatTTS pushes the boundaries of conversational AI with the ability to generate natural, expressive speech.

ChatTTS is optimized for multi-speaker dialogue tasks, making it ideal for AI assistants and interactive conversation models. The model also allows fine-grained prosody control, such as pauses, laughter, and interjections, significantly enhancing the expressiveness of synthesized speech.

Its ability to predict and replicate natural speech patterns surpasses many open-source TTS models.

ChatTTS was trained on 100,000+ hours of English and Chinese audio and is open-sourced for research use.

Repo https://github.com/2noise/ChatTTS
Example notebook https://github.com/2noise/ChatTTS/blob/main/examples/ipynb/example.ipynb

4. Table-Augmented Generation

Read more

Welcome to a new AI Tidbits series! Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

1. MindSearch

A new open-source search engine is rivaling top-tier AI products like Perplexity.ai Pro and ChatGPT-Web.

MindSearch is an innovative AI search engine framework that combines LLMs and a multi-agent system to tackle three critical issues that often limit LLM-powered search engines:

1. LLMs struggle to decompose complex queries into simpler, actionable requests

2. Search results often contain too much noise, making it hard to filter and extract relevant information

3. Iterative searches can quickly overload the LLM’s input length capacity
   

MindSearch utilizes two main components:

• WebPlanner - decomposes complex queries into sub-tasks and creates a dynamic graph structure for problem-solving

• WebSearcher - conducts fine-grained searches and delivers summarized information back to WebPlanner for further refinement
  

This approach allows MindSearch to handle massive web content (e.g., more than 300 pages) effectively, surpassing limitations faced by traditional LLM-based search systems.

According to subjective evaluations from human experts, MindSearch significantly outperforms major search engines like ChatGPT-Web and Perplexity.ai Pro. Its superior depth, breadth, and factual accuracy make it a breakthrough solution for both open-set and closed-set QA tasks.

Technical report https://arxiv.org/abs/2407.20183
Code https://github.com/InternLM/MindSearch

2. Advanced RAG techniques

A new GitHub repository provides the most comprehensive RAG tutorials you’ll find, showcasing advanced techniques to enhance the accuracy, efficiency, and contextual richness of RAG systems.

The repository offers easy-to-start notebooks covering methods like:

• Reliable RAG – refining and validating retrieved information for better accuracy

• Proposition Chunking – breaking down text into meaningful sentences for improved control over query handling

• Query Transformations – optimizing queries by rewriting and decomposing complex ones into sub-queries.

• Semantic Chunking – dividing documents based on semantic coherence for more meaningful retrieval.

GitHub repo https://github.com/NirDiamant/RAG_Techniques

Become a premium member to get full access to my content and $1k+ in free credits for leading AI tools and APIs, including Claude, Hugging Face, Deepgram. It’s common to expense the paid membership from your company’s learning and development education stipend.

Support AI Tidbits as a premium member

3. Zerox OCR

OCR just got simpler thanks to Zerox OCR, a dead simple open-source solution for extracting text from documents for AI ingestion.

Documents are visual by nature, filled with tricky layouts, tables, and charts, making vision models the perfect fit. Zerox uses GPT-4o Mini to turn visual documents into characters, à la OCR.

The process is straightforward:

1. Feed in a PDF

2. PDF is converted into a series of images

3. Each image is sent to GPT, which is tasked to convert it into markdown format

4. The response of each image is aggregated into a cohesive Markdown file
   

While it may sound basic, Zerox OCR with gpt-4o-mini is both cost-effective and delivers superior results compared to existing specialized solutions like AWS Textract, Google Document AI, and Azure Document AI.

Try it out https://github.com/getomni-ai/zerox

4. Anthropic’s metaprompt

Read more

September has been a busy month for everyone in the AI space. It has been packed with groundbreaking developments across various AI domains, from industry giants to open-source breakthroughs.

In the realm of large language models, we've seen significant strides from both industry leaders and open-source initiatives. OpenAI introduced its advanced o1-preview and o1-mini models, excelling in high-level reasoning for coding and math. Meanwhile, Alibaba released the impressive Qwen 2.5 family of open multilingual models, handling an expansive 128K tokens. Meta also made waves with Llama 3.2, featuring edge-optimized text models and their first large multimodal models.

Multimodal AI saw remarkable progress, with AI2's Molmo models rivaling and surpassing industry giants like GPT-4V and Gemini 1.5. Nvidia's NVLM 1.0 and Apple's MM1.5 further pushed the boundaries of vision-language reasoning and diverse task performance.

In the audio domain, OpenAI released Whisper Large v3 Turbo, a faster and more capable speech-to-text model, while Google developed a promising zero-shot Voice Transfer module for cross-lingual applications.

The image and video generation landscape continued to evolve, with Meta's Imagine Yourself technology enabling personalized image generation and advancements in text-to-video models like CogVideoX-5B pushing the boundaries of visual content creation.

This month's roundup features these breakthroughs and many more exciting updates across AI tools, research methodologies, and vision AI.

Let's dive in!

Overview

• Industry announcements (11 entries)

• Large Language Models

  • Open-source (11 entries)

  • Research (9 entries)

• Multimodal (7 entries)

• Autonomous Agents (3 entries)

• Image and Video (8 entries)

• Audio (4 entries)

• AI Tools (5 entries)

• Open-source Packages (5 entries)

Recent Deep Dives

Industry announcements

1. OpenAI introduces o1-preview and o1-mini, advanced models that excel in high-level reasoning for coding and math, with o1-mini being a faster, cost-efficient option 

2. As part of its DevDay event, OpenAI releases the Realtime API, allowing developers to create low-latency, voice-to-voice interactions with continuous audio streaming, along with new tools like vision fine-tuning, prompt caching, and model distillation

3. OpenAI releases Advanced Voice Mode for ChatGPT Plus and Team users, adding Custom Instructions, Memory, and five new voices

4. Anthropic releases the Quickstarts repo, providing ready-to-deploy app projects powered by its API, starting with a Claude-based customer support agent

5. In its annual Connect conference, Meta announced multiple AI-related releases, including Ray-Ban Meta smart glasses with real-time AI video processing, AI-powered visual search for Instagram, translation and dubbing tools for creator content with lip sync, and Meta AI vocal responses across platforms with customizable celebrity voices

6. OpenAI's CTO, Mira Murati, along with other OpenAI execs to step down and leave the company

7. Pika Labs launches its new video generating model, featuring new "Pikaffects" that transform video subjects with surreal, physics-defying effects like melting and cake-ifying objects 

8. Google introduces new Gemini-1.5 Pro and Flash models, with faster response speeds, reduced prices (>50%) and improved task performance

9. Deepgram unveils its Voice Agent API, enabling natural, real-time human-machine conversations powered by high-performance speech recognition and synthesis models

10. Hume releases Empathic Voice Interface 2 (EVI 2) - a GPT-4o-like voice model, allowing users to converse with its AI chatbot with sub-second response times

11. Replit announces Replit Agent - an AI tool that automates software development tasks like environment setup and deployment

👆 OpenAI’s Realtime API powering Speak’s language learning app

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Claude, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Large Language Models

Open-source

1. Alibaba releases Qwen 2.5 - a family of open multilingual models handling 128K tokens and outperforming competitors like Mistral 2 (123B) on major benchmarks, offering multilingual support in 29 languages and specialized models across Math and coding

2. Meta releases Llama 3.2 - a new family of open models featuring edge-optimized text models (1B and 3B) and Meta's first large multimodal models (11B and 90B) supporting 128K tokens

3. Kyutai Labs openly releases Moshi - a 7.6B open speech-to-speech model with cutting-edge performance and low latency, alongside Mimi, a SoTA streaming audio codec that compresses 24 kHz audio to 1.1 kbps for optimized real-time speech communication

4. The DeepSeek team releases DeepSeek-V2.5 - a SOTA versatile open model integrating DeepSeek-Coder with advanced features like Function Calling and JSON output

5. 01.AI unveils Yi-Coder - a high-performing series of code LLMs with up to 9B parameters, excelling in long-context modeling and outperforming other bigger models

6. Alibaba proposes DocOwl2 - a state-of-the-art model for multi-page document understanding that reduces GPU usage and inference time by compressing high-resolution document images into 324 tokens

Read more

Welcome to a new AI Tidbits series! Each month, I'll share my five top-performing LinkedIn posts, bringing you the best of AI straight from the frontlines of academia and industry.

As a frequent LinkedIn contributor, I regularly share insights on groundbreaking papers, promising open-source packages, and significant AI product launches. These posts offer more depth and detail than our weekly snippets, providing a comprehensive look at the latest AI developments.

Whether you're not on LinkedIn or simply missed a post, this monthly roundup ensures you stay informed about the most impactful AI news and innovations.

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Claude, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

1. PDF Extract Kit

Extracting information from documents has been one of AI’s holy grails. A new open-source project deploys specialized AI models to tackle this challenge head-on.

PDF-Extract-Kit is a comprehensive pipeline that breaks down PDF content extraction into several components:

1. Layout detection - leveraging LayoutLMv3 to precisely identify regions like images, tables, titles, and text

2. Table recognition - featuring StructEqTable for converting complex tables into LaTeX

3. OCR - utilizing PaddleOCR for high-performance text extraction in multiple languages

4. Formula detection - using YOLOv8 to accurately detect inline and isolated formulas

5. Formula recognition - employing UniMERNet to rival commercial software in formula recognition quality

Trained on diverse datasets, these models handle various document types, from academic papers to financial reports.

GitHub repo https://github.com/opendatalab/PDF-Extract-Kit

2. Mistral fine-tuning API + SDK

Mistral just dropped a game-changing fine-tuning API and SDK to help developers easily fine-tune Mistral variants on a single GPU. Clone, prep, train.

The SDK is a lightweight GitHub repository that leverages LoRA, allowing for memory-efficient training by freezing most model weights and only updating 1-2% with low-rank matrix perturbations. It's optimized for multi-GPU setups but can also be used with a single GPU for smaller models like the 7B.

To get started:

1. Clone the repo and install dependencies

2. Download and prepare your model and data

3. Validate and start training with a few simple commands

This repository is opinionated to simplify the finetuning process, focusing on Mistral models and specific hardware. It also includes a Colab notebook to hit the ground running.

Full details and setup instructions are in the GitHub repo https://github.com/mistralai/mistral-finetune

3. Perplexica

A new open-source project called Perplexica replicates the $2.5B startup Perplexity so developers can easily launch AI-powered search tools.

Perplexica is an open-source AI-powered search tool that dives deep into the internet to find precise answers. Perplexica not only searches the web but also understands your questions, delivering clear answers with cited sources.

It supports local models such as Llama3 and Mixtral for faster and cheaper inference and has six specialized modes tailored to answer specific types of questions:

1. All Mode - searches the entire web for the best results

2. Writing Assistant Mode - assists with writing tasks without web searches

3. Academic Search Mode - ideal for finding articles and papers for academic research

4. YouTube Search Mode - finds YouTube videos based on search queries

5. Wolfram Alpha Search Mode - uses Wolfram Alpha for calculations and data analysis

6. Reddit Search Mode - searches Reddit for discussions and opinions
   

Unlike other tools that use outdated data, Perplexica provides the latest information using a metasearch engine called SearxNG.

GitHub repo https://github.com/ItzCrazyKns/Perplexica

4. HippoRAG

Read more

August has been a month of remarkable progress across various AI domains, from industry giants to open-source breakthroughs.

Elon Musk's xAI made waves with the unveiling of Grok-2 and Grok-2 mini, showcasing advanced capabilities that rival top models. OpenAI continued to refine its offerings with a more efficient GPT-4o and the introduction of Structured Outputs. The open-source community saw significant advancements, with Microsoft's Phi 3.5 series and AI21's Jamba models pushing the boundaries of what's possible with freely available models.

In the realm of multimodal AI, Nvidia's Eagle and Alibaba's Qwen2-VL demonstrated impressive performance in visual understanding tasks. The image and video generation field saw major leaps with Black Forest Labs' FLUX.1 and Tsinghua University's CogVideoX-5B. Audio AI also made strides, with Qwen2-Audio enabling multilingual voice interaction and HuggingFace's Parler TTS v1 offering enhanced text-to-speech capabilities.

Perhaps most intriguingly, Sakana AI introduced The AI Scientist, a system that could revolutionize scientific research by automating idea generation, execution, and documentation.

These developments, along with many more exciting updates across language models, multimodal AI, and specialized applications, are part of this month's comprehensive roundup.

Let's dive in!

Overview

• Industry announcements (8 entries)

• Large Language Models

  • Open-source (8 entries)

  • Research (8 entries)

• Multimodal (11 entries)

• Image and Video (10 entries)

• Audio (6 entries)

• Robotics (2 entries)

• Open-source Packages (8 entries)

Recent Deep Dives

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Claude, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Industry announcements

1. Elon Musk's xAI unveils Grok-2 and Grok-2 mini, showcasing advanced chat, coding, and reasoning capabilities that outperform top models like Claude 3.5 Sonnet and GPT-4 Turbo

2. OpenAI releases a new GPT-4o version that is slightly better and 50% cheaper than the previous GPT-4o model

3. OpenAI releases Structured Outputs, ensuring AI-generated data conforms precisely to developer-supplied JSON schemas for enhanced reliability, achieving perfect accuracy with the new gpt-4o-2024-08-06 model

4. Anthropic introduces prompt caching on its API, reducing costs and latency for large prompts by up to 90% and 85%, respectively, now in public beta for Claude 3.5 Sonnet and Claude 3 Haiku

5. Google releases Gemini Live - a voice-interactive AI chatbot with enhanced emotional expression and real-time adaptive dialogue, similar to ChatGPT's new Advanced Voice Mode capability, offering hands-free and long-context conversational capabilities

6. DeepMind releases Imagen 3 - a latent diffusion model that outperforms state-of-the-art models in generating high-quality images from text prompts, with built-in measures to enhance safety and representation

7. OpenAI introduces fine-tuning for GPT-4o, allowing developers to tailor model responses and improve performance on domain-specific tasks like software engineering and text-to-SQL 

8. Ideogram introduces Ideogram 2.0 - a new version of its text-to-image model, outperforming DALL-E, Midjourney, and FLUX Pro with improved text accuracy and an API for developers

   

Large Language Models

Open-source

1. Microsoft releases three new open-source AI models in the Phi 3.5 series: Phi 3.5 mini-instruct, MoE-instruct, and vision-instruct models, offering scalable reasoning capabilities for commercial and scientific use across languages

2. AI2 introduces OLMoE - a sparse Mixture-of-Experts language model that activates only 1B parameters per token, achieving state-of-the-art performance and outperforming larger models like Llama2-13B-Chat

3. AI21 releases Jamba Large and Jamba Mini - two new language models in its family of Mamba-Transformer models, featuring the longest context window for open models (256k) and rivaling state-of-the-art models like Llama 3.1 and Mistral Large

4. Researchers open source OpenResearcher - an AI-driven platform that integrates LLMs with domain-specific knowledge through Retrieval-Augmented Generation, enabling researchers to efficiently navigate and generate insights from scientific literature

Read more

July was filled with remarkable progress across various AI domains, from LLMs to text-to-video models. But most of all, July was the month of open-source AI, with Meta's Llama 3.1 suite and Google's Gemma-2-2B pushing the boundaries of what's possible with freely available models, rivaling top proprietary LLM providers such as GPT-4o and Claude Sonnet.

OpenAI made waves with the release of GPT-4o mini, a cost-effective multimodal model, and the unveiling of SearchGPT, a potential Google Search competitor. Meanwhile, Mistral AI flexed its muscles with Mistral Large 2 and partnered with Mamba creators to release the impressive Codestral-Mamba 7B.

In the realm of research, DeepMind's AlphaProof and AlphaGeometry 2 showcased AI's growing prowess in advanced mathematics. The emergence of autonomous agents continued with OpenDevin, while image and video generation saw leaps forward with Meta's Segment Anything Model (SAM) 2 and Stability AI's Stable Video 4D.

These developments, along with many more exciting updates across language models, multimodal AI, and specialized applications, are part of this month's comprehensive roundup.

Let's dive in!

Overview

• Industry announcements (6 entries)

• Large Language Models

  • Open-source (14 entries)

  • Research (8 entries)

• Autonomous Agents (4 entries)

• Multimodal (3 entries)

• Image and Video (10 entries)

• Audio (2 entries)

• AI Tools (5 entries)

• Open-source Packages (7 entries)

Recent Deep Dives

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Perplexity, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Industry announcements

1. OpenAI releases GPT-4o mini - a cost-effective multimodal model, offering superior performance at 60% less cost than GPT-3.5 Turbo

2. Mistral releases Mistral Large 2, boasting a 128k context window and superior performance in multiple languages and coding tasks

3. OpenAI unveils SearchGPT - a Google Search competitor combining real-time web information with conversational abilities for precise and fast answers

4. OpenAI begins alpha testing Advanced Voice Mode - ChatGPT's new real-time and natural conversational AI demoed in OpenAI's April Spring Updates event

5. Anthropic releases a revamped developer console to help LLM users and developers optimize and evaluate prompts, including a side-by-side comparison of outputs

6. Odyssey, a new startup emerging from stealth, presents Hollywood-grade visual AI, offering filmmakers and artists detailed control over geometry, materials, lighting, and motion for high-end productions

   

Large Language Models

Open-source

1. Meta unveils Llama 3.1 - an open suite of three language models (7B, 80B, 405B), with the largest one outperforming GPT-4o and Claude Sonnet on multiple benchmarks

2. Google releases Gemma-2-2B - a commercially permissive compact 2 billion parameter model that outperforms models 20x its size, including Mixtral 8x7B, GPT 3.5, and Llama-2 70B

3. Mistral partners with the Mamba creators to release the commercially permissive Codestral-Mamba 7B - the strongest code model for its size, featuring a 256k context window

4. Mistral releases Mathstral - an open model excelling in multi-step logical reasoning for STEM subjects, achieving state-of-the-art scores on key benchmarks 

5. Mistral AI releases Mistral NeMo - a powerful 12B open language model developed with Nvidia, featuring a 128k context window and outperforming competitors in reasoning and coding tasks

Read more

As we step into summer, the AI landscape continues moving with groundbreaking innovations and exciting developments. June has been a month of significant strides across various AI domains, from industry giants to open-source breakthroughs.

Apple's Worldwide Developers Conference (WWDC) took center stage, unveiling a suite of AI features that promise to revolutionize the user experience across Apple devices. Meanwhile, the language model arena saw remarkable advancements, with Anthropic's Claude 3.5 Sonnet pushing the boundaries of performance, a new software agent that scored 19% on SWE-bench, and a new state-of-the-art version of DeepSeek-Coder.

In video generation, companies like Runway and Luma AI challenge the status quo with their hyperrealistic video creation tools.

This month's roundup also spotlights impressive progress in multimodal AI, with Microsoft openly releasing Florence-2, a commercially permissive state-of-the-art small vision model family, and EPFL and Apple's new training approach setting new benchmarks for multimodal AI.

In addition to these highlights, June’s roundup features novel LLM techniques (e.g. Mixture-of-Agents), promising open-source projects (e.g. Open Interpreter), and a host of other developments in autonomous agents and multimodal AI.

Let's dive in!

Overview

• ✨ Special feature: Apple Worldwide Developers Conference (WWDC)

• Industry announcements (9 entries)

• Large Language Models

  • Open-source (10 entries)

  • Research (10 entries)

• Autonomous Agents (4 entries)

• Multimodal (4 entries)

• Image and Video (8 entries)

• Audio (3 entries)

• Open-source Packages (6 entries)

Recent Deep Dives

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Perplexity, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

✨Special feature: Apple Worldwide Developers Conference (WWDC)

• Apple Intelligence - a suite of new AI features for iPhone, Mac, and other Apple devices. This includes a more conversational Siri, custom AI-generated "Genmoji," and integration with OpenAI's GPT-4o. Apple’s on-device models excel in specific tasks using an adapter strategy, with on-device models outperforming larger models in summarizing and composing text.

• Enhanced Siri capabilities - Siri will gain new abilities such as managing notifications, writing and summarizing text, and carrying out actions across multiple apps. Users can interact with Siri through voice or typing.

• Genmoji and Image Playground - Apple is launching Genmoji to create emoji-like reactions on demand and Image Playground for AI-generated images. These features will be integrated into various apps, including Photos, which will have improved search and editing capabilities similar to Google's Magic Eraser.

• OpenAI integration - Siri will leverage ChatGPT, powered by GPT-4o, for complex requests, ensuring user permission before sending data. ChatGPT will be available across iOS, macOS, and iPadOS, supporting AI writing and image generation tools.

—> Apple WWDC 2024 keynote in 18 minutes

Industry announcements

1. Anthropic releases Claude 3.5 Sonnet - a model with a 200k token context window, outperforming GPT-4o and featuring a new dynamic Artifacts workspace

2. A new startup called Etched is trying to take on Nvidia by presenting the world’s first specialized chip for Transformers, delivering over 500,000 tokens per second and claims to be >10x faster and cheaper than NVIDIA’s next-generation Blackwell

3. Factory emerges out of stealth to automate software engineering by modeling the cognitive processes of developers, achieving top performance on SWE-bench (19.27% compared to Devin's 13.86%)

4. Ilya Sutskever, OpenAI's former chief scientist, starts Safe Superintelligence Inc. - a new company to build safe superintelligence

5. Runway releases Gen-3 Alpha - an AI model generating high-quality, hyperrealistic videos with expressive human characters and smooth transitions 

6. A Chinese company launches a Sora competitor called Kling - an AI model that generates realistic video clips from text prompts using advanced 3D AI techniques

7. Luma AI debuts Dream Machine - an OpenAI Sora-like tool enabling users to create realistic videos from text prompts in just two minutes

8. ElevenLabs releases Sound Effects, turning text into rich sounds

9. Mistral introduces model customization on its platform, enabling efficient fine-tuning of AI models to meet specific user needs with reduced costs and expertise

Large Language Models

Open-source

1. Microsoft releases Florence-2 - a commercially permissive state-of-the-art small vision model family (200M, 800M params) that outperforms larger specialized models in tasks like image description and object recognition

2. DeepSeek-AI releases DeepSeek-Coder-V2 - an open-source language model supporting 338 programming languages, beating top commercial models like GPT-4 Turbo in code generation and mathematics

3. Open Interpreter introduces Local III - a suite of tools for running powerful language models locally to control your personal computer, enhancing control and privacy

Read more

Welcome to the May edition of AI Tidbits Monthly, where we unravel the latest and greatest in AI. This month has been particularly eventful, with major updates from industry leaders such as Google, OpenAI, and Microsoft.

In its Spring Updates event, OpenAI introduced GPT-4o, a multimodal model processing text, vision, and audio with real-time emotion recognition and adaptive speech responses. They expanded the free tier to include ChatGPT Plus features and announced a new Mac app, with a Windows version coming soon.

Google I/O was also filled with announcements, including a Gemini 1.5 Pro with featuring a 2M-token context window and Gemini 1.5 Flash, optimized for speed and cost-efficiency. Google unveiled Project Astra, a real-time multimodal AI assistant, and Veo, a long-form video generator, and announced the Trillium chip (TPU v6) for AI datacenters.

Lastly, Microsoft held its annual developers conference, Microsoft Build, introducing Copilot+ PCs, AI-optimized devices with advanced silicon and all-day battery life. They launched agents as part of Copilot to complete tasks autonomously, unveiled Phi-3 Vision and Phi Silica models, and announced AI-powered real-time video translation for the Edge browser.

In addition to these highlights, May's roundup includes groundbreaking advancements in language models (a new coding model from Mistral!), research, and open-source projects.

Let's dive in!

Overview

• ✨ Special feature: AI updates from Google, OpenAI, and Microsoft

• Large Language Models

  • Open-source (10 entries)

  • Research (8 entries)

• Autonomous Agents (2 entries)

• Multimodal (5 entries)

• Image and Video (5 entries)

• Audio (4 entries)

• Open-source Packages (6 entries)

Recent Deep Dives

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Perplexity, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

✨ Special feature: AI updates from Google, OpenAI, and Microsoft

OpenAI Spring Updates

OpenAI introduced GPT-4o, a cutting-edge multimodal model that processes text, vision, and audio, offering superior speed and cost-efficiency compared to GPT-4 Turbo. Key enhancements include real-time emotion recognition and adaptive speech responses, inspired by the movie "Her." The new voice assistant features real-time translation, facial expression reading, and dynamic voice adaptation, significantly improving interactivity. OpenAI expanded its free tier, providing features previously exclusive to ChatGPT Plus users, and limited access to GPT-4o. Additionally, a new desktop app for Mac was announced, with a Windows version coming soon, and potential integration with Apple devices is on the horizon.

—> More here

Google I/O

At Google I/O 2024, Google unveiled Gemini 1.5 Pro, boasting an expanded context window of up to two million tokens, and Gemini 1.5 Flash, optimized for speed and cost-efficiency. New projects include Project Astra, a real-time multimodal AI assistant, and Veo, a long-form video generator. AI capabilities are being integrated across Google's ecosystem, enhancing Search, Gmail, Google Photos, and Android. The Trillium chip (TPU v6) was introduced, designed for AI datacenters to enhance processing power and energy efficiency, along with Gemini Nano, bringing on-device multimodal capabilities to Pixel devices.

—> More here

Microsoft Build

Microsoft introduced Copilot+ PCs, a new category of AI-optimized Windows devices with advanced silicon and all-day battery life. The company expanded Copilot AI agents to handle autonomous tasks, with new capabilities launching in Copilot Studio. Phi-3 Vision, a compact multimodal AI model, and Phi Silica, a local language model optimized for Copilot+ PCs, were unveiled. Additionally, the Edge browser will soon feature AI-powered real-time video translation, enabling multilingual video accessibility and enhancing global communication.

—> More here

Large Language Models (LLMs)

Open-source

1. Microsoft open sources new Phi-3 models, including a 7B, 14b, and a new multimodal variant with vision capabilities featuring a 128k context window

2. Mistral releases Codestral - a 32k context window coding model supporting over 80 programming languages and setting a new standard in performance and latency

3. Mistral releases 7B v0.3 - a new version of its small and powerful open language model, with function calling support 

4. Google releases Gemma 2 - featuring a new, efficient architecture that offers class-leading performance with fewer parameters and lower deployment costs, optimized for diverse hardware

5. Google releases PaliGemma - a state-of-the-art open vision-language model that can perform deeper analysis of images and provide useful insights, such as captioning for images and short videos, object detection, and reading text embedded within images

6. Researchers open-source Prometheus 2 - a groundbreaking, cost-effective language model evaluator, achieving unmatched performance in aligning with human assessments

7. Cohere open sources Aya 23 - a new family of multilingual LLMs (8B, 35B) that supports 23 languages and outperforms existing models in various linguistic tasks

8. IBM open sources for commercial use the Granite Code models - a series of code LLMs optimized for a wide range of coding tasks, from code generation to repository maintenance, achieving state-of-the-art performance in software development tasks

9. Researchers unveil AutoCoder - the first coding model to outperform GPT-4 Turbo and GPT-4o on the Human Eval benchmark, achieving a pass@1 score of 90.9% and featuring an enhanced code interpreter with the capability to install external packages

10. CMU showcases MAmmoTH2 - a novel LLM that significantly boosts reasoning performance by using web-extracted instruction data, setting new standards in efficiency and effectiveness

    

Research

1. Anthropic releases a detailed study peeking into LLMs' brains, showcasing how millions of concepts such as gender, conversational styles, and political views are represented, revealing the first-ever detailed look inside a modern LLM

2. Anthropic releases a new tool that turns task descriptions into production-ready optimized prompts

Read more

Welcome to the March edition of AI Tidbits Monthly, where we uncover the latest and greatest in AI. This month has been filled with groundbreaking announcements from industry leaders and exciting progress in open-source AI, showcasing the rapid advancements in the field.

March saw the release of Cognition's Devin AI, the world's first autonomous AI software engineer, and Anthropic's Claude 3, setting new industry benchmarks across various domains. Mistral AI also introduced Mistral Large, a top-tier model rivaling GPT-4, now available on Azure through a new partnership with Microsoft.

In the realm of LLM security, research from UIUC, Cornell, DeepMind, and ETH Zurich highlighted potential cybersecurity risks and the urgent need for improved security measures against adversarial attacks.

Open-source initiatives continued to thrive, with AI21's Jamba, Sakana AI's Evolutionary Model Merge, xAI's Grok-1, and Apple's MM-1, showcasing novel approaches and achieving state-of-the-art results with enhanced efficiency.

Image and video generation also saw significant advancements, with Stability AI's Stable Diffusion 3, Alibaba's EMO framework for life-like portrait animation, and the introduction of YOLOv9 and GELAN for improved object detection.

Lastly, AI agents took center stage with DeepMind's Genie, transforming images into interactive 2D worlds, and SIMA, a versatile AI capable of following natural-language instructions across different video game environments.

These and many more exciting updates across various AI domains are part of this month's roundup.

Let's dive in!

Overview

• Industry Announcements (6 entries)

• ✨ Special feature: LLMs Security and Safety (4 entries)

• Large Language Models

  • Open-source (11 entries)

  • Research (6 entries)

• Autonomous Agents (4 entries)

• Image and Video (12 entries)

• Audio (2 entries)

• Multimodal (3 entries)

• Open-source Packages (7 entries)

• AI tools (2 entries)

Recent Deep Dives

Industry announcements

1. Cognition releases Devin AI - the world's first autonomous AI software engineer, excelling in complex tasks and learning from feedback, outperforming in real-world coding benchmarks

2. Anthropic announces Claude 3 - three state-of-the-art language models, setting new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision

3. Mistral AI releases Mistral Large - a top-tier model that rivals GPT-4 with advanced multilingual reasoning and competitive pricing, now available on Azure as part of a new partnership with Microsoft

4. Ideogram releases Ideogram 1.0 - a text-to-image model excelling in text rendering and photorealism

5. Figure partners with OpenAI to enhance its humanoid robot, Figure 01, showcasing human-like communication and reasoning in an unprecedented demo

6. Nvidia unveils Blackwell at GTC 2024 - its next generation and the world's most powerful AI superchip

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Perplexity, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

✨ Special feature: LLMs Security and Safety

1. UIUC shows that GPT-4 can autonomously hack websites, performing advanced tasks like SQL injections and finding vulnerabilities, highlighting potential cybersecurity risks

2. DeepMind and ETH Zurich introduce a novel attack capable of extracting detailed information from black-box language models, determining the exact hidden dimensions of notable models like OpenAI's ChatGPT for under $20

3. Cornell unveils Morris II - a computer worm targeting GenAI systems, demonstrating the urgent need for improved security against adversarial self-replicating prompts

4. Scale and the Center for AI Safety release the WMDP benchmark - a safety evaluation for LLMs to gauge their knowledge in biosecurity, chemical security, and cybersecurity

   

Large Language Models (LLMs)

Open-source

1. AI21 open-sources Jamba - a pioneering Mamba SSM-Transformer model, enhancing AI performance with a 256K context window and tripled throughput on long contexts

Read more

Welcome to the February edition of AI Tidbits Monthly, where we unravel the latest and greatest in AI. February continued January’s strong momentum for commercial and open-source AI across modalities.

On the commercial LLMs front, Google released Gemini 1.5, supporting a groundbreaking context window of 10M tokens. Anthropic released Claude 3, a suite of powerful language models with image understanding capabilities and performance that outperform GPT-4. Mistral launched its largest and most powerful model to date, Mistral Large.

Open-source language models experienced a step change in performance, with Google’s Gemma, Abacus’ Smaug, and Qwen 1.5—all demonstrating GPT-3.5-level performance with a commercially permissive license.

Nonetheless, February’s biggest announcement was OpenAI’s new text-to-video model, Sora, which produces Hollywood-grade one-minute videos. Alibaba unveiled a remarkable new framework designed to bring portraits to life with incredibly realistic expressions and accurate lip-syncing. Lastly, Google released a pioneering tool that turns any image into an interactive 2D game.

These breakthroughs, along with many more across speech, video, multimodal AI, and autonomous agents, are featured in this month’s roundup.

Let's dive in!

Overview

• Industry announcements (7 entries)

• ✨ Special feature: Speech recognition and text-to-speech AI (5 entries)

• Large Language Models

  • Open-source (15 entries)

  • Research (9 entries)

• Autonomous Agents (5 entries)

• Image and Video (13 entries)

• Audio (3 entries)

• Multimodal (5 entries)

• Robotics (4 entries)

• Open-source packages (5 entries)

• AI tools (5 entries)

Recent Deep Dives

Industry announcements

1. OpenAI unveils Sora - a groundbreaking text-to-video model that creates realistic videos up to a minute long from text prompts

2. Anthropic announces Claude 3 - three state-of-the-art language models, setting new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision

3. Google releases Gemini 1.5, featuring a groundbreaking 10M context window for superior performance across multiple modalities with reduced compute

4. Mistral AI releases Mistral Large - a top-tier model that rivals GPT-4 with advanced multilingual reasoning and competitive pricing, now available on Azure as part of a new partnership with Microsoft

5. Stability AI unveils Stable Diffusion 3, featuring enhanced multi-subject and advanced text prompt handling

6. Stability AI launches SVD 1.1 - a text-to-video model optimized for better motion and consistency

7. Ideogram releases Ideogram 1.0 - a text-to-image model excelling in text rendering and photorealism

   

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs like Perplexity, Replicate, and Hugging Face. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

✨ Special feature: Speech recognition and text-to-speech AI

1. MetaVoice open sources a commercially permissive 1B base model for text-to-speech, supporting voice cloning and emotional speech synthesis

2. Amazon unveils Base TTS - the largest text-to-speech model trained on 100K hours of speech, achieving unprecedented naturalness in speech synthesis with novel tokenization

3. Nvidia presents Audio Flamingo - a novel audio language model that improves LLMs' abilities to understand audio

4. Nvidia releases Canary 1 - a state-of-the-art automatic speech recognition and speech translation model leading the Open ASR Leaderboard across four languages

5. Nvidia introduces Parakeet-TDT - revolutionizing speech recognition with unparalleled accuracy and 64% faster processing speed compared to previous models

Large Language Models (LLMs)

Open-source

1. Google open sources Gemma - a suite of small language models (7B, 12B) that outperforms Llama 2 and Mistral 7B, permitting commercial use

Read more

Welcome to the first monthly edition of AI Tidbits Monthly, where we unravel the latest and greatest in AI. January has kicked off 2024 with a strong start and breakthroughs across generative AI modalities, from text to video and audio.

More significantly, January was the month of image and video generation. Google unveiled Lumiere, its new text-to-video generative model, TikTok open-sourced a new state-of-the-art depth estimation model, and an image synthesis tool called InstantID took the internet by storm.

We also saw substantial progress in multimodal AI with the release of LLaVA 1.6 and Qwen-VL-Max, two commercially permissive models capable of understanding images and reading documents.

On the open-source front, Microsoft allowed the commercial use of its powerful and small Phi-2 model, Meta open-sourced the 70B version of its powerful coding language model Code Llama, and a non-transformer model rivaled large transformer-based SOTA models while being cheaper and faster.

This roundup includes these and many other exciting updates across generative audio, autonomous agents, useful AI-powered tools, and open-source repositories are part of this month’s roundup.

Let's dive in!

Overview

• Industry announcements (6 entries)

• Large Language Models

  • Open-source (10 entries)

  • Research (16 entries)

• Autonomous Agents (2 entries)

• Image and Video (12 entries)

• Audio (4 entries)

• Multimodal (5 entries)

• Open-source packages (5 entries)

• AI tools (7 entries)

Recent Deep Dives

Industry announcements

1. OpenAI announces new embedding models with superior performance and affordability, alongside reduced pricing for GPT-3.5 Turbo and an improved GPT-4 Turbo preview addressing response "laziness"

2. OpenAI launches its GPT Store, its platform for sharing and monetizing custom GPTs

3. Rabbit introduces R1 - a $199 standalone AI device featuring voice control and a Large Action Model for universal application control

4. Microsoft launches a Pro plan for its Copilot chatbot, infusing its Microsoft 365 suite of apps (Outlook, Word, Excel, etc.) with ChatGPT-like capabilities

5. Meta's Mark Zuckerberg announces the development of AGI and Llama 3, integrating FAIR and GenAI teams, and a significant GPU expansion to advance AI capabilities

6. Stability AI introduces Stable LM 2 1.6B - an efficient multilingual language model, setting a new benchmark in small-scale LMs

Large Language Models (LLMs)

Open-source

1. Microsoft changes the license for its small open-source powerful language model phi-2, allowing commercial use

2. Meta open-sources Code Llama 70B - a coding language model outperforming GPT-4 in complex coding tasks, available for commercial use

3. Alibaba open sources Qwen-VL-Max - a large vision language model outperforming all previous open-source models and performing on par with Gemini Ultra and GPT-4V

4. Abacus AI releases SMAUG, the world-leading 30B open-source LLM, with a top MMLU score of 76%, inching towards GPT-4 performance

5. RWKV releases Eagle 7B, a non-transformer multilingual language model rivaling larger models in performance while being faster and cheaper, available for open commercial use 

6. Moondream1 - a tiny 1.6B parameter vision language model that punches above its weight

7. Researchers open source TinyLlama - a 1.1B parameter language model, showcasing superior performance over similar-sized models

8. Researchers release Orion-14B - a collection of state-of-the-art multilingual LLMs achieving superior performance in diverse tasks

9. Researchers release Medusa-2 - combining parallel token predictions and tree-based attention to achieve up to 3.6x faster inference without compromising quality or accuracy

10. A new leaderboard on Hugging Face to track and evaluate hallucinations in open-source language models

    

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Research

1. Mistral published the Mixtral MoE 8x7B paper, shedding light on its architecture and training process

2. Anthropic uncovers that LLMs can learn and retain deceptive behaviors, with standard safety techniques proving ineffective

3. University of Maryland introduces Binoculars - a 90% accurate detection method for distinguishing LLM-generated text, outperforming existing tools with minimal false positives

Read more

Welcome to the December edition of AI Tidbits Monthly, where we unravel the latest and greatest in AI. December provided an exciting finale to a year filled with innovative breakthroughs and groundbreaking research.

This December, Google debuted its long-awaited large multimodal AI, Gemini, incorporating it into its Bard chatbot and providing API access. Apple also made its first major strides into generative AI with its efficient on-device inference framework, an open-source multimodal model named Ferret, and a new robust Apple silicon framework for enhanced ML efficiency.

On the open-source front, Mistral released a fully open-source Mixture of Experts model that outperforms GPT-3.5 and Llama 70B. Deci released a state-of-the-art 7B base model, and Microsoft introduced a coding LLM, CodeOcean, that beats the current SOTA open and closed LLMs on coding tasks.

Also on the open-source front, though for speech understanding and generation, Nvidia released Parakeet, a speech-to-text model that outperforms Whisper v3. Additional noteworthy developments include the unveiling of OpenVoice, a novel voice cloning technology, and Amphion, an extensive toolkit dedicated to generating audio, music, and speech.

These and many more exciting updates across novel promoting frameworks, autonomous agents, multimodal AI, and open-source repositories are part of this month’s roundup.

Let's dive in!

Overview

• Industry announcements (6 entries)

• Large Language Models

  • Open-source (9 entries)

  • Prompting techniques (3 entries)

  • Research (8 entries)

• Autonomous Agents (3 entries)

• Image and Video (8 entries)

• Audio (5 entries)

• Multimodal (5 entries)

• Open-source packages (4 entries)

Recent Deep Dives

Industry announcements

1. Google debuts Gemini - its large multimodal AI coming in three sizes: Nano, optimized for mobile devices and offering offline functionality; Pro, now powering Google's Bard and designed for a wide range of AI services; and Ultra, the most powerful version, targeting data centers and enterprise applications, set to release in Q1 ’24

2. With €385M in new funding, Mistral AI debuts a new platform featuring its Mixtral MoE model, becoming a direct competitor of OpenAI, Google, and Anthropic

3. DeepMind releases Imagen 2 - its next-generation text-to-image model that competes with DALL-E 3 and Midjourney

4. Meta launches a standalone AI-powered image generator

5. Midjourney breaks out of Discord and launches an alpha web-based image generation platform, offering an enhanced interface

6. The New York Times files a billion-dollar lawsuit against OpenAI and Microsoft, accusing them of copyright infringement for using its articles in training ChatGPT and Copilot

   

Large Language Models (LLMs)

Open-source

1. Mistral AI releases a commercially permissive Mixture of Experts model, featuring eight experts with 7B parameters each, outperforming models like GPT-3.5 and Llama 2 70B 

2. Microsoft publishes CodeOcean and WaveCoder, outperforming existing models in code-related tasks by generating and leveraging high-quality instruction data

3. Microsoft open sources E5 mistral-7b - a groundbreaking text embedding approach, leveraging synthetic data from LLMs and efficient training to achieve top performance on major benchmarks

4. Alibaba releases Qwen-72B, a 32K context window LLM, and Qwen-1.8B, an efficient AI model requiring only 3GB GPU memory

5. TII open-sources its family of Falcon LLMs, led by Falcon-180B, and publishes a paper outlining detailed evaluations and training methods used to develop Falcon

6. DeciAI open sources DeciLM 7B - the fastest and most cost-effective 7B pretrained model, topping the OpenLLM Leaderboard

7. Korean AI startup Upstage introduces SOLAR 10.7B, leveraging Depth Up-Scaling to surpass larger models in natural language tasks and achieve top rankings in the HF Open Leaderboard without the complexities of MoE scaling

8. Meta releases Purple Llama - a project encompassing CyberSec Eval and Llama Guard to enhance generative AI safety and trust

9. Apple open-sources MLX - a robust framework for Apple silicon featuring user-friendly APIs and advanced computational features for enhanced ML efficiency 

   

Become a premium member to get full access to my content and $1k in free credits for leading AI tools and APIs. It’s common to expense the paid membership from your company’s learning and development education stipend.

Upgrade to Premium

Prompting techniques

1. OpenAI releases a prompt engineering guide to improve LLM performance

Read more

This is the monthly curated round-up, so if you're pressed for time and can only catch one AI Tidbits edition, this is the one to read—featuring the absolute must-knows.

Upgrade to premium to access monthly AI round-ups and deep dives into key topics like OpenAI's DevDay and autonomous agents.

Become a premium member

Welcome to the November edition of AI Tidbits, where we unravel the latest and greatest in AI. November was filled with groundbreaking announcements from industry leaders and exciting progress in open-source AI.

This month, OpenAI announced GPTs, Assistants, and GPT-4 Turbo, followed by Anthropic's new release of Claude 2.1 and Perplexity's novel language models having access to online information. On the open source front, Yi open-sourced a 200k context window model, and Berkeley unveiled Starling 7B which has achieved performance on par with GPT-4, thanks to RLHF.

In the arenas of generative image and video, there has been a surge of remarkable advancements. Pika unveiled a groundbreaking text-to-video model, pushing the boundaries of visual content creation. Stability AI introduced Stable Diffusion XL, revolutionizing the generation of high-quality images at remarkable speeds, and launched Stable Video Diffusion, marking their foray into video generation.

These and many more exciting updates across novel promoting frameworks, autonomous agents, multimodal AI, and open-source repositories are part of this month’s roundup.

Let's dive in!

Overview

• Industry announcements (12 entries)

• Large Language Models

  • Open-source (5 entries)

  • Research (8 entries)

  • ✨ Special feature: Prompting techniques (6 entries) - Next Sunday, a new AI Tidbits Deep Dive will be sent out, detailing research-backed advanced prompting strategies to boost the performance of LLMs

• Autonomous Agents (5 entries)

• Image and Video (9 entries)

• Audio (3 entries)

• Multimodal (5 entries)

• Open-source (13 entries)

• Cool Tools (4 entries)

Recent AI Tidbits Deep Dives

Industry announcements

1. OpenAI announces a host of product releases in its inaugural dev conference: GPT-4 Turbo with a 128k-context window, a new text-to-speech model, and an autonomous agents API

2. Anthropic announces Claude 2.1 - a new version of its chatbot with a 200k context window, function calling support, and fewer hallucinations

3. Perplexity unveils two new language models, pplx-7b-online and pplx-70b-online, providing up-to-date and accurate, accessible through a first-of-its-kind public API

4. xAI announces Grok-1 - a language model outperforming GPT-3.5 and Llama 2 having real-time direct access to tweets

5. Amazon introduces Amazon Q - its new chatbot for the enterprise, allowing companies to synthesize content and answer questions based on the company's data

6. Pika Labs launches Pika 1.0 - a substantially improved model for video generation and editing

7. Inflection releases Inflection-2 - a new version of its language model

8. Humane introduces the AI Pin - a $699 OpenAI-powered wearable device for everyday use

9. Adept announces Adept Experiments - a new program for users to explore Adept's models, along with ACT-2 - Adept's new multimodal model for UI understanding and action-taking

10. Together AI releases Together Inference Engine - the world's fastest, achieving unparalleled speeds of up to 171 tokens per second

11. Figma introduces the new FigJam AI to automate mundane design tasks

12. Runway ML has updated its Gen-2 video generator to improve video quality and consistency, and to support higher-resolution video generation from images

    

Large Language Models (LLMs)

Open-source

1. 01.ai openly releases Yi-34B - the first Chinese model to top Hugging Face's LLM Leaderboard supporting a 200k context window

Read more

This is the monthly curated round-up, so if you're pressed for time and can only catch one AI Tidbits edition, this is the one to read—featuring the absolute must-knows.

If you find AI Tidbits valuable, share it with a friend and consider showing your support.

Support AI Tidbits

Welcome to the October edition of AI Tidbits, where we unravel the latest and greatest in AI. October was filled with innovative breakthroughs and groundbreaking research showcasing the astounding pace of progress in AI.

Leading the charge were open-source multimodal models with the likes of Adept’s Fuyu, LLaVA 1.5, and Obsidian - the world’s smallest multimodal AI. On the open-source front, Hugging Face released Zephyr, a language model beating Anthropic’s Claude 2 on AlpacaEval, and Distil-Whisper - a speech2text model that is 6x faster compared to OpenAI’s Whisper.

Apple joined the generative AI race with a few new papers (Matryoshka, SAM-CLIP) and Google DeepMind was hard at work with new techniques to generate high-quality training data for robotics.

These and many more exciting updates across multimodal AI, video models, and open-source tools are part of this month’s roundup.

Let's dive in!

Overview

• Large Language Models

  • Commercial (5 entries)

  • Research (4 entries)

  • Open-source (8 entries)

• ✨ Special feature: Multimodal AI (10 entries)

• Autonomous Agents (4 entries)

• Image and Video (10 entries)

• Robotics (5 entries)

• Cool Tools (5 entries)

• Open-source (6 entries)

Recent AI Tidbits Deep Dives

Large Language Models (LLMs)

Commercial

1. OpenAI's ChatGPT now supports all of its modes in one conversation: Browsing, Advanced Data Analysis, and DALL-E 

2. Phind releases a new 16k context model that beats GPT-4 at coding at GPT-3.5-like speed

3. Perplexity releases two new language models, pplx-7b-chat and pplx-70b-chat, that substantially outperform Llama 2 according to human evaluators

4. Google brings image generation to its Bard chatbot through its text2image model Imagen 

5. Amazon rolls out a suite of AI-powered image generation tools to help advertisers improve their product images

Research

1. DeepMind presents Step-Back Prompting - a two-step abstraction-and-reasoning process resulting in significant performance gains, including a 27% improvement on TimeQA and up to 36% over other prompting methods

2. Nvidia introduces SteerLM - a technique that enables real-time customization of LLMs during inference, showcasing superior performance on benchmarks and broad applicability across gaming, education, and enterprise sectors

3. CMU and Google introduce AutoMix - directing queries to larger LMs based on smaller LMs' output reliability to reduce costs while maintaining performance

4. Researchers release Self-RAG - a framework and models (7B + 13B) boosting LLMs' accuracy and quality by adaptively retrieving relevant information as needed

Open-source

1. Hugging Face releases Zephyr - a series of Mistral-based chat models with comparable performance to Anthropic's Claude 2 on AlpacaEval

Read more