AI Roundup 09/19 -> 09/26/2024

Open multimodal models rivaling GPT-4o, Meta's debut of its first multimodal Llama models, ChatGPT's long due Advanced Voice release, DeepMind's method for LLMs self-correction, and text2podcasts repo

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (7 entries)

  • ✨ Special feature: Meta Connect 2024

• Language Models (10 entries)

• Multimodal (2 entries)

• Autonomous Agents (2 entries)

• Vision (6 entries)

• Audio (1 entry)

• Open-source Packages (4 entries)

✨ Highlights

1. AI2 releases Molmo - a new family of multimodal models, with the smallest variant (1B) nearly matching GPT-4V, the medium model (7B) rivaling GPT-o, and the largest model (72B) surpassing Gemini 1.5 and Claude 3.5 Sonnet, all available for commercial use (AI2)

2. Meta releases Llama 3.2 - a new family of open models featuring edge-optimized text models (1B and 3B) and Meta's first large multimodal models (11B and 90B) supporting 128K tokens (Meta AI)

   

3. OpenAI releases Advanced Voice Mode for ChatGPT Plus and Team users, adding Custom Instructions, Memory, and five new voices (X)

4. Deepgram unveils its Voice Agent API, enabling natural, real-time human-machine conversations powered by high-performance speech recognition and synthesis models (Company blog)

5. Labelbox releases new leaderboards and a multimodal AI evaluation system that uses expert human evaluations and metrics like Elo and TrueSkill to assess image, speech, and video generation models (Company blog)

   

6. Former top Apple designer Jony Ive confirms collaborating on a new AI device with OpenAI (The Verge)

7. OpenAI's CTO, Mira Murati, along with other OpenAI execs to step down and leave the company (X)

We have partnered with Deepgram to offer AI Tidbits subscribers $200 in free credits to build with Deepgram’s new Voice AI API (no credit card required). Build voice agents that automate customer service, manage orders in real-time, assist with scheduling and reminders, and more.

Meta Connect 2024

Meta announced a host of AI-related releases at its annual Meta Connect event:

• Ray-Ban Meta smart glasses introduced real-time AI video processing, enabling users to ask questions about their surroundings while also offering live translation and reminder features

• Meta showcased its AI’s visual search capabilities, allowing users to edit and share images based on feedback directly to Instagram

• Meta is testing AI-powered tools for translating and dubbing creator content, syncing dubbed voices with accurate lip movements, currently limited to English and Spanish

• Meta AI now offers vocal responses across platforms like Messenger, WhatsApp, and Instagram, allowing users to interact via voice commands with options for celebrity voices

Full coverage

Recent AI Tidbits Deep Dives

LinkedIn Highlights, Aug 2024

Top 8 leaderboards to choose the right AI model for your task

12 techniques to reduce your LLM API bill and launch blazingly fast products

Language Models

1. Alibaba releases Qwen2.5-Coder - two powerful code-specific models (1.5B, 7B) that excel in diverse code-related tasks, outperforming larger models and offering broad applicability through its permissive licensing

   

2. HuggingChat releases a native macOS app with Qwen and Llama models, featuring markdown support, web browsing, and code syntax highlighting

   

3. Anthropic releases Contextual Retrieval - improving RAG retrieval accuracy by embedding document chunks with relevant context generated by Claude, reducing failures by up to 49% (Anthropic)

   

4. Google introduces new Gemini-1.5 Pro and Flash models, with faster response speeds, reduced prices (>50%) and improved task performance

   

5. DeepMind develops SCoRe - a reinforcement learning-based approach that enhances self-correction in LLMs by 15.6% and 9.1% on MATH and HumanEval benchmarks without external supervision

   

6. Microsoft presents GRIN - a new training method for Mixture-of-Experts models, achieving impressive performance and efficiency by outperforming dense models on various tasks (6.6B activated parameters rivaling the performance of a 14B dense model)

   

7. Princeton develops Time-MoE - a state-of-the-art scalable time series forecasting model, which uses sparse mixture-of-experts to achieve superior precision while reducing inference costs and outperforming dense models across nine domains

   

8. UT Austin Researchers show that Chain-of-Thought prompting is most beneficial for math and logic tasks but offers minimal gains for other tasks, underlining the need for selective application to save inference costs

   

9. UC Santa Cruz shows OpenAI's o1 as a model that improves upon GPT-4's medical task performance by up to 6.6%, though they note ongoing issues with hallucination and multilingual consistency

   

10. Researchers introduce HelloBench - a benchmark for evaluating long text generation capabilities of LLMs, revealing their limitations in generating text over 4000 words and common quality issues like repetition

    

Multimodal

1. CUHK and ByteDance propose MMSearch-Engine, which empowers LMMs for multimodal search, outperforming commercial models like Perplexity Pro in comprehensive search tasks

   

2. KAIST introduces Phantom - an efficient large language vision model family with model sizes up to 7B parameters, achieving large-model performance by temporarily expanding latent dimensions during self-attention, offering superior performance with reduced hardware demands

Autonomous Agents

1. Researchers publish HyperAgent - a generalist multi-agent system automating the complete software lifecycle, achieving a 31.4% success rate in GitHub issue resolution and surpassing other coding methods

   

2. Berkeley introduces TinyAgent - a framework for task-specific small language models that outperform larger models in function calling while being fully deployable at the edge

   

Vision

1. Meta releases Imagine Yourself - an AI model that generates personalized images from a single reference image using synthetic data and a novel dual-processing architecture

   

2. ETH Zurich researchers showcase a method using YOLO models to bypass reCAPTCHAv2 with 100% accuracy, raising concerns about the future of image-based CAPTCHAs

   

3. CUHK and Tencent Researchers develop LVCD - a video diffusion framework for reference-based lineart colorization, offering state-of-the-art quality and temporal consistency by incorporating innovations like Sketch-guided ControlNet and Reference Attention to handle large motion animations

   

4. Alibaba unveils MIMO - a character video synthesis framework that enables controllable, 3D-aware video generation from simple user inputs while supporting arbitrary characters, complex motions, and real-world scene interactions

   

5. Researchers present StoryMaker - a tuning-free image generation solution that preserves character consistency in facial features, clothing, and body, enabling coherent storytelling across images while maintaining high fidelity

   

6. Researchers present the YesBut dataset and propose challenging tasks for Vision-Language models, including detecting, understanding, and completing satirical images, revealing poor performance in Zero-Shot settings

Audio

1. Google develops a zero-shot Voice Transfer module that enables cross-lingual voice transfer with 73% similarity and restores vocal identity for dysarthric speakers, demonstrating significant utility in multi-lingual TTS systems

Open-source Packages

1. PDF2Audio - turn PDFs into audio podcasts, inspired by Google’s NotebookLM recent release

2. Qwen-Agent - an agent framework and applications built upon Qwen2, featuring Function Calling, Code Interpreter, RAG, and Chrome extension

3. Outlines - get structured data back from LLMs, consistently

4. QA-MDT - state-of-the-art text2music generation

Plus >70 more open-source packages for AI engineers

Generate podcasts similar to the ones generated by Google’s NotebookLM

Last week’s AI Tidbits roundup

AI Roundup 09/12 -> 09/19/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.