AI Roundup 09/12 -> 09/19/2024

OpenAI's o1 models excel in reasoning, a cutting-edge open speech2speech model, Alibaba's SOTA Qwen 2.5 LLMs, Luma + Runway video generation APIs, and open-source models for PII stripping and LLM eval

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (6 entries)

• Language Models (8 entries)

• Multimodal (1 entry)

• Autonomous Agents (3 entries)

• Vision (4 entries)

• Audio (2 entries)

• AI Tools (2 entries)

• Open-source Packages (3 entries)

✨ Highlights

1. OpenAI introduces o1-preview and o1-mini, advanced models that excel in high-level reasoning for coding and math, with o1-mini being a faster, cost-efficient option (OpenAI)

   

2. Kyutai Labs openly releases Moshi - a 7.6B open speech-to-speech model with cutting-edge performance and low latency, alongside Mimi, a SoTA streaming audio codec that compresses 24 kHz audio to 1.1 kbps for optimized real-time speech communication (X)

3. Alibaba releases Qwen 2.5 - a family of open multilingual models handling 128K tokens and outperforming competitors like Mistral 2 (123B) on major benchmarks, offering multilingual support in 29 languages and specialized models across Math and coding (Company blog)

   

4. Luma Labs and Runway launch APIs for video generation and editing (VentureBeat)

   

5. Microsoft releases new Microsoft 365 Copilot features, including Copilot Pages for real-time collaboration, Python integration in Excel for advanced data analysis, and Copilot Agents for automating business tasks with AI (Microsoft Blog)

   

6. YouTube introduces Dream Screen, enabling creators to generate AI-powered videos with Veo and Imagen 3 models, enhancing creative workflows on Shorts (DeepMind)

   

Recent AI Tidbits Deep Dives

Top 8 leaderboards to choose the right AI model for your task

12 techniques to reduce your LLM API bill and launch blazingly fast products

Harnessing research-backed prompting techniques for enhanced LLM performance

Language Models

1. Mistral AI releases a free API tier and cuts prices by up to 80%, making their AI models more accessible to developers and businesses

   

2. Haize Labs introduce Bijection Learning - a jailbreak attack that teaches language models custom encoded languages, achieving an 86.3% success rate on safety-trained models like GPT-4o and Claude 3.5 Sonnet

   

3. Flow AI open sources Flow Judge - a lightweight 3.8B parameter model optimized for fast and accurate LLM evaluations with customizable rubrics

   

4. Researchers open source a model to strip Personal Identifiable Information (PII) from input text, achieving 98% recall and precision

   

5. Google presents DataGemma - a set of models that reduce hallucinations in LLMs by using both RAG and RIG (Retrieval-Interleaved Generation) techniques to enhance the accuracy of LLM outputs by grounding them in over 240B real-world data points

   

6. John Hopkins University releases Promptriever - the first retrieval model that can be prompted like a language model, achieving state-of-the-art performance on instruction-following and retrieval tasks

   

7. Aleph Alpha and the Technical University of Darmstadt propose T-FREE - a tokenizer-free language model with 85% parameter reduction and strong multilingual transfer learning performance

   

8. Chai Discovery presents Chai-1 - an AI model that predicts 3D biomolecular structures with impressive accuracy, outperforming AlphaFold in key areas

   

Multimodal

1. Nvidia releases NVLM 1.0 - a multimodal LLM that advances vision-language reasoning, boosts text-only performance, and excels in math and coding tasks through innovative architecture and curated datasets

   

Autonomous Agents

1. Microsoft releases WindowsAgentArena (WAA) - a scalable benchmark for evaluating multi-modal agents in a real Windows OS environment, with tasks requiring planning and tool usage

   

2. CMU and MIT publish Agent Workflow Memory (AWM) - a method that enables agents to learn and reuse task workflows, improving their performance on long-horizon tasks like web navigation by 24.6% on Mind2Web and 51.1% on WebArena

3. MIT unveils AgentTorch - a scalable framework that integrates LLMs into agent-based models, enabling the simulation of millions of adaptive agents for more realistic behavior modeling

Vision

1. Researchers introduce InstantDrag - an optimization-free image editing pipeline that leverages drag instructions to perform real-time edits without requiring additional inputs like masks or text prompts

   

2. Researchers introduce OmniGen - a simplified and unified diffusion model for image generation that supports tasks like editing, pose recognition, and edge detection without needing additional modules

   

3. CUHK proposes DrawingSpinUp - a system that transforms 2D character drawings into high-quality 3D animations, overcoming challenges like contour lines and slim structures for more realistic results

   

4. Nanyang Technological University releases Vchitect 2.0 - an open text-to-video model capable of generating 20-second high-quality videos

Audio

1. Researchers release EzAudio - an open text-to-audio diffusion model that improves generation quality and efficiency by leveraging 1D waveform latent spaces and an optimized diffusion transformer architecture

2. ByteDance introduces Seed-Music - a Suno-like versatile music generation system that allows for high-quality, controlled music creation using multi-modal inputs and direct post-production editing tools

AI Tools

1. Diffusers Image Outpaint - extend images beyond their original boundaries with AI

2. EzAudio - generate high-quality audio from text

   

   Outpainting powered by diffusion models

Open-source Packages

1. Mini-Omni - an open-source multimodal LLM that can hear, talk while thinking, featuring real-time end-to-end speech input and streaming audio output conversational capabilities

2. ChatTTS - a generative speech model for daily dialogue

3. ell - a lightweight, functional prompt engineering framework to launch better LLM-powered apps

Plus >70 more open-source packages for AI engineers

Multimodal prompt tracing with ell

Last week’s AI Tidbits roundup

AI Roundup 09/05 -> 09/12/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.