AI Roundup 09/26 -> 10/03/2024

OpenAI debuts Realtime API for low-latency voice interactions, Whisper v3 Turbo boosts multilingual speech-to-text, Pika Labs introduces surreal video effects, and Salesforce's SOTA LLM-as-a-Judge

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (5 entries)

• Language Models (8 entries)

• Multimodal (6 entries)

• Vision (5 entries)

• AI Tools (3 entries)

• Open-source Packages (2 entries)

✨ Highlights

1. As part of its DevDay event, OpenAI releases the Realtime API, allowing developers to create low-latency, voice-to-voice interactions with continuous audio streaming, along with new tools like vision fine-tuning, prompt caching, and model distillation (OpenAI)

2. OpenAI releases Whisper Large v3 Turbo - a new version of its speech-to-text model that is 8x faster with better multilingual support, allowing commercial use

   

3. Pika Labs launches its new video generating model, featuring new "Pikaffects" that transform video subjects with surreal, physics-defying effects like melting and cake-ifying objects (X)

   

4. DeepMind openly releases to AlphaChip - an AI-based system that uses reinforcement learning to design superhuman chip layouts in hours rather than taking weeks or months of human effort (X)

   

5. Microsoft upgrades Copilot with new AI features like Copilot Voice for natural voice control and Copilot Vision for real-time screen understanding (Microsoft Blog)

Recent AI Tidbits Deep Dives

The Great AI Consolidation

LinkedIn Highlights, Aug 2024

12 techniques to reduce your LLM API bill and launch blazingly fast products

Language Models

1. OpenAI releases a multimodal moderation model based on GPT-4, enhancing the detection of harmful text and images across multiple languages

   

2. Salesforce AI Research presents SFR-Judge - a family of models designed to evaluate LLM outputs with higher accuracy and explainability, outperforming competitors in 10 out of 13 benchmarks

   

3. Meta AI and University of Illinois researchers propose CrossEval, showing that LLM performance on complex tasks is limited by their weakest skills

4. Salesforce develops GemFilter - a training-free, efficient method that accelerates LLM inference by 2.4x, reduces memory by 30%, and maintains strong performance across long-context benchmarks

   

5. Nvidia presents MaskLLM - a new way to make LLMs faster and smaller by learning which parts to remove, improving performance across tasks

   

6. Researchers present Ruler - a novel technique using Meta Length Tokens to help LLMs meet length constraints, improving their performance on length-specific tasks with notable gains in evaluation metrics

   

7. Researchers provide a survey on LLM honesty, offering insights into challenges and strategies to enhance how models express their knowledge and handle uncertainty

   

8. Researchers develop ProX - an innovative data refinement framework that enhances LLM pre-training by over 2%, with up to 20.3% gains in domain-specific tasks and improved training efficiency

Multimodal

1. Apple presents MM1.5 - a new family of multimodal models (1B to 30B) designed for visual and textual tasks, leveraging diverse data mixtures

   

2. Researchers introduce EMOVA - an omni-modal assistant that integrates speech capabilities into LLMs while maintaining strong vision-language performance and supporting emotional spoken dialogues

3. Researchers introduce MIO - a multimodal model that can handle speech, text, images, and videos in an end-to-end manner, offering advanced capabilities like interleaved video-text generation (code and models released soon)

   

4. Researchers introduce LLaVA-3D - a framework that extends 2D large multimodal models to 3D scene understanding, achieving state-of-the-art results and 3.5x faster convergence than existing 3D LMMs

   

5. Stanford proposes a cooperative learning approach for bimanual control, enabling efficient synthesis of dexterous two-hand motions by training each hand separately and synchronizing them through latent space manipulation

   

6. Berkeley unveils Robot See Robot Do (RSRD) - a system that enables robots to imitate object manipulation from a single monocular video, achieving a 60% end-to-end success rate using pretrained vision models without task-specific training

   

Vision

1. Alibaba openly releases ACE - an all-in-one diffusion model that excels in visual generation and editing tasks using a unified condition format for multi-modal inputs

   

2. UIUC introduce PhysGen - a cutting-edge method that blends physical simulation and data-driven video generation to create controllable, realistic videos from images, surpassing existing models in realism and performance

   

3. Meta and Oxford unveil Flex3D - a novel framework for 3D generation using arbitrary input views, combining high-quality view selection with transformer-based reconstruction to set new performance standards

   

4. Researchers introduce Emu3 - a next-token prediction-based multimodal model that outperforms diffusion and compositional approaches in tasks like image and video generation, surpassing models like SDXL and LLaVA-1.6

   

5. HKUST introduces Lotus - a diffusion-based model that improves zero-shot dense prediction by predicting annotations directly and simplifying the diffusion process, achieving state-of-the-art performance with vastly enhanced efficiency

AI Tools

1. CogVideoX-5B - generate videos from text

2. PDF to Audio - convert PDFs into an audio podcast, lecture, summary and others

3. Voice-Restore - fix background noise, reverberations, distortions, and signal loss

Open-source Packages

1. Show Me - a visual and transparent alternative to open-source ChatGPT o1

2. MinerU - a one-stop, open-source, high-quality data extraction tool, supports PDF/webpage/e-book extraction

Plus >70 more open-source packages for AI engineers

Show-Me breaks down questions into smaller subtasks to better understand the LLM’s reasoning process

Last week’s AI Tidbits roundup

AI Roundup 09/19 -> 09/26/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.