AI Roundup 07/04 -> 07/11/2024

A new text-to-video startup to disrupt Hollywood, better tools from Anthropic for AI developers, a Math Olympiad winning LLM, and novel models for document extraction

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (4 entries)

• Language Models (11 entries)

• Multimodal (3 entries)

• Vision (9 entries)

• Audio (1 entry)

• AI Tools (3 entries)

• Open-source Packages (3 entries)

Recent Deep Dives

Top 8 leaderboards to choose the right AI model for your task

12 techniques to reduce your LLM API bill and launch blazingly fast products

Harnessing research-backed prompting techniques for enhanced LLM performance

✨ Highlights

1. Odyssey, a new startup emerging from stealth, presents Hollywood-grade visual AI, offering filmmakers and artists detailed control over geometry, materials, lighting, and motion for high-end productions (Company blog)

   

2. Alibaba openly release FunAudioLLM - a model family enhancing voice interactions with LLMs through SenseVoice for speech and emotion recognition and CosyVoice for multilingual speech generation (Project website)

   

3. Anthropic releases a revamped developer console to help LLM users and developers optimize and evaluate prompts, including a side-by-side comparison of outputs (X)

   

4. Perplexity AI presents an upgraded Pro Search capable of multi-step reasoning and advanced computational tasks, transforming it into a comprehensive research (Perplexity)

   

Language Models

1. Meta proposes MobileLLM - an efficient LLM designed for mobile devices, achieving a 4.3% accuracy boost over previous 125M/350M state-of-the-art models through innovative architecture

   

2. The AI Math Olympiad winning model has been openly released, including for commercial use

   

3. Researchers propose ColPali - a new model architecture that excels in leveraging visual cues for high-quality contextualized embeddings and introduce ViDoRe - a benchmark for visually rich document retrieval

   

4. Researchers present DictaLM2.0 and DictaLM2.0-Instruct - LLMs trained on Hebrew and English using novel adaptation techniques for low-resource languages

5. DeepMind proposes PEER - a novel layer design that efficiently utilizes a massive pool of over a million tiny experts to outperform dense feedforward and coarse-grained MoE layers in transformer models

6. Meta showcases self-supervised methods to distill System 2 techniques into LLM generations, enhancing performance and reducing inference costs compared to traditional System 2

   

7. Stanford and UCSD introduce Test-Time Training (TT) layers - a new class of sequence modeling layers with linear complexity and expressive hidden states that train even on test sequences, outperforming Transformers and RNNs in long context tasks

   

8. Oxford develops semantic entropy probes (SEPs) - an efficient method for hallucination detection in LLMs that significantly reduces computational overhead by approximating semantic entropy from hidden states

9. Tsinghua University and Peking University develop the Internet of Agents (IoA) framework to enable agents to seamlessly collaborate and achieve greater intelligence, outperforming state-of-the-art baselines

10. Snorkel AI develops the SWiM framework and medoid voting method to benchmark and improve long-context language models, enhancing performance by up to 24% in accuracy and addressing mid-context information loss

    

11. Researchers present RegMix - an automatic method to identify high-performing data mixtures for large language model pre-training, significantly improving performance with minimal compute

    

Multimodal

1. ByteDance and HKUST develop LLaVA-NeXT-Interleave - an advanced Large Multimodal Model achieving top results in multi-image, video, and 3D tasks while maintaining single-image performance and demonstrating cross-modal task transfer capabilities

   

2. Google published the technical report of its recently released PaliGemma - an open Vision Language Model (VLM)

   

3. Researchers present Anole - an open-source multimodal model excelling in interleaved text-image generation with minimal training, built upon Meta's Chameleon's strengths

   

Vision

1. Researchers introduce ChartGemma - a PaliGemma-based chart understanding model trained on chart images to capture visual trends and patterns, achieving state-of-the-art results across multiple benchmarks

   

2. Researchers find that state-of-the-art LLMs with vision capabilities such as GPT-4o and Gemini 1.5 struggle with basic visual tasks despite excelling in benchmarks, indicating their vision is akin to severe myopia or intelligent blindness

   

3. MIT and TUM develop Diffusion Forcing - a method combining autoregressive and diffusion models for stable long-sequence video generation and robust robotic planning

   

4. Researchers introduce EVE - an encoder-free vision-language model that integrates vision and language inputs via a unified decoder and enhances visual recognition through extra supervision

   

5. Stanford and Google unveil Video-STaR - a self-training approach for video instruction tuning that enhances large vision LMs using labeled video datasets, significantly improving performance in video QA and downstream tasks

   

6. Researchers propose MJ-Bench, a comprehensive benchmark for evaluating multimodal judges in text-to-image models, highlighting GPT-4o's superior feedback performance across key criteria

   

7. Researchers introduce This&That - a robot learning method that uses video generative models and language-gesture conditioning to improve task planning and execution

   

8. UNC-Chapel Hill and Brown introduce RULE - a method for improving factual accuracy in Medical visual language models by optimally calibrating retrieved contexts and fine-tuning the model to balance inherent and retrieved knowledge

   

9. Mila and McGill University develops the Aurora Dataset and benchmark, significantly enhancing image editing models' performance on complex tasks and proposing improved evaluation metrics

Audio

1. Tencent showcases a state-of-the-art video-to-audio generation model that leverages vision encoders, auxiliary embeddings, and data augmentation techniques, achieving high-quality and synchronized audio output

AI Tools

1. FoleyCrafter - add sound to silent videos

2. Tile Upscaler - improve image quality using ControlNet

3. LivePortrait - turn static portraits into short animations

LivePortrait generates short animated portraits

Open-source Packages

1. Pipecat - an open-source framework for voice and multimodal conversational AI

2. Fabric - a framework for augmenting humans using AI, providing a modular framework for solving specific problems using a crowdsourced set of AI prompts

3. Micro Agent - an AI agent that writes code and tests for you

Plus >70 more open-source packages for AI engineers

Micro Agent in action

Last week’s AI Tidbits roundup

AI Roundup 06/27 -> 07/04/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.