AI Roundup 07/11 -> 07/18/2024

Mistral's Codestral-Mamba sets new standards for code models, Microsoft's SpreadsheetLLM to optimize spreadsheet tasks, a new model to detect hallucinations, and a Claude-powered AI engineer

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (6 entries)

• Language Models (13 entries)

• Multimodal (6 entries)

• Autonomous Agents (1 entry)

• Vision (12 entries)

• Audio (2 entries)

• Open-source Packages (3 entries)

Recent Deep Dives

Top 8 leaderboards to choose the right AI model for your task

12 techniques to reduce your LLM API bill and launch blazingly fast products

Harnessing research-backed prompting techniques for enhanced LLM performance

✨ Highlights

1. Mistral partners with the Mamba creators to release the commercially permissive Codestral-Mamba 7B - the strongest code model for its size, featuring a 256k context window (Mistral)

   

2. Mistral releases Mathstral - an open model excelling in multi-step logical reasoning for STEM subjects, achieving state-of-the-art scores on key benchmarks (Mistral)

   

3. Lynx releases a commercially permissive state-of-the-art hallucination detection model that surpasses GPT-4 and Claude-3-Sonnet, offering enhanced accuracy across diverse scenarios (Company blog)

   

4. Hugging Face releases a new state-of-the-art model series of its multimodal on-device LLM, SmolLM (135M, 360M, 1.7B), outperforming Phi-1.5 and Qwen2 (Hugging Face)

   

5. Google has started rolling out access to its AI-powered video generation product for presentations, Google Vids (Google Blog)

   

6. The European Union releases the full and final text of the EU AI Act, implementing risk-based regulations for AI applications with full enforcement by mid-2026 (TechCrunch)

Language Models

1. Together AI showcases FlashAttention-3 - achieving significant speed improvements for LLMs by harnessing the power of Nvidia’s H100s to enhance efficiency and performance

   

2. Microsoft presents SpreadsheetLLM - an encoding method that optimizes LLM performance on spreadsheets, achieving a state-of-the-art 80% F1 score and a 25x compression ratio, surpassing existing models by 12.3%

   

3. OpenAI introduces "prover-verifier games", where AI models generate and verify solutions to enhance accuracy and transparency

   

4. The Qwen team publishes the technical report for its powerful Qwen2 series of multimodal LLMs ranging from 0.5 to 72B parameters, surpassing previous models in diverse benchmarks

   

5. Groq and Glaive openly release a new Llama 3 based model for tool use (aka function calling), achieving 90.7% accuracy

   

6. UCSD and Google unveil Speculative RAG - a framework combining large and small language models to enhance RAG performance, achieving up to 13% improved accuracy and 51% reduced latency on key benchmarks

   

7. Researchers present NeedleBench - a framework for evaluating the long-context capabilities of LLMs, revealing significant room for improvement in identifying and reasoning with relevant information from lengthy documents

   

8. Huawei and UCL Researchers introduce EM-LLM, enabling LLMs to process infinite contexts by mimicking human episodic memory, achieving a 4.3% improvement over SOTA models

9. DeepMind presents an analysis revealing common tactics of generative AI misuse based on the analysis of 200 incidents, including misinformation and fraud

10. EPFL reveals that past tense reformulations can bypass refusal training in LLMs, increasing jailbreak success rates and highlighting the brittleness of current alignment techniques

    

11. Datadog presents Toto - a state-of-the-art foundation model for time series forecasting, excelling in observability metrics and general-purpose forecasting with one trillion data points

    

12. Georgia Tech develops GraphFM - a scalable multi-graph pretraining approach using a Perceiver-based encoder to generalize node classification across diverse datasets, achieving competitive performance with specialist models and reducing dataset-specific training needs

    

13. Sakana AI reveals that frozen transformers exhibit uniform middle layers and robustness to layer reorganization, allowing for effective accuracy-latency trade-offs and potential model efficiency improvements

    

14. Nvidia and HKUST showcase DoRA - a novel weight decomposition method that enhances LoRA's learning capacity and stability, consistently outperforming it on various downstream tasks without additional inference costs

    

Multimodal

1. OpenGVLab openly releases the InternVL2 family of powerful multimodal LLMs, ranging from 1B to 108B

2. Nvidia presents MambaVision - a hybrid backbone combining Mamba and Vision Transformers, achieving state-of-the-art performance in image classification and outperforming comparable models in various downstream tasks

   

3. DeepMind showcases Mobility VLA - enabling robots to navigate complex real-world environments using Gemini 1.5 Pro with up to 90% success using multimodal inputs

   

4. Researchers present LMMS-EVAL - a comprehensive benchmark framework for Large Multimodal Models with over 50 tasks, addressing the evaluation trilemma with LMMs-Eval-Lite and Multimodal LiveBench

   

5. Researchers introduce E5-V - a framework that adapts MLLMs for universal multimodal embeddings, achieving state-of-the-art performance and reducing training costs by 95% through single-modality training

   

6. CUHK and Peking University introduce MAVIS - a tuning paradigm for multimodal LLMs, addressing visual math challenges and enhancing mathematical reasoning with 900K annotated problems and specialized training stages

Autonomous Agents

1. Researchers develop and fully open-source Make-An-Agent - a policy generator that uses conditional diffusion models to create effective control policies from a single demonstration, applicable in simulations and real-world robotics

   

Vision

1. Fal introduces AuraFlow - a 6.8B parameters open text-to-image generation model, showcasing advanced training techniques

   

2. Tencent introduces Mira - an experimental framework pushing the boundaries of long-duration to generate Sora-level high-dynamic text-to-video generation

   

3. Google introduces Still-Moving - a novel framework for customizing text-to-video models without special video data, combining text-to-image details with video movements

   

4. KAUST presents Goldfish - a method for comprehending videos of arbitrary lengths, achieving a 41.78% accuracy on the TVQA-long benchmark by efficiently retrieving relevant video clips

   

5. Tencent unveils SEED-Story - a method using a Multimodal LLM to generate extended multimodal stories with coherent text and consistent images, introducing a multimodal attention sink for efficient sequence generation

   

6. Tsinghua University unveils CharacterGen - a framework for generating 3D characters using a multi-view diffusion model and sparse-view reconstruction, producing high-quality shapes and textures

   

7. Snap and Northeastern University present a weight generator that uses a diffusion model to synthesize neural weights for deep neural network initialization, achieving a 15x training acceleration and improved image generation quality

   

8. Google releases YouTube-SL-25 - a multilingual corpus with over 3000 hours of sign language videos and captions, making it the largest parallel sign language dataset to date

9. Researchers develop DiT-MoE - a scalable and efficient sparse diffusion Transformer that achieves state-of-the-art performance in conditional image generation, optimizing inference with innovative expert routing and balance loss designs

10. ETH Zurich and CTU present WildGaussians - a novel method that leverages DINO features and appearance modeling to achieve state-of-the-art 3D scene reconstruction for in-the-wild data, matching real-time rendering speeds

    

11. CMU publishes a method leveraging dense gradient information from pre-trained reward models to align video diffusion models efficiently, surpassing prior gradient-free methods in computation and sample efficiency

    

12. Google introduces CUBE - a new benchmark to evaluate the cultural competence of text-to-image models, focusing on cultural awareness and diversity across multiple countries and concepts

Audio

1. Researchers present Emilia - a 101k-hour multilingual speech generation dataset, and Emilia-Pipe, a preprocessing pipeline that converts in-the-wild speech data into high-quality training data in minutes

   

2. Microsoft introduces MELLE - a new method for text-to-speech that generates smoother audio directly from text, outperforming older models by avoiding quality loss

Open-source Packages

1. Claude Engineer - an interactive command-line interface that leverages the power of Anthropic's Claude-3.5-Sonnet to assist with software development tasks

2. CosyVoice - multi-lingual large voice generation model, providing inference, training, and deployment full-stack ability

3. LiteLLM - build LLM-agnostic application by supporting all LLM APIs, including Bedrock, OpenAI, Cohere, Anthropic, HuggingFace, and more

Plus >70 more open-source packages for AI engineers

Building a web app with Claude Engineer

Last week’s AI Tidbits roundup

AI Roundup 07/04 -> 07/11/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.