AI Roundup 09/05 -> 09/12/2024

Mistral releases its first multimodal model, Fish Audio debuts an advanced open-source voice cloning system, novel open coding models, and Alibaba unveils a cutting-edge multi-page document model

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (7 entries)

• Language Models (18 entries)

• Multimodal (2 entries)

• Autonomous Agents (1 entry)

• Vision (5 entries)

• Audio (2 entries)

• Open-source Packages (3 entries)

Recent Deep Dives

Top 8 leaderboards to choose the right AI model for your task

12 techniques to reduce your LLM API bill and launch blazingly fast products

Harnessing research-backed prompting techniques for enhanced LLM performance

✨ Highlights

1. Mistral openly releases Pixtral 12B - its first multimodal model (Hugging Face)

   

2. Fish Audio releases Fish Speech 1.4 - an advanced multilingual voice model with instant voice cloning and low-latency TTS, fully open-source for global developers (X)

3. Hume releases Empathic Voice Interface 2 (EVI 2) - a GPT-4o-like voice model, allowing users to converse with its AI chatbot with sub-second response times (X)

4. Replit announces Replit Agent - an AI tool that automates software development tasks like environment setup and deployment (X)

   

5. Stanford shows that LLM-generated ideas are judged more novel than expert human ideas in a head-to-head comparison, while slightly weaker on feasibility (X)

   

6. Altera presents Project Sid - the first AI civilization with autonomous agents forming societies, governments, and economies in a Minecraft server (X)

   

7. Adobe previews its generative AI video tools, allowing users to create short video clips from text and images with adjustable camera settings (The Verge)

Language Models

1. Anthropic releases the Quickstarts repo, providing ready-to-deploy app projects powered by its API, starting with a Claude-based customer support agent

2. The DeepSeek team releases DeepSeek-V2.5 - a SOTA versatile open model integrating DeepSeek-Coder with advanced features like Function Calling and JSON output (X)

   

3. Alibaba proposes DocOwl2 - a state-of-the-art model for multi-page document understanding that reduces GPU usage and inference time by compressing high-resolution document images into 324 tokens

   

4. Researchers openly release GOT - a highly versatile OCR-2.0 580M model that excels in processing a wide range of optical characters and supports dynamic resolution, multi-page OCR, and region-level recognition for enhanced performance

   

5. 01.AI unveils Yi-Coder - a high-performing series of code LLMs with up to 9B parameters, excelling in long-context modeling and outperforming other bigger models

   

6. Arcee AI unveils Arcee-SuperNova - state-of-the-art models (8B, 70B) and the first distilled and optimized versions of Llama-3.1 for superior instruction-following and human preference alignment as an OpenAI alternative

   

7. Phind releases Phind-405B, excelling in coding and technical tasks with a 92% HumanEval score and a 128K token context

   

8. Jina AI releases two multilingual small LMs (0.5B, 1.5B) that outperform larger counterparts in converting HTML to markdown, supporting up to 256K tokens

   

9. Researchers introduce XCoder - a state-of-the-art coding model family finetuned from LLaMA3, using an efficient data pruning strategy that addresses data leakage in code datasets and sets new performance standards

   

10. Microsoft showcases dynamic few-shot prompts, retrieving relevant examples from a vector store to improve AI response accuracy, latency, and cost

    

11. Researchers introduce ECHO - a self-harmonized CoT method that consolidates diverse solution paths, achieving state-of-the-art performance across three reasoning domains

    

12. Researchers propose SongCreator - a song-generation system featuring a dual-sequence language model that excels in generating both vocals and accompaniment from lyrics, achieving state-of-the-art performance

    

13. SambaNova announces SambaNova Cloud, offering the fastest AI inference service with Llama 3.1 models, delivering record speeds of 132 tokens/sec for the 405B model and 461 tokens/sec for the 70B model

    

14. Peking University and Alibaba propose a unified framework that connects various LLM alignment strategies by breaking down the process into four key components: model, data, feedback, and algorithm

    

15. Researchers propose GroUSE - a new meta-evaluation benchmark that identifies key failure modes in RAG systems and highlights the limitations of existing automated evaluation frameworks, including those using GPT-4 as a judge

    

16. UIUC and DeepMind introduce a multi-turn direct preference learning framework that significantly boosts LLMs' performance on GSM8K (77.5% to 83.9%) and MATH (46.1% to 51.2%) through tool integration and multi-turn reasoning

    

17. Tsinghua University and UCSD unveil bricks - modular functional units in LLMs, to improve computational efficiency and scalability by enabling dynamic model assembly for complex tasks

    

18. Ligo Biosciences releases an open-source implementation of AlphaFold 3 for efficient protein structure prediction, supporting single-chain proteins

Multimodal

1. Researchers propose LLaMA-Omni - a new model architecture that achieves low-latency, high-quality speech interaction with LLMs, bypassing transcription and providing a 226ms response time

   

2. Researchers introduce MMEvol - an innovative framework that evolves multimodal instruction data, improving MLLM performance by 3.1 points and achieving state-of-the-art results in 9 vision-language tasks

Autonomous Agents

1. MIT develops SciAgents - an AI system that autonomously uncovers interdisciplinary scientific insights and generates hypotheses using large-scale knowledge graphs, LLMs, and multi-agent systems

   

Vision

1. Researchers present ViewCrafter - an innovative method combining video diffusion models and 3D point clues to create high-fidelity, consistent, novel views from sparse images, showcasing strong performance in various applications

   

2. A new tutorial on diffusion models explains their core principles and importance in generative AI, aimed at students interested in exploring their application in text-to-image and text-to-video generation

3. Tencent and Tsinghua University present Open-MAGVIT2 - an open-source family of auto-regressive image generation models that achieve state-of-the-art reconstruction, employing innovative token factorization and prediction techniques to improve scalability and quality

   

4. Researchers propose LinFusion - a linear attention mechanism that reduces the time and memory complexity of diffusion models while achieving high-resolution generation (up to 16K resolution) performance on par with StableDiffusion

   

5. Stanford introduces a data selection framework based on perplexity-benchmark correlations, which outperforms DSIR and matches DataComp-LM's best selector, all without pretraining new models

   

Audio

1. Bytedance presents Loopy - an audio-only video diffusion model that surpasses existing methods by capturing long-term motion patterns and improving audio-motion correlation, delivering more lifelike and high-quality results

2. Researchers present Draw an Audio - a cutting-edge video-to-audio synthesis model that excels in automatic sound generation tasks by maintaining content consistency and temporal alignment, outperforming benchmarks

Open-source Packages

1. Text-Guided-Image-Colorization - an interactive image colorization tool that leverages Stable Diffusion and CLIP for user-controlled color generation

2. Mem0 - equip AI assistants and agents with an intelligent memory layer for personalized AI interactions

3. LightEval - a lightweight framework for LLM evaluation

Plus >70 more open-source packages for AI engineers

Colorized old images with the Text-Guided-Image-Colorization repo

Last week’s AI Tidbits roundup

AI Roundup 08/22 -> 09/05/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.