AI Roundup 08/22 -> 09/05/2024

A new text-to-video model flooding the internet, Alibaba's new Qwen2-VL to dominate vision tasks, an AI coding startup featuring a 400M context window, new techniques for RAG, and $1B for safe AI

Welcome to the weekly edition of AI Tidbits, where I curate the firehose of AI research papers and tools every week so you won’t have to.

📩 Published a new breakthrough paper? Just released an open-source package? Submit it here to ensure we don’t miss it and that it gets featured in next week’s post.

---

Overview

• ✨ Highlights (7 entries)

• Language Models (17 entries)

• Multimodal (6 entries)

• Vision (8 entries)

• Audio (4 entries)

• Open-source Packages (4 entries)

Recent Deep Dives

Top 8 leaderboards to choose the right AI model for your task

12 techniques to reduce your LLM API bill and launch blazingly fast products

Harnessing research-backed prompting techniques for enhanced LLM performance

✨ Highlights

1. Tsinghua University openly release its state-of-the-art text-to-video generation model, CogVideoX-5B (Hugging Face)

   

2. Alibaba Cloud openly releases Qwen2-VL - a commercially permissive open vision-language model suite (72B, 2B) that excels in visual understanding and surpasses or rivals major competitors in performance, including GPT-4o

   

3. Anthropic announces Claude Enterprise, offering a 500,000-token context window and GitHub integration for secure, large-scale organizational use (Anthropic)

   

4. Magic AI, the AI coding startup, presents LTM-2-mini - an ultra-long context model that processes 100 million tokens (~10M lines of code) for advanced code synthesis and real-time reasoning, optimizing memory efficiency (Company blog)

   

5. Researchers present Diffusion Explainer - an interactive tool that visually explains how Stable Diffusion converts text prompts into images, greatly enhancing accessibility for non-experts (Project website)

   

6. LM Studio releases version 0.3.0, adding document-based chat with RAG, OpenAI-like JSON output support, and automatic GPU/CPU balancing for local LLMs (Company blog)

   

7. Ex-OpenAI chief scientist Ilya Sutskever raises $1B at a $5B valuation to develop safe superintelligence (TechCrunch)

Language Models

1. AI21 releases the technical report for Jamba-1.5 - an open hybrid Transformer-Mamba model with up to 94B parameters and 256K-token context length, offering high throughput and low memory usage while outperforming other open-weight models

   

2. AI2 introduces OLMoE - a sparse Mixture-of-Experts language model that activates only 1B parameters per token, achieving state-of-the-art performance and outperforming larger models like Llama2-13B-Chat

   

3. Nvidia compresses Llama 3.1 8B and Mistral NeMo 12B into 4B and 8B models using pruning and distillation, introducing the high-performing MN-Minitron-8B model

   

4. Developers openly release Llama-3.1-Storm-8B - a language model outperforming Meta's Llama-3.1-8B models in critical AI tasks like question answering and function calling

   

5. Cerebras releases DocChat - a fully open-sourced conversational model series for document-based retrieval tasks

   

6. Researchers present Open-FinLLMs - a suite of Financial LLMs that surpass existing models in financial comprehension and multimodal tasks, including tables and time-series data

   

7. Google releases experimental models Gemini 1.5 Flash-8B, Pro, and Flash, optimized for multimodal tasks and improved coding performance

8. Google presents ShieldGemma - an advanced LLM-based safety content moderation suite that outperforms models like Llama Guard and WildCard in identifying harmful content across multiple categories

   

9. Chatbot Arena develops a methodology that isolates writing style from content quality in chatbot rankings, revealing that once style is controlled, models like Claude 3.5 Sonnet and Llama-3.1-405B outperform GPT-4o-mini

   

10. Writer.com showcases Writing in the Margins (WiM) - a novel inference method that enhances the handling of long sequences, improving accuracy by 7.5% on reasoning tasks and boosting F1-scores by over 30% on aggregation tasks

    

11. Salesforce partners with Together AI to introduce LlamaRank - an advanced reranking solution that enhances enterprise search by filtering irrelevant documents and supporting semi-structured data

    

12. Berkeley and Stanford propose TAG - a new paradigm for answering natural language questions over databases by combining language models' reasoning with database querying, outperforming existing methods like Text2SQL and RAG

    

13. Jina AI presents Jina ColBERT v2 - a versatile open retrieval model supporting 89 languages and capable of processing longer text for enhanced global search capabilities

    

14. Nvidia unveils OP-RAG, which enhances RAG for long-context tasks by preserving chunk order, showing superior performance and token efficiency compared to long-context LLMs

    

15. Tsinghua University develops FocusLLM - a framework that extends the context length of decoder-only LLMs, enabling them to efficiently process up to 400K tokens while maintaining strong language modeling ability

    

16. Researchers develop GenRM - a generative verifier that significantly improves LLM performance in algorithmic reasoning tasks by leveraging chain-of-thought reasoning and majority voting, outperforming discriminative models with a 16-64% boost in solution accuracy

    

17. Google showcases MediaPipe's on-device LLM inference, offering on-device efficient, cross-platform AI applications without server-side dependencies

    

Multimodal

1. Nvidia openly releases Eagle - a family of multimodal LLMs that effectively use a mixture of vision encoders with simple token concatenation, surpassing state-of-the-art models on multimodal LLM benchmarks

   

2. Researchers develop Show-o - a unified transformer model that outperforms or matches specialized models in multimodal tasks, combining autoregressive and diffusion modeling to handle diverse inputs and outputs

   

3. Hugging Face publishes a comprehensive paper discussing the challenges in building vision-language models and offering methods to address these challenges

4. Google introduces GameNGen - a neural-powered game engine that simulates classic games like DOOM in real-time with high-quality visuals and stable long-term gameplay

   

5. Stanford and Berkeley present the Law of Vision Representation, unveiling a linear link between cross-modal alignment and multimodal LLM performance, drastically reducing computational costs by 99.7% without needing to fine-tune the language model

   

6. LAION presents Re-LAION-5B - a vision-language dataset with enhanced safety, removing flagged content while maintaining over 5.5 billion image-text pairs for research

Vision

1. Meta introduces Sapiens - a family of human-centric 0.3B to 2B parameters vision models trained on 300 million images, excelling in 2D pose estimation, body segmentation, depth estimation, and surface normal prediction

   

2. Zhipu AI and Tsinghua University propose the CogVLM2 family, a new generation of visual language models that achieve state-of-the-art results in image and video understanding, supporting resolutions up to 1344×1344 pixels

   

3. Researchers present VideoLLaMB - a novel framework for long video understanding, capable of encoding 100% of video content and achieving state-of-the-art performance across multiple benchmarks

   

4. Twelve Labs presents TWLV-I - a video foundation model that outperforms existing models in both motion and appearance understanding, showing a 7.7% improvement across action recognition benchmarks

5. KAUST unveils Build-A-Scene - a diffusion-based text-to-image generation method with 3D layout control, allowing interactive object modifications while preserving scene consistency

   

6. CUHK and ByteDance develop SAM2Point - a novel 3D segmentation method based on SAM 2, achieving zero-shot segmentation across various 3D datasets and scenarios without additional training

   

7. University of Washington and DeepMind present a method for generating video sequences between two key frames by adapting a pretrained diffusion model to predict motion both forward and backward in time

   

8. Researchers introduce DepthCrafter - a state-of-the-art video depth estimation method capable of generating long, detailed depth sequences for open-world videos

Audio

1. Hugging Face open sources a speech-to-speech library, merging state-of-the-art models for efficient real-time voice processing with latency as low as 500 milliseconds

2. Researchers propose WavTokenizer - a codec tokenizer that achieves extreme compression and state-of-the-art audio reconstruction quality using as few as 40-75 tokens per second of 24kHz audio

   

3. Researchers openly release FluxMusic - a model that turns text into music more effectively by using powerful pre-trained language models and a novel process to capture both simple and detailed information from text

   

4. Researchers present a comprehensive review of music foundation models, identifying underexplored areas and emphasizing their potential in diverse applications like music generation and medical uses

   

Open-source Packages

1. Kotaemon - a RAG-based tool for chatting with your documents

2. KTransformers - optimized LLM inference

3. v1 - build a production-ready SaaS using the latest Next.js framework

4. StreamingT2V - turn text and images into long videos

Plus >70 more open-source packages for AI engineers

StreamingT2V outperforms proprietary providers such as Runway’s Gen-3

Last week’s AI Tidbits roundup

AI Roundup 08/15 -> 08/22/2024

Reach AI builders, researchers, and entrepreneurs by partnering with AI Tidbits

If you find AI Tidbits valuable, share it with a friend and consider showing your support.