多模态论文串讲·下【论文精读】

TL;DR

Key insights

BLIP's Bootstrapping and Unified Framework

• Introduces a 'bootstrapping' method to clean noisy web-scraped datasets by using a captioner to generate synthetic data and a filter to remove mismatched pairs.
• Proposes a 'unified' framework that can handle both understanding tasks (like retrieval, VQA) and generation tasks (like image captioning) within a single model architecture.
• Leverages insights from previous models like ALBEF and VLMO, combining ALBEF's architecture with VLMO's flexible parameter sharing for different modalities and tasks.
• The MED (Mixture of Encoder and Decoder) architecture allows for flexible task adaptation by sharing parameters across different text encoders and decoders.

CoCa: Contrastive Captioning for Scalability

• Builds upon ALBEF, using a similar architecture but with key differences like attentional pooling and a unified decoder-only text branch for efficiency.
• Employs a dual-objective training strategy: Contrastive Loss (ITC) and Captioning Loss (Language Modeling), enabling it to perform well on both unimodal and multimodal tasks.
• Achieves state-of-the-art results across a wide range of benchmarks, including ImageNet and video action recognition, by scaling up model size and training data significantly.
• Demonstrates the effectiveness of a unified framework by achieving strong performance across diverse tasks, visualized through a compelling multi-dimensional performance plot.

BEiT-3: Unified Masked Modeling as a Universal Framework

• Proposes 'Big Convergence' by treating images as a 'foreign language' (Imagelish) and unifying all tasks under a single Masked Modeling objective.
• Utilizes a Multi-Way Transformer architecture (similar to VLMO's MOME) with shared self-attention layers and modality-specific feed-forward networks.
• Achieves state-of-the-art performance across unimodal (vision, language) and multimodal tasks, surpassing previous models like CoCa and Flamingo.
• Emphasizes the use of public datasets and a simple, scalable framework, highlighting the importance of data quality and model architecture over complex loss functions.

Evolution of Multimodal Architectures and Training Strategies

• Traces the progression from early object-detection-based vision encoders (Oscar, Uniter) to Vision Transformers (ViLT, CLIP) for simplified feature extraction.
• Highlights the development of unified frameworks like ALBEF (fusion encoder), VLMO (parameter sharing), BLIP (captioning/filtering), CoCa (contrastive captioning), and BEiT-3 (masked modeling) for broader task coverage.
• Discusses the trend towards larger models and datasets ('Big Convergence') and the increasing importance of data quality and efficient training strategies.
• Explores the concept of 'Generalist Models' and 'Language as Interface' (e.g., MetaLM, Poli) aiming for truly unified frameworks that adapt to tasks via prompts rather than architectural changes.

Key moments

• 00:00Introduction to BLIP: Discussing its contributions of 'Bootstrapping' for dataset cleaning and a 'Unified' framework for diverse multimodal tasks.
• 03:49BLIP's Captioner and Filter modules for generating synthetic data and cleaning noisy image-text pairs.
• 06:26Overview of BLIP's model architecture, drawing inspiration from ALBEF and VLMO, featuring a flexible MED (Mixture of Encoder and Decoder) structure.
• 15:59Detailed explanation of BLIP's CapFilter model and its effectiveness in improving dataset quality and model performance.
• 32:43Introduction to CoCa (Contrastive Captioning): A Google Research model building on ALBEF, achieving SOTA results through large-scale training with contrastive and captioning losses.
• 42:09Introduction to BEiT-3: A unified framework treating images as a foreign language, using Masked Modeling and a Multi-Way Transformer for broad task applicability.
• 53:33BEiT-3's flexible architecture and transfer learning capabilities, demonstrating its versatility across unimodal and multimodal tasks.