
GLIP是一个学习了object-level, language-aware, and semantic-rich visual representations 的模型。
统一对象检测和短语接地进行预训练。
什么是 phrase grounding:
Phrase Grounding refers to the task of associating or “grounding” a natural language phrase (like a sentence or a word) to a specific region or object in an image. In other words, it’s about finding which part of the image corresponds to the object or concept described by a given text phrase.
## Grounded Language Image Pre-training 将经典对象检测任务投入到grounding问题中,并提出**Unified Formulation**For instance, if you have the phrase “the red ball on the table” and an image of a room with a red ball placed on a table, the goal of phrase grounding is to identify the exact region in the image that corresponds to the “red ball on the table”, distinguishing it from other objects in the image.
传统的物体检测方法会把每个region分类进c个classes,而本文使用的Object detection as phrase grounding.
我们通过将每个区域与文本提示中的c(class)短语进行接地/对齐,将检测重新制定为基础任务
the classification prompt “person. bicycle. car. … . toothbrush”
将不同帧$X_t$中的特征集合在M
中特征点的公式:
,
通过结合[[DINO]]和grounded-pretraining,可以使用人类输入(例如类别名称或转介表达式)检测任意对象
Open-Vocab. Det
an open-set object detector that can detect any objects with respect to an arbitrary free-form text prompt. The model was trained on over 10 million images, including detection data, visual grounding data, and image-text pairs. It has a strong zero-shot detection performance. However, the model needs text as inputs and can only detect boxes with corresponding phrases.
什么是feature fusion?
- 在多模态领域,feature fusion 特指将不同模态的特征(如视觉、文本、音频等)进行融合的技术。CLIP 应该被看作是 Middle Fusion 的一种形式, 在特征提取后就进行融合对齐 #### large-scale grounded pre-train for concept generalization Reformulating **object detection** as a **phrase grounding task** and introducing **contrastive training** between object regions and language phrases on large-scale datahttps://github.com/IDEA-Research/Grounded-Segment-Anything
By [[Grounding-DINO]] + SAM
Achieving Open-Vocab. Det & Seg
本文提出的模型主要想解决3D密集标注和交互式规划。
结合
需要align the dense 3D visual information with the textual embedding space of a pre-trained LLM。3D点集由于其连续坐标系以及需要适应场景状态变化的表示形式而构成了一个独特的问题
3D-VQA
VLN(Visual-Language Navigation)
和[[3D-LLM]]一样,都是多视角采集D-RGB信息然后整合为3D frame
标注信息来自于Mini-GPT-V2(capable of generating captions and object descriptions from images by using caption and grounded caption identifiers)。
Uses image frames and a 2D-VLM(Mini-GPT-V2) to generate frame descriptions
3D场景数据是通过基于其相机姿势汇总的3D帧来重建
使用Llama-2-Chat-70B [65]生成场景的语言注释
From Vision Studio 对于VLM生成内容使用的self-checking: [83]prompted with a mix of context data including generated frame captions, frame object descriptions, annotated object lists, and annotated bounding boxes. These prompts lead to diverse instruction-following data types like dense caption, object caption, task decomposition, functionality enhancement, question-answering, and human-robot dialogues
场景-LLM是一种3D视觉语言模型(VLM),具有简单而有效的体系结构,旨在理解以基于本体和场景级别的3D视觉信息,使其能够成功执行交互式计划任务。本节概述了3D视觉特征提取过程,我们的模型的体系结构,3D视觉信息与数据集的对齐以及使用Scene-LLM进行推理。
Employ visual language semantic features [51] to represent 3D visual semantics
Tokenize 3D visual features for LLM input:
To bridge 3D visual tokens(F) with the LLM’s tokenized space
FC(1030, 768)->GELU->FC(768,768)
Llama-2-7b as the foundational LLM backbone
在两个坐标系统(camera和世界坐标)下使用3D帧数据,以确保场景-LLM理解以自我为中心和以场景为中心的观点。
在此阶段,仅训练了projection layer,可以有效地对齐具有文本特征的3D视觉特征,同时保持LLM参数(φ)不变。
优化Scene-llm,以准确响应用户说明。我们使用标识符令牌“我看到”将3D帧语言和3D场景语言数据合并到前言。文本描述分为指令($T_{INST}$)及其相应的响应($T_{ANS}$)。利用转换后的3D视觉令牌($T_{3D}$)和指令令牌($T_{INST}$),我们的目标是微调LLM(φ)以自动生成$T_{ANS}$.
在这里,我们共同微调了投影层和LLM,由θ= {ψ,φ}表示
Recent works have explored aligning images and videos with LLM for a new generation of multi-modal LLMs that equip LLMs with the ability to understand and reason about 2D images.
但是仍缺少对于3D物理空间进行分析的模型, which involves richer concepts such as spatial relationships, affordances, physics and interaction so on.
由此提出了inject the 3D world into large language models
, 介绍一个全新的3D-llm模型族,可以将3D表示(即带有功能的3D点云)作为输入,并执行一系列与3D相关的任务。
优势:
挑战