Q&A

1

What is standard dense supervised learning? Mentioned in [[CenterNet]].

Standard dense supervised learning typically refers to a supervised learning setup where:

1. Standard supervised learning means:
   • You have input data X and corresponding ground truth labels Y.
   • The goal is to train a model $f_\theta(X)$ that maps inputs to outputs by minimizing a loss function (e.g., cross-entropy, MSE) between the predicted labels and ground truth.
   • The training dataset is fully labeled (i.e., each input has a corresponding label).
2. Dense refers to:
   • A per-pixel or per-element prediction task, where every element in the input gets a corresponding label.
   • Common in vision tasks like:
     • Semantic segmentation (each pixel is labeled with a class).
     • Depth estimation (each pixel has a depth value).
     • Optical flow (each pixel has a motion vector).
     • Surface normal estimation (each pixel has a 3D orientation vector).

In contrast to sparse supervision, where only a subset of the input (e.g., bounding boxes, keypoints) is labeled, dense supervision provides full annotations for every relevant part of the input.

Example
In semantic segmentation:

• Input: an RGB image (e.g., 512×512 pixels).
• Output: a label map of the same size (512×512), where each pixel has a class label like “road”, “car”, “sky”, etc.
• Model: often a Fully Convolutional Network (FCN) or encoder-decoder like U-Net or DeepLab.
• Loss: usually pixel-wise cross-entropy.

参考： https://www.youtube.com/watch?v=T35ba_VXkMY&t=1744s

DETR是一个使用transformer作为基本架构的 object detection 模型。

Insight

Object queries (something that can be learned):

>Visualization of all box predictions on all images from COCO 2017 val set for 20 out of total N = 100 prediction slots in DETR decoder. Each box prediction is represented as a point with the coordinates of its center in the 1-by-1 square normalized by each image size. The points are color-coded so that green color corresponds to small boxes, red to large horizontal boxes and blue to large vertical boxes. We observe that each slot learns to specialize on certain areas and box sizes with several operating modes. We note that almost all slots have a mode of predicting large image-wide boxes that are common in COCO dataset.

,

通过结合[[DINO]]和grounded-pretraining，可以使用人类输入（例如类别名称或转介表达式）检测任意对象
Open-Vocab. Det

an open-set object detector that can detect any objects with respect to an arbitrary free-form text prompt. The model was trained on over 10 million images, including detection data, visual grounding data, and image-text pairs. It has a strong zero-shot detection performance. However, the model needs text as inputs and can only detect boxes with corresponding phrases.

Grounding-DINO

Principle

Tight modality fusion based on [[DINO]]

什么是feature fusion?

- 在多模态领域，feature fusion 特指将不同模态的特征（如视觉、文本、音频等）进行融合的技术。CLIP 应该被看作是 Middle Fusion 的一种形式, 在特征提取后就进行融合对齐 #### large-scale grounded pre-train for concept generalization Reformulating **object detection** as a **phrase grounding task** and introducing **contrastive training** between object regions and language phrases on large-scale data