Used in [[CenterNet]]

pre: https://www.youtube.com/watch?v=HRLMSrxw2To&t=308s

解决的问题

Modeling spatial transformations is a long standing problem in computer vision

• Deformation (human pose)
• Scale
• Viewpoint variation
• Intra-class variation (不同设计的同一种物体)

Traditional approaches:

• build datasets with sufficient desired variations
• use transformation-invariant features and algorithms

架构

优势

与传统CNN拥有相同的输入输出

• regular convolution -> deformable convolution
• regular RoI pooling -> deformable RoI pooling

可以端到端训练且无需额外监督信号

直接认为是一种在物体检测方面即插即用的模块即可

Q&A

1

What is standard dense supervised learning? Mentioned in [[CenterNet]].

Standard dense supervised learning typically refers to a supervised learning setup where:

1. Standard supervised learning means:
   • You have input data X and corresponding ground truth labels Y.
   • The goal is to train a model $f_\theta(X)$ that maps inputs to outputs by minimizing a loss function (e.g., cross-entropy, MSE) between the predicted labels and ground truth.
   • The training dataset is fully labeled (i.e., each input has a corresponding label).
2. Dense refers to:
   • A per-pixel or per-element prediction task, where every element in the input gets a corresponding label.
   • Common in vision tasks like:
     • Semantic segmentation (each pixel is labeled with a class).
     • Depth estimation (each pixel has a depth value).
     • Optical flow (each pixel has a motion vector).
     • Surface normal estimation (each pixel has a 3D orientation vector).

In contrast to sparse supervision, where only a subset of the input (e.g., bounding boxes, keypoints) is labeled, dense supervision provides full annotations for every relevant part of the input.

Example
In semantic segmentation:

• Input: an RGB image (e.g., 512×512 pixels).
• Output: a label map of the same size (512×512), where each pixel has a class label like “road”, “car”, “sky”, etc.
• Model: often a Fully Convolutional Network (FCN) or encoder-decoder like U-Net or DeepLab.
• Loss: usually pixel-wise cross-entropy.

Web: https://mistral.ai/news/pixtral-12b
Demo: https://chat.mistral.ai/chat
Finetune: https://github.com/2U1/Pixtral-Finetune
Model: https://huggingface.co/mistralai/Pixtral-12B-2409

概念梳理

Scene Graph

A scene graph is a structural representation, which can capture detailed semantics by explicitly Modeling:

• objects (‘‘man’’, ‘‘fire hydrant’’, ‘‘shorts’’)
• attributes of objects (‘‘fire hydrant is yellow’’)
• relations between paired objects (‘‘man jumping over fire hydrant’’)

A scene graph is a set of visual relationship triplets in the form of <subject, relation, object> or <object, is, attribute>

Scene graphs should serve as an **objective semantic representation** of the state of the scene