概念梳理

Scene Graph

A scene graph is a structural representation, which can capture detailed semantics by explicitly Modeling:

• objects (‘‘man’’, ‘‘fire hydrant’’, ‘‘shorts’’)
• attributes of objects (‘‘fire hydrant is yellow’’)
• relations between paired objects (‘‘man jumping over fire hydrant’’)

A scene graph is a set of visual relationship triplets in the form of <subject, relation, object> or <object, is, attribute>

Scene graphs should serve as an **objective semantic representation** of the state of the scene

The architecture of RLSV is a three-layered hierarchical projection that projects a visual triple onto the attribute space, the relation space, and the visual space in order.

The **Union Visual Translation Embedding network (UVTransE)**, which learns three projection matrices $W_{s}$, $W_{o}$, $W_{u}$ which map the respective feature vectors of the bounding boxes enclosing the subject, object, and union of subject and object into a common embedding space, as well as translation vectors $t_{p}$ (to be consistent with [[(VtransE) Visual Translation Embedding Network for Visual Relation Detection]]) in the same space corresponding to each of the predicate labels that are present in the dataset.

VTransE

The **extensibility** and **inference speed** of a SGG framework is crucial for accelerating down-stream tasks. This paper studied the efficiency and scalability in SGG ## Insights 最大的亮点是将相似的相互作用区域的对象对聚集到子图中并共享短语表示（称为子图特征），然后再在子图上refine

我的想法是将场景进行panoptic segmentation 之后再在每个物体上进行hierarchical part relation detection，异曲同工。

RelTR是自下而上的方法, 使用基于Transformer的 object detector（例如DETR）生成对象候选者，然后使用relation transformer来预测object pairs之间的关​​系。它还设计了一种基于积分的关系表示方法，该方法将关系编码为二维矢量场。

One stage relation detection (detects objects and relations simultaneously).

这个模型受启发于 [[CenterNet]] 和 [[OpenPose Using Part Affinity Fields]]，通过添加一个新的用于生成RAF的卷积头来获取物体之间的关系。

具体RAF是如何定义的，以及如何训练(Loss)：

Results