通过LLM来判断位置关系，以此构建scene graph

还是只能判断object-level空间关系，做不了part-level manipulation

主要的思想都在上面这个伪代码里，通过只展开部分场景图（严格层级结构），来控制输入llm的场景图大小。

A scalable approach to ground LLM-based task planners across environments spanning multiple rooms and floors

Scene Graph 通过networkx (python package)表示

Insight

• 每一次场景的展开与否，该节点是否是任务关注的节点都是由LLM决定的，这一点和我的想法一致。等于是将LLM作为一个检查器一层层遍历查找任务的兴趣点。
• Scene Graph Simulator作为任务是否可行的验证器。

贡献：

• The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language, and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks.
• The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into taskrelevant objects and regions
• 基于以上，实现了一个实时的pipeline

提出了针对不同任务需要不同粒度的语义信息，本文是通过结合SAM和[[CLIP多模态预训练模型]]实现，但是忽略了物体之间的谓语关系或者父子关系。本质还是智能做导航，拾取，放下，导航的基本操作。

The architecture of RLSV is a three-layered hierarchical projection that projects a visual triple onto the attribute space, the relation space, and the visual space in order.

The **Union Visual Translation Embedding network (UVTransE)**, which learns three projection matrices $W_{s}$, $W_{o}$, $W_{u}$ which map the respective feature vectors of the bounding boxes enclosing the subject, object, and union of subject and object into a common embedding space, as well as translation vectors $t_{p}$ (to be consistent with [[(VtransE) Visual Translation Embedding Network for Visual Relation Detection]]) in the same space corresponding to each of the predicate labels that are present in the dataset.

VTransE

The **extensibility** and **inference speed** of a SGG framework is crucial for accelerating down-stream tasks. This paper studied the efficiency and scalability in SGG ## Insights 最大的亮点是将相似的相互作用区域的对象对聚集到子图中并共享短语表示（称为子图特征），然后再在子图上refine

我的想法是将场景进行panoptic segmentation 之后再在每个物体上进行hierarchical part relation detection，异曲同工。

RelTR是自下而上的方法, 使用基于Transformer的 object detector（例如DETR）生成对象候选者，然后使用relation transformer来预测object pairs之间的关​​系。它还设计了一种基于积分的关系表示方法，该方法将关系编码为二维矢量场。