Routine

爱心屋签到： aixinwu.sjtu.edu.cn/products/asw-store
每日二GRISSO💊

Starting from 2/20:

Non-routine

论文

Due

• 医保扫码补缴 3/29~3/30
• 计算机视觉hw1 4/10
• 博客三周年纪念 4/10
• 数据挖掘hw1 4/20
• 读书会吃饭 2026/3/23

No Due

• 服务器shell配置
• 请糕吃烤鱼
• Godot Learning [[Godot Learning Plan]]
• 琪琪水彩
• 流萤水彩
• 美罗城 云海肴 汽锅鸡
• 夏季expo 志愿者
• 《负负得正》
• 《涉过愤怒的海》
• 《被讨厌的勇气》读书会p16
• 《桶川跟踪狂杀人事件》读书会p17
• 《被寄生的家庭》读书会p18

有生之年系列

• 手工🎸
• 侍奉社

Archived

• AMD 2025冬令营 1/16 🔒Done! 2025-01-19 🕸️ Routine > Non-routine > Due
• 凉水彩 🔒Done! 2025-01-19 🕸️ Routine > Non-routine > Due > No Due
• wyf 生日 1/22 🔒Done! 2025-01-24 🕸️ Routine > Non-routine > Due
• JI VPN application 🔒Done! 2025-01-24 🕸️ Routine > Non-routine > Due > No Due
• 糕生日 2/11 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > Due
• WW2-Naval Mod 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
  • 潜艇
    • 深弹
    • 水箱毁伤
    • 水听
    • 视距外脱锁
    • 罗盘UI
    • 声纳
    • 发电机启停UI
    • 备用鱼雷
  • 舰载机修复
    • 水机回收
    • 机库
• 取邮政快递 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
• 《悉达多》[[《悉达多》 读书会p14]] 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
• Scene-LLm 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• 3D-LLM 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• Transformer from scratch 2/14 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > Due
• NSFC 环境重建大纲 2/18 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > Due
• 场景语义化的文献归纳 https://sjtu.feishu.cn/docx/DV4Sd0wM8oFGe9xFokJcHHJ9nlf 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > No Due
• DINO 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• PointLLM 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• ProgPrompt9 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• SayPlan 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• Grounding DINO 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• DELTA 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• MaskCLIP 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• ConceptFusion 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• 春季选课 Registration 2/28 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > Due
• 本科学分转换申请 文件 2/28 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > Due
• Holo-Arm PRP application sheet 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > No Due
• 《禅与摩托车的维修艺术》[[《禅与摩托车维修艺术》读书会p10]] 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > No Due
• 知识图谱表格细节完善 3/10 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > Due
• 褴褛飞旋电吉他 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > No Due
• DeepSeek from scratch 2/20 (deleted) 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > Due
• 场景语义理解/重建 文献综述，表格 3/13 🔒Done! 2025-03-21 🕸️ Routine > Non-routine > 论文 > Due
• 给陈天华发上海大学生机械工程创新大赛证书 3/22 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > Due
• 大众餐厅草莓 3/24 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > Due
• 《素食者》读书会p15 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > No Due

My repository: https://github.com/Chen-Yulin/Semantic-SAM

Installation

官方步骤：

1
2
3
4
5
6
7
8

pip3 install torch==1.13.1 torchvision==0.14.1 --extra-index-url https://download.pytorch.org/whl/cu113
python -m pip install 'git+https://github.com/MaureenZOU/detectron2-xyz.git'
pip install git+https://github.com/cocodataset/panopticapi.git
git clone https://github.com/UX-Decoder/Semantic-SAM
cd Semantic-SAM
python -m pip install -r requirements.txt

export DATASET=/pth/to/dataset  # path to your coco data

一些绊脚石 ^ ^

1

根据[[Cuda+Torch]]，需要先安装cudatoolkit和cuda-toolkit

1
2
3
4
5

conda install nvidia/label/cuda-11.7.0::cuda-toolkit -c nvidia/label/cuda-11.7.0 
conda install cudatoolkit
conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/Reconst/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/Reconst/include/"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/Reconstruct/"

然后按照torch官网的安装指令：

1

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

2

第二行直接运行可能会报错，提示系统gcc版本过高，安装gcc=11.2.0

1

conda install -c conda-forge gcc=11.2.0

如果编译时出现ld: cannot find -lcudart: No such file or directory collect2: error: ld returned 1 exit status 报错，只是因为没有安装cudatoolkit ^ ^

3

安装完成后直接import semantic_sam会报错ModuleNotFoundError: No module named 'MultiScaleDeformableAttention' ^ ^
提示：

1
2
3

Please compile MultiScaleDeformableAttention CUDA op with the following commands:
	`cd mask2former[/modeling/pixel_decoder/ops](http://127.0.0.1:8888/modeling/pixel_decoder/ops)`
	`sh make.sh`

需要手动make一下 Mask2Former:

1
2

cd Mask2Former/mask2former/modeling/pixel_decoder/ops/
sh make.sh

4

一些版本问题

1

pip install gradio==3.37.0

Demo 🐱

Generate multi-granularity Mask on CLICK

1

python demo.py --ckpt ./weights/swinl_only_sam_many2many.pth

Comment: 效果相较于SAM更多体现了语义的一致性，而不是基于texture进行分割。

Automatically Generate Mask on Different Granularity

1

python demo_auto_generation.py --ckpt ./weights/swinl_only_sam_many2many.pth

很重要的编码优化论文，MHE的概念：

通过LLM来判断位置关系，以此构建scene graph

还是只能判断object-level空间关系，做不了part-level manipulation

主要的思想都在上面这个伪代码里，通过只展开部分场景图（严格层级结构），来控制输入llm的场景图大小。

A scalable approach to ground LLM-based task planners across environments spanning multiple rooms and floors

Scene Graph 通过networkx (python package)表示

Insight

• 每一次场景的展开与否，该节点是否是任务关注的节点都是由LLM决定的，这一点和我的想法一致。等于是将LLM作为一个检查器一层层遍历查找任务的兴趣点。
• Scene Graph Simulator作为任务是否可行的验证器。

贡献：

• The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language, and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks.
• The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into taskrelevant objects and regions
• 基于以上，实现了一个实时的pipeline

提出了针对不同任务需要不同粒度的语义信息，本文是通过结合SAM和[[CLIP多模态预训练模型]]实现，但是忽略了物体之间的谓语关系或者父子关系。本质还是智能做导航，拾取，放下，导航的基本操作。

The architecture of RLSV is a three-layered hierarchical projection that projects a visual triple onto the attribute space, the relation space, and the visual space in order.