Track

Routine

爱心屋签到： aixinwu.sjtu.edu.cn/products/asw-store
每日二GRISSO💊

Starting from 2/20:

Non-routine

论文

Due

• 实验室网站调研
• connected paper调研
• 夏季选课 (10:00 on April 25 to 13:00 on April 28)
• 桶川读书会 4/26 19:00
• 筹备老妈生日 4/27
• 千岛湖 4/27
  • 记得带便携烤架
  • 攻略
  • 酒店
  • 车票
• 数据挖掘hw2 5/5
• 计算机视觉hw2 5/8
• 自然辩证法考试 5/9 18:00
• AR论文修改 5/10
• draumurvakna.github.io 一周年纪念日 2026/10/27
• 读书会吃饭 2026/3/23

No Due

• 老妈生日礼物：妙界按摩仪
• 穹彻智能
• 想游泳…
• 请糕吃烤鱼
• Godot Learning [[Godot Learning Plan]]
• 琪琪水彩
• 流萤水彩
• 美罗城 云海肴 汽锅鸡
• 夏季expo 志愿者
• 《负负得正》
• 《涉过愤怒的海》
• 《被讨厌的勇气》读书会p16
• 《桶川跟踪狂杀人事件》读书会p17
• 《被寄生的家庭》读书会p18
• 《断舍离》读书会p20
• 迪士尼
• 想买…
  • ipad（matebook e go）
  • 欧珑无极乌龙
  • bose大鲨2耳塞套
  • 综合效果器 ..?
  • 清音音箱

有生之年系列

• 手工🎸
• 侍奉社

Archived

• AMD 2025冬令营 1/16 🔒Done! 2025-01-19 🕸️ Routine > Non-routine > Due
• 凉水彩 🔒Done! 2025-01-19 🕸️ Routine > Non-routine > Due > No Due
• wyf 生日 1/22 🔒Done! 2025-01-24 🕸️ Routine > Non-routine > Due
• JI VPN application 🔒Done! 2025-01-24 🕸️ Routine > Non-routine > Due > No Due
• 糕生日 2/11 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > Due
• WW2-Naval Mod 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
  • 潜艇
    • 深弹
    • 水箱毁伤
    • 水听
    • 视距外脱锁
    • 罗盘UI
    • 声纳
    • 发电机启停UI
    • 备用鱼雷
  • 舰载机修复
    • 水机回收
    • 机库
• 取邮政快递 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
• 《悉达多》[[《悉达多》 读书会p14]] 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
• Scene-LLm 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• 3D-LLM 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• Transformer from scratch 2/14 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > Due
• NSFC 环境重建大纲 2/18 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > Due
• 场景语义化的文献归纳 https://sjtu.feishu.cn/docx/DV4Sd0wM8oFGe9xFokJcHHJ9nlf 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > No Due
• DINO 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• PointLLM 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• ProgPrompt9 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• SayPlan 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• Grounding DINO 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• DELTA 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• MaskCLIP 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• ConceptFusion 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• 春季选课 Registration 2/28 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > Due
• 本科学分转换申请 文件 2/28 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > Due
• Holo-Arm PRP application sheet 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > No Due
• 《禅与摩托车的维修艺术》[[《禅与摩托车维修艺术》读书会p10]] 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > No Due
• 知识图谱表格细节完善 3/10 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > Due
• 褴褛飞旋电吉他 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > No Due
• DeepSeek from scratch 2/20 (deleted) 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > Due
• 场景语义理解/重建 文献综述，表格 3/13 🔒Done! 2025-03-21 🕸️ Routine > Non-routine > 论文 > Due
• 给陈天华发上海大学生机械工程创新大赛证书 3/22 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > Due
• 大众餐厅草莓 3/24 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > Due
• 《素食者》读书会p15 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > No Due
• 医保扫码补缴 3/29~3/30 🔒Done! 2025-03-31 🕸️ Routine > Non-routine > 论文 > Due
• PRP Application 3/30 🔒Done! 2025-03-31 🕸️ Routine > Non-routine > 论文 > Due
• 自然辩证法pre 4/2 🔒Done! 2025-04-03 🕸️ Routine > Non-routine > 论文 > Due
• 服务器shell配置 🔒Done! 2025-04-03 🕸️ Routine > Non-routine > 论文 > No Due
• 计算机视觉hw1 4/10 🔒Done! 2025-04-07 🕸️ Non-routine > 论文 > Due
• 数据挖掘pre registration 4/8 🔒Done! 2025-04-08 🕸️ Routine > Non-routine > 论文 > Due
• 加入雪糕的steam家庭成员 🔒Done! 2025-04-09 🕸️ Routine > Non-routine > 论文 > Due
• 确定490最终人选 4/10 (顾，孙一豪，安瑞童) 🔒Done! 2025-04-12 🕸️ Track > Routine > Non-routine > Due
• 数据挖掘pre 4/10 🔒Done! 2025-04-12 🕸️ Track > Routine > Non-routine > Due
• 博客三周年纪念 4/10 🔒Done! 2025-04-12 🕸️ Track > Routine > Non-routine > Due
• 4903 kickoff 信息通知 🔒Done! 2025-04-15 🕸️ Track > Routine > Non-routine > Due
• 《简单的激情》读书会p19 🔒Done! 2025-04-15 🕸️ Track > Routine > Non-routine > No Due
• 俞国雄 🔒Done! 2025-04-15 🕸️ Track
• meet 毕 11:00 4/17 🔒Done! 2025-04-17 🕸️ Track > Routine > Non-routine > Due
• 490 kickoff meeting 4/17 🔒Done! 2025-04-18 🕸️ Track > Routine > Non-routine > Due
  • background
  • 分工
    • 核心模块复现
    • 文献综述，针对要解决的问题(已经做了什么工作，我们怎么解决,inspiration)，针对大背景(Introduction)
      • 限定范围
    • 数据准备
• 中午羽毛球 4/18 🔒Done! 2025-04-18 🕸️ Track > Routine > Non-routine > Due
• meet wyf 4/19 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• CV meeting 4/19 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• 计算机视觉 project proposal 4/24 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• 数据挖掘hw1 4/20 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• 490 论文下发 ed.1 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due

FCSGG Repository Summary

FCSGG (Fully Convolutional Scene Graph Generation) is a PyTorch implementation of the paper “Fully Convolutional Scene Graph Generation” published in CVPR 2021. The project focuses on scene graph generation, which is the task of detecting objects in an image and identifying the relationships between them.

Core Components:

1. Architecture:

   • Built on Detectron2, a popular object detection framework by Facebook
   • Uses a one-stage detector approach (CenterNet) as the meta-architecture
   • Supports various backbones including ResNet, HRNet (High-Resolution Network), Hourglass networks, and DLA

2. Key Features:

   • Fully convolutional approach to scene graph generation
   • Multiple backbone options with different feature pyramid networks (FPN, BiFPN, HRFPN)
   • Various head designs including multiscale heads and attention mechanisms
   • Support for different input resolutions and training strategies

3. Dataset:

   • Primarily designed for the Visual Genome dataset, a large-scale dataset for scene understanding
   • Includes custom data loaders and preprocessing for scene graph generation

4. Model Components:

   • Backbones: Various CNN architectures (ResNet, HRNet, Hourglass, DLA)
   • Necks: Feature pyramid networks and variants (FPN, BiFPN, HRFPN, Trident)
   • Heads: Detection and relationship prediction heads
   • Loss Functions: Custom losses for object detection and relationship prediction

5. Utilities:

   • Visualization tools for scene graphs
   • Evaluation metrics for scene graph generation
   • Training and inference scripts

Project Structure:

• fcsgg/: Main module containing model implementation

  • modeling/: Neural network architecture components
    • backbone/: Feature extraction networks
    • necks/: Feature pyramid networks
    • heads/: Detection and relationship prediction heads
    • meta_arch/: High-level model architecture (CenterNet)
  • data/: Dataset handling and preprocessing
  • evaluation/: Metrics and evaluation code
  • utils/: Helper functions and utilities
  • layers/: Custom neural network layers
  • structures/: Data structures for scene graphs

• configs/: Configuration files for different model variants

• tools/: Training, evaluation, and visualization scripts

• GraphViz/: Visualization tools for scene graphs

Key Innovations:

The project implements a fully convolutional approach to scene graph generation, which differs from traditional two-stage methods. Instead of first detecting objects and then predicting relationships, it uses a one-stage detector to simultaneously predict objects and their relationships in a fully convolutional manner.

Benchmarks:

The repository provides several pre-trained models with different backbones:

1. HRNetW32-1S
2. ResNet50-4S-FPN×2
3. HRNetW48-5S-FPN×2

These models achieve competitive performance on the Visual Genome dataset for scene graph generation tasks.

Usage:

The project provides tools for training, evaluation, and visualization of scene graphs. It requires the Visual Genome dataset and can be run using Docker or directly with PyTorch.

In summary, FCSGG is a comprehensive implementation of a state-of-the-art approach to scene graph generation using fully convolutional networks, offering various model architectures and training configurations.

How Detectron2 is Used in FCSGG

FCSGG is built on top of Detectron2, Facebook’s object detection framework, and leverages many of its components while extending it for scene graph generation. Here’s a detailed breakdown:

1. Core Architecture Integration

• Meta Architecture: FCSGG registers a custom meta architecture called “CenterNet” with Detectron2’s META_ARCH_REGISTRY. This extends Detectron2’s modular architecture system while maintaining compatibility.

• Backbone Networks: FCSGG uses Detectron2’s backbone networks (ResNet, etc.) directly and also implements custom backbones like HRNet while following Detectron2’s backbone interface.

• Feature Pyramid Networks (FPN): The repository uses Detectron2’s FPN implementation and extends it with custom variants like BiFPN and HRFPN.

2. Configuration System

• YAML Configuration: FCSGG adopts Detectron2’s YAML-based configuration system, extending it with custom configurations for scene graph generation through add_fcsgg_config().

• Command Line Arguments: The training script uses Detectron2’s default_argument_parser() to maintain the same command-line interface.

3. Data Handling

• Dataset Registration: Visual Genome dataset is registered with Detectron2’s DatasetCatalog and MetadataCatalog, making it available through Detectron2’s data loading pipeline.

• Custom Dataset Mapper: FCSGG implements a custom DatasetMapper class that extends Detectron2’s mapper to handle scene graph annotations.

• Data Loaders: The repository uses Detectron2’s build_detection_train_loader and build_detection_test_loader with custom mappers.

4. Training and Evaluation

• Trainer Class: FCSGG extends Detectron2’s DefaultTrainer class to customize the training loop, evaluation metrics, and data loading.

• Checkpointing: The repository uses Detectron2’s DetectionCheckpointer for model saving and loading.

• Distributed Training: FCSGG leverages Detectron2’s distributed training utilities through detectron2.utils.comm and the launch function.

• Custom Evaluators: The repository implements a custom VGEvaluator for scene graph evaluation while following Detectron2’s evaluator interface.

5. Visualization and Logging

• Event Storage: FCSGG uses Detectron2’s event storage system for logging metrics during training.

• Visualization Tools: The repository leverages Detectron2’s visualization utilities for debugging and result analysis.

6. Extensions for Scene Graph Generation

• Custom Heads: While using Detectron2’s architecture, FCSGG implements custom prediction heads for relationship detection.

• Scene Graph Structures: The repository defines custom data structures for scene graphs that integrate with Detectron2’s Instances class.

• Loss Functions: FCSGG implements specialized loss functions for scene graph generation while maintaining compatibility with Detectron2’s loss computation framework.

7. Installation and Dependencies

• Submodule Integration: Detectron2 is included as a Git submodule, ensuring version compatibility.

• Build Process: The installation process includes building Detectron2 from source to ensure proper integration.

In summary, FCSGG uses Detectron2 as its foundation, leveraging its modular architecture, data handling, training infrastructure, and configuration system while extending it with custom components for scene graph generation. This approach allows FCSGG to benefit from Detectron2’s robust implementation and optimizations while adding specialized functionality for relationship detection between objects.

Official repo:
https://github.com/liuhengyue/fcsgg
Our repo:
https://github.com/PSGBOT/KAF-Generation

Installation

Environment Preparation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

git clone git@github.com:liuhengyue/fcsgg.git
cd fcsgg
git submodule init
git submodule update
conda create --name fcsgg
conda create -n fcsgg python=3.10
conda install nvidia/label/cuda-11.8.0::cuda-toolkit -c nvidia/label/cuda-11.8.0
conda install cudatoolkit
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# for building detectron
conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/fcsgg/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/fcsgg/include/"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/fcsgg/"

conda deactivate
conda activate fcsgg

export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

pip install -r requirements.txt
python -m pip install -e detectron2

Downloads

Datasets:

1
2
3
4
5

cd ~/Reconst
wget https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip -P ./Data/vg/
wget https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip -P ./Data/vg/
unzip -j ./Data/vg/images.zip -d ./Data/vg/VG_100K
unzip -j ./Data/vg/images2.zip -d ./Data/vg/VG_100K

Download the scene graphs and extract them to datasets/vg/VG-SGG-with-attri.h5.

Issues

1

1

AttributeError: module 'PIL.Image' has no attribute 'LINEAR'. Did you mean: 'BILINEAR'?

LINEAR-> BILINEAR: commit

2

在尝试训练的过程中报错：

1
2
3

  File "/home/cyl/Reconst/fcsgg/fcsgg/data/detection_utils.py", line 432, in generate_score_map
    masked_fmap = torch.max(masked_fmap, gaussian_mask * k)
RuntimeError: The size of tensor a (55) must match the size of tensor b (56) at non-singleton dimension 1

modify detection_utils.py: commit

Training

首先更改训练的配置文件./config/quick_schedules/Quick-FCSGG-HRNet-W32.yaml, (原文件使用预训练的参数)

1
2
3
4

MODEL:
  META_ARCHITECTURE: "CenterNet"
  HRNET:
    WEIGHTS: "output/FasterR-CNN-HR32-3x.pth"

更改为train from scratch

1
2
3
4

MODEL:
  META_ARCHITECTURE: "CenterNet"
  HRNET:
    WEIGHTS: ""  # Empty string to train from scratch

再运行：

1

python tools/train_net.py --num-gpus 1 --config-file configs/quick_schedules/Quick-FCSGG-HRNet-W32.yaml

成功训练✌

1
2
3
4
5
6

...
[04/23 10:21:01] d2.utils.events INFO:  eta: 0:05:37  iter: 1159  total_loss: 1.042  loss_cls: 0.7593  loss_box_wh: 0.08827  loss_center_reg: 0.02485  loss_raf: 0.1754    time: 0.4042  last_time: 0.4519  data_time: 0.0043  last_data_time: 0.0044   lr: 0.001  max_mem: 4141M
[04/23 10:21:09] d2.utils.events INFO:  eta: 0:05:29  iter: 1179  total_loss: 1.028  loss_cls: 0.7208  loss_box_wh: 0.09246  loss_center_reg: 0.02669  loss_raf: 0.1625    time: 0.4042  last_time: 0.4035  data_time: 0.0041  last_data_time: 0.0044   lr: 0.001  max_mem: 4141M
[04/23 10:21:17] d2.utils.events INFO:  eta: 0:05:21  iter: 1199  total_loss: 1.01  loss_cls: 0.671  loss_box_wh: 0.1038  loss_center_reg: 0.02432  loss_raf: 0.1635    time: 0.4042  last_time: 0.3943  data_time: 0.0042  last_data_time: 0.0043   lr: 0.001  max_mem: 4141M
[04/23 10:21:25] d2.utils.events INFO:  eta: 0:05:13  iter: 1219  total_loss: 0.9737  loss_cls: 0.6887  loss_box_wh: 0.0929  loss_center_reg: 0.02574  loss_raf: 0.1749    time: 0.4041  last_time: 0.4101  data_time: 0.0041  last_data_time: 0.0042   lr: 0.001  max_mem: 4141M
...

Explanation