Repository:
https://github.com/PSGBOT/pixtral-12B-Inference

本地图片上传

1
2
3
4
5
6
7
8
9
10
11

def encode_image(image_path):
    """Encode the image to base64."""
    try:
        with open(image_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode('utf-8')
    except FileNotFoundError:
        print(f"Error: The file {image_path} was not found.")
        return None
    except Exception as e:  # Added general exception handling
        print(f"Error: {e}")
        return None

Prompt

Instance Description Result

1

2

FCSGG Repository Summary

FCSGG (Fully Convolutional Scene Graph Generation) is a PyTorch implementation of the paper “Fully Convolutional Scene Graph Generation” published in CVPR 2021. The project focuses on scene graph generation, which is the task of detecting objects in an image and identifying the relationships between them.

Core Components:

1. Architecture:

   • Built on Detectron2, a popular object detection framework by Facebook
   • Uses a one-stage detector approach (CenterNet) as the meta-architecture
   • Supports various backbones including ResNet, HRNet (High-Resolution Network), Hourglass networks, and DLA

2. Key Features:

   • Fully convolutional approach to scene graph generation
   • Multiple backbone options with different feature pyramid networks (FPN, BiFPN, HRFPN)
   • Various head designs including multiscale heads and attention mechanisms
   • Support for different input resolutions and training strategies

3. Dataset:

   • Primarily designed for the Visual Genome dataset, a large-scale dataset for scene understanding
   • Includes custom data loaders and preprocessing for scene graph generation

4. Model Components:

   • Backbones: Various CNN architectures (ResNet, HRNet, Hourglass, DLA)
   • Necks: Feature pyramid networks and variants (FPN, BiFPN, HRFPN, Trident)
   • Heads: Detection and relationship prediction heads
   • Loss Functions: Custom losses for object detection and relationship prediction

5. Utilities:

   • Visualization tools for scene graphs
   • Evaluation metrics for scene graph generation
   • Training and inference scripts

Project Structure:

• fcsgg/: Main module containing model implementation

  • modeling/: Neural network architecture components
    • backbone/: Feature extraction networks
    • necks/: Feature pyramid networks
    • heads/: Detection and relationship prediction heads
    • meta_arch/: High-level model architecture (CenterNet)
  • data/: Dataset handling and preprocessing
  • evaluation/: Metrics and evaluation code
  • utils/: Helper functions and utilities
  • layers/: Custom neural network layers
  • structures/: Data structures for scene graphs

• configs/: Configuration files for different model variants

• tools/: Training, evaluation, and visualization scripts

• GraphViz/: Visualization tools for scene graphs

Key Innovations:

The project implements a fully convolutional approach to scene graph generation, which differs from traditional two-stage methods. Instead of first detecting objects and then predicting relationships, it uses a one-stage detector to simultaneously predict objects and their relationships in a fully convolutional manner.

Benchmarks:

The repository provides several pre-trained models with different backbones:

1. HRNetW32-1S
2. ResNet50-4S-FPN×2
3. HRNetW48-5S-FPN×2

These models achieve competitive performance on the Visual Genome dataset for scene graph generation tasks.

Usage:

The project provides tools for training, evaluation, and visualization of scene graphs. It requires the Visual Genome dataset and can be run using Docker or directly with PyTorch.

In summary, FCSGG is a comprehensive implementation of a state-of-the-art approach to scene graph generation using fully convolutional networks, offering various model architectures and training configurations.

How Detectron2 is Used in FCSGG

FCSGG is built on top of Detectron2, Facebook’s object detection framework, and leverages many of its components while extending it for scene graph generation. Here’s a detailed breakdown:

1. Core Architecture Integration

• Meta Architecture: FCSGG registers a custom meta architecture called “CenterNet” with Detectron2’s META_ARCH_REGISTRY. This extends Detectron2’s modular architecture system while maintaining compatibility.

• Backbone Networks: FCSGG uses Detectron2’s backbone networks (ResNet, etc.) directly and also implements custom backbones like HRNet while following Detectron2’s backbone interface.

• Feature Pyramid Networks (FPN): The repository uses Detectron2’s FPN implementation and extends it with custom variants like BiFPN and HRFPN.

2. Configuration System

• YAML Configuration: FCSGG adopts Detectron2’s YAML-based configuration system, extending it with custom configurations for scene graph generation through add_fcsgg_config().

• Command Line Arguments: The training script uses Detectron2’s default_argument_parser() to maintain the same command-line interface.

3. Data Handling

• Dataset Registration: Visual Genome dataset is registered with Detectron2’s DatasetCatalog and MetadataCatalog, making it available through Detectron2’s data loading pipeline.

• Custom Dataset Mapper: FCSGG implements a custom DatasetMapper class that extends Detectron2’s mapper to handle scene graph annotations.

• Data Loaders: The repository uses Detectron2’s build_detection_train_loader and build_detection_test_loader with custom mappers.

4. Training and Evaluation

• Trainer Class: FCSGG extends Detectron2’s DefaultTrainer class to customize the training loop, evaluation metrics, and data loading.

• Checkpointing: The repository uses Detectron2’s DetectionCheckpointer for model saving and loading.

• Distributed Training: FCSGG leverages Detectron2’s distributed training utilities through detectron2.utils.comm and the launch function.

• Custom Evaluators: The repository implements a custom VGEvaluator for scene graph evaluation while following Detectron2’s evaluator interface.

5. Visualization and Logging

• Event Storage: FCSGG uses Detectron2’s event storage system for logging metrics during training.

• Visualization Tools: The repository leverages Detectron2’s visualization utilities for debugging and result analysis.

6. Extensions for Scene Graph Generation

• Custom Heads: While using Detectron2’s architecture, FCSGG implements custom prediction heads for relationship detection.

• Scene Graph Structures: The repository defines custom data structures for scene graphs that integrate with Detectron2’s Instances class.

• Loss Functions: FCSGG implements specialized loss functions for scene graph generation while maintaining compatibility with Detectron2’s loss computation framework.

7. Installation and Dependencies

• Submodule Integration: Detectron2 is included as a Git submodule, ensuring version compatibility.

• Build Process: The installation process includes building Detectron2 from source to ensure proper integration.

In summary, FCSGG uses Detectron2 as its foundation, leveraging its modular architecture, data handling, training infrastructure, and configuration system while extending it with custom components for scene graph generation. This approach allows FCSGG to benefit from Detectron2’s robust implementation and optimizations while adding specialized functionality for relationship detection between objects.

Official repo:
https://github.com/liuhengyue/fcsgg
Our repo:
https://github.com/PSGBOT/KAF-Generation

My venv: fcsgg

Installation

Environment Preparation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

git clone git@github.com:liuhengyue/fcsgg.git
cd fcsgg
git submodule init
git submodule update
conda create --name fcsgg
conda create -n fcsgg python=3.10
conda install nvidia/label/cuda-11.8.0::cuda-toolkit -c nvidia/label/cuda-11.8.0
conda install cudatoolkit
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# for building detectron
conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/fcsgg/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/fcsgg/include/"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/fcsgg/"

conda deactivate
conda activate fcsgg

export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

pip install -r requirements.txt
python -m pip install -e detectron2

Downloads

Datasets:

1
2
3
4
5

cd ~/Reconst
wget https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip -P ./Data/vg/
wget https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip -P ./Data/vg/
unzip -j ./Data/vg/images.zip -d ./Data/vg/VG_100K
unzip -j ./Data/vg/images2.zip -d ./Data/vg/VG_100K

Download the scene graphs and extract them to datasets/vg/VG-SGG-with-attri.h5.

Issues

1

1

AttributeError: module 'PIL.Image' has no attribute 'LINEAR'. Did you mean: 'BILINEAR'?

LINEAR-> BILINEAR: commit

2

在尝试训练的过程中报错：

1
2
3

  File "/home/cyl/Reconst/fcsgg/fcsgg/data/detection_utils.py", line 432, in generate_score_map
    masked_fmap = torch.max(masked_fmap, gaussian_mask * k)
RuntimeError: The size of tensor a (55) must match the size of tensor b (56) at non-singleton dimension 1

modify detection_utils.py: commit

Training

首先更改训练的配置文件./config/quick_schedules/Quick-FCSGG-HRNet-W32.yaml, (原文件使用预训练的参数)

1
2
3
4

MODEL:
  META_ARCHITECTURE: "CenterNet"
  HRNET:
    WEIGHTS: "output/FasterR-CNN-HR32-3x.pth"

更改为train from scratch

1
2
3
4

MODEL:
  META_ARCHITECTURE: "CenterNet"
  HRNET:
    WEIGHTS: ""  # Empty string to train from scratch

再运行：

1

python tools/train_net.py --num-gpus 1 --config-file configs/quick_schedules/Quick-FCSGG-HRNet-W32.yaml

成功训练✌

1
2
3
4
5
6

...
[04/23 10:21:01] d2.utils.events INFO:  eta: 0:05:37  iter: 1159  total_loss: 1.042  loss_cls: 0.7593  loss_box_wh: 0.08827  loss_center_reg: 0.02485  loss_raf: 0.1754    time: 0.4042  last_time: 0.4519  data_time: 0.0043  last_data_time: 0.0044   lr: 0.001  max_mem: 4141M
[04/23 10:21:09] d2.utils.events INFO:  eta: 0:05:29  iter: 1179  total_loss: 1.028  loss_cls: 0.7208  loss_box_wh: 0.09246  loss_center_reg: 0.02669  loss_raf: 0.1625    time: 0.4042  last_time: 0.4035  data_time: 0.0041  last_data_time: 0.0044   lr: 0.001  max_mem: 4141M
[04/23 10:21:17] d2.utils.events INFO:  eta: 0:05:21  iter: 1199  total_loss: 1.01  loss_cls: 0.671  loss_box_wh: 0.1038  loss_center_reg: 0.02432  loss_raf: 0.1635    time: 0.4042  last_time: 0.3943  data_time: 0.0042  last_data_time: 0.0043   lr: 0.001  max_mem: 4141M
[04/23 10:21:25] d2.utils.events INFO:  eta: 0:05:13  iter: 1219  total_loss: 0.9737  loss_cls: 0.6887  loss_box_wh: 0.0929  loss_center_reg: 0.02574  loss_raf: 0.1749    time: 0.4041  last_time: 0.4101  data_time: 0.0041  last_data_time: 0.0042   lr: 0.001  max_mem: 4141M
...

Explanation

See [[FCSGG Repo Explanation]]

docs

Start

1

pip install omegaconf

1

from omegaconf import OmegaConf

Create

You can create OmegaConf objects from multiple sources.

From dict

1
2
3
4
5
6
7
8
9

conf = OmegaConf.create({"k" : "v", "list" : [1, {"a": "1", "b": "2", 3: "c"}]})
print(OmegaConf.to_yaml(conf))

k: v
list:
- 1
- a: '1'
  b: '2'
  3: c

From list

1
2
3
4
5
6
7
8

conf = OmegaConf.create([1, {"a":10, "b": {"a":10, 123: "int_key"}}])
print(OmegaConf.to_yaml(conf))

- 1
- a: 10
  b:
    a: 10
    123: int_key

From yaml

1
2
3
4
5
6
7
8
9
10
11
12

conf = OmegaConf.load('source/example.yaml')
# Output is identical to the YAML file
print(OmegaConf.to_yaml(conf))

server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2

From dot-list

1
2
3
4
5
6
7
8
9
10
11

dot_list = ["a.aa.aaa=1", "a.aa.bbb=2", "a.bb.aaa=3", "a.bb.bbb=4"]
conf = OmegaConf.from_dotlist(dot_list)
print(OmegaConf.to_yaml(conf))

a:
  aa:
    aaa: 1
    bbb: 2
  bb:
    aaa: 3
    bbb: 4

From command line arguments

1
2
3
4
5
6
7
8

sys.argv = ['your-program.py', 'server.port=82', 'log.file=log2.txt']
conf = OmegaConf.from_cli()
print(OmegaConf.to_yaml(conf))

server:
  port: 82
log:
  file: log2.txt

官方文档

LMOD软件管理

https://lmod.readthedocs.io/en/latest/010_user.html

交大hpc使用lmod来管理用户软件，每一次重新登陆都会重置module，需要将软件重新load。
常用指令：

1
2
3
4
5

ml # list these loaded modules
ml miniconda3 # load the miniconda3 module
ml -miniconda3 # unload the miniconda3 module
module spider # inspect the possible modules that can be loaded
module spider cuda # show available cuda versions

oh-my-zsh更好的shell

hpc默认已安装zsh, 通过omzsh的安装脚本进行安装，并安装插件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

export CHSH=no  # 避免它自动切换默认 shell（HPC 通常不允许）
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git \
  ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting
git clone https://github.com/zsh-users/zsh-autosuggestions \
  ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions
git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf
~/.fzf/install
git clone https://github.com/psprint/zsh-navigation-tools ~/.zsh-navigation-tools

## vim add configuration in `.zshrc`
plugins=(
  git
  zsh-navigation-tools
  rails
  ruby
  zsh-syntax-highlighting
  zsh-autosuggestions
)
source ${(q-)ZSH_CUSTOM:-$ZSH/custom}/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

## :wq

nvim 更好的编辑器

因为在服务器上没有sudo权限所以需要手动编译

首先使用conda创建编译环境

1

conda create -n nvim-build cmake libtool gettext curl unzip ninja gcc gxx -y -c conda-forge

clone & build

1
2
3
4

git clone https://github.com/neovim/neovim.git
cd neovim
git checkout stable
make CMAKE_BUILD_TYPE=Release

Tricks

交互式计算资源

某些情况下需要进行并行编译，编译需要耗费大量资源，并不一定可以在登录节点使用：

1

srun -p cpu -n 4 --pty /bin/bash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

(base) chenyulin@pilogin2 ~> srun -p cpu -n 4 --pty /bin/bash
srun: job 43090290 queued and waiting for resources
srun: job 43090290 has been allocated resources
(base) chenyulin@cas242 ~> ls
condalist.txt  dinov2  log  test.slurm
(base) chenyulin@cas242 ~> cd dinov2        
(base) chenyulin@cas242 ~/dinov2> conda activate dinov2
(dinov2) chenyulin@cas242 ~/dinov2> export CC=$CONDA_PREFIX/bin/gcc                       main
export CXX=$CONDA_PREFIX/bin/g++
(dinov2) chenyulin@cas242 ~/dinov2> which g++                                             main
~/.conda/envs/dinov2/bin/g++
(dinov2) chenyulin@cas242 ~/dinov2> vim conda_cyl.yaml                                    main
(dinov2) chenyulin@cas242 ~/dinov2> conda env update -f conda_cyl.yaml                    main
Retrieving notices: ...working... done
Channels:
 - xformers
 - pytorch
 - nvidia
 - conda-forge
 - defaults
 - nvidia/label/cuda-11.7.0
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
Installing pip dependencies: |

文件传输

本地向云端传输

使用hpc studio传输
https://studio.hpc.sjtu.edu.cn/pun/sys/dashboard/files/fs//lustre/home/acct-umjbyy/chenyulin/Data

云端之间互相传输

例如拷贝conda环境

1

cp -r /lustre/home/acct-umjbyy/chenyulin/.conda /dssg/home/acct-umjbyy/chenyulin/

Repository:

• https://github.com/facebookresearch/dinov2
• https://github.com/itsprakhar/Downstream-Dinov2

Installation

official:

1

conda env create -f conda.yaml

不建议使用official的conda.yaml, 使用更改后的conda_cyl.yaml。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1

conda install nvidia/label/cuda-11.7.0::cuda-toolkit -c nvidia/label/cuda-11.7.0
conda install cudatoolkit
conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/dinov2/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/dinov2/include/"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/dinov2/"

export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

# check with `which g++`

conda env update -f conda_cyl.yaml

pip3 install -U xformers==0.0.18

conda env config vars set PYTHONPATH="/home/cyl/Reconst/dinov2/"

Demo 🐱

官方提供了 depth estimation 和 segmentation 的 notebook，可以找时间理解一下

Train

使用的数据集为Imagenet-mini

1
2
3
4

imagenet-mini
├── labels.txt
├── train
└── val

Note: 需要额外添加一个label.txt

使用脚本生产数据集的meta data:

1

My repository: https://github.com/Chen-Yulin/Semantic-SAM
My venv: ssam

Installation

测试过的python版本：3.8,3.10
官方步骤：

1
2
3
4
5
6
7
8

pip3 install torch==1.13.1 torchvision==0.14.1 --extra-index-url https://download.pytorch.org/whl/cu113
python -m pip install 'git+https://github.com/MaureenZOU/detectron2-xyz.git'
pip install git+https://github.com/cocodataset/panopticapi.git
git clone https://github.com/UX-Decoder/Semantic-SAM
cd Semantic-SAM
python -m pip install -r requirements.txt

export DATASET=/pth/to/dataset  # path to your coco data

一些绊脚石 ^ ^

1

根据[[Cuda+Torch]]，需要先安装cudatoolkit和cuda-toolkit

1
2
3
4
5

conda install nvidia/label/cuda-11.7.0::cuda-toolkit -c nvidia/label/cuda-11.7.0 
conda install cudatoolkit # no need to specify version
conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/<name>/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/<name>/include/:/usr/include" # `/usr/include`for missing `crypt.h`
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/<name>/"

然后按照torch官网的安装指令：

1

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

2

第二行直接运行可能会报错，提示系统gcc版本过高，安装gcc=11.2.0

1
2
3
4
5
6
7
8
9
10
11
12

conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

# 指定编译器路径
export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

# 找不到crypt.h的情况
sudo pacman -S libxcrypt-compat

export CXXFLAGS="${CXXFLAGS} -fuse-ld=/usr/bin/ld"

如果编译时出现ld: cannot find -lcudart: No such file or directory collect2: error: ld returned 1 exit status 报错，只是因为没有安装cudatoolkit ^ ^

3

安装完成后直接import semantic_sam会报错ModuleNotFoundError: No module named 'MultiScaleDeformableAttention' ^ ^
提示：

1
2
3

Please compile MultiScaleDeformableAttention CUDA op with the following commands:
	`cd mask2former[/modeling/pixel_decoder/ops](http://127.0.0.1:8888/modeling/pixel_decoder/ops)`
	`sh make.sh`

需要手动make一下 Mask2Former:

1
2

cd Mask2Former/mask2former/modeling/pixel_decoder/ops/
sh make.sh

4

一些版本问题

1
2

pip install gradio==3.37.0
pip install matplotlib==3.7.0

Demo 🐱

Generate multi-granularity Mask on CLICK

1

python demo.py --ckpt ./weights/swinl_only_sam_many2many.pth

Comment: 效果相较于SAM更多体现了语义的一致性，而不是基于texture进行分割。

Automatically Generate Mask on Different Granularity

1

python demo_auto_generation.py --ckpt ./weights/swinl_only_sam_many2many.pth

需要解决的问题

同level下mask重合

Part-seg-dataset Generation

效果

原图：

Instance identification:

Part segmentation

nvidia-smi返回的是driver所能支持的最新的cuda版本
系统安装的cuda版本可以随意，torch会优先使用虚拟环境中安装的cuda版本

Conda管理Cuda

安装指定版本cuda-toolkit

1

conda install nvidia/label/cuda-12.4.0::cuda-toolkit -c nvidia/label/cuda-12.4.0  

安装最新版本

1

conda install cuda-toolkit

某些仓库需要指定cuda路径才能编译包

1
2
3

conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/gsam/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/:$LD_LIBRARY_PATH"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/gsam/lib/python3.10/site-packages/nvidia/cuda_runtime/include/:$CPATH"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/gsam/"

Note: 注意改变了库路径之后nvim中的lsp会报错，建议之后改回去

1
2
3

conda env config vars set LD_LIBRARY_PATH=""
conda env config vars set CPATH=""
conda env config vars set CUDA_HOME=""

Note: To find the correct path for CUDA_HOME use which nvcc. In my case, output of the command was:

1
2

>>> which nvcc 
/home/user/miniconda3/envs/py12/bin/nvcc

Therefore, I set the CUDA_HOME as /home/user/miniconda3/envs/py12/.

Note: To find the correct path for LD_LIBRARY_PATH use find ~ -name cuda_runtime_api.h. In my case, output of the command was:

1
2
3
4

>>> find ~ -name cuda_runtime_api.h
...
/home/user/miniconda3/envs/py12/targets/x86_64-linux/include/cuda_runtime_api.h
...

So I set the LD_LIBRARY_PATH as /home/user/miniconda3/envs/py12/targets/x86_64-linux/lib/ and CPATH as /home/user/miniconda3/envs/py12/targets/x86_64-linux/include/. If you have multiple CUDA installations, the output of find ~ -name cuda_runtime_api.h will display multiple paths. Make sure to choose the path that corresponds to the environment you have created.

ref:https://github.com/IDEA-Research/GroundingDINO/issues/355

Note: Always reboot the computer after the cuda is upgraded

Note: 在更改LD_LIBRARY_PATH后可能会导致neovim的pyright无法运行，所以建议在编译完成后设回该变量

1

conda env config vars set LD_LIBRARY_PATH=""

cudatoolkit和cuda-toolkit

这两个可以同时安装
如果不安装cudatoolkit可能会在编译时出现ld: cannot find -lcudart: No such file or directory collect2: error: ld returned 1 exit status 报错

使用以下指令获取版本信息

1

python -c 'import torch;print(torch.__version__);print(torch.version.cuda)'

1
2

2.0.0+cu117
11.7