LifeGame Python
Posted Updated Note11 minutes read (About 1667 words)

LifeGame Python

使用 Python 和 Pygame 实现生命游戏

0. 教学目标

通过本实验，学习：

  • 使用 二维数组表示离散网格世界

  • 理解并实现 生命游戏的局部规则 → 全局演化

  • 使用 Pygame 进行基础的二维图形显示

  • 使用 键盘事件控制程序行为

  • 将一个问题分解为：
    数据结构 → 规则函数 → 显示函数 → 主循环

1. 什么是生命游戏（Conway’s Game of Life）

生命游戏是一个离散、确定性、无随机的元胞自动机：

  • 空间：二维网格

  • 时间：离散时间步

  • 每个格子（细胞）只有两种状态：

    • 活 (1)

    • 死 (0)

1.1 演化规则（只依赖局部）

对每个细胞，统计它周围 8 个邻居 中活细胞的数量：

当前状态 活邻居数 下一状态
活 2 或 3 活
活 其他 死
死 3 活
死 其他 死

重要思想：

全局复杂行为来自简单的局部规则

2. 程序整体结构设计

在开始写代码前，先规划模块结构：

1
2
3
4
5
6
7
8
生命游戏程序
│
├── 参数设置（网格大小、颜色）
├── 二维数组表示网格
├── 邻居统计函数
├── 下一代计算函数
├── Pygame 绘制函数
└── 主循环（事件处理 + 显示）

3. 第一步：准备环境和基本参数

3.1 安装并导入库

1
pip install pygame
1
2
import pygame
import random

3.2 定义网格和显示参数

1
2
3
4
5
6
CELL_SIZE = 10          # 每个细胞在屏幕上的像素大小
GRID_WIDTH = 80         # 网格列数
GRID_HEIGHT = 60        # 网格行数

SCREEN_WIDTH = GRID_WIDTH * CELL_SIZE
SCREEN_HEIGHT = GRID_HEIGHT * CELL_SIZE

解释：

  • 我们的“世界”是逻辑网格

  • Pygame 显示的是像素

  • 一个细胞 ↔ 一个矩形

3.3 定义细胞状态和颜色

1
2
3
4
5
6
ALIVE = 1
DEAD = 0

COLOR_BG = (0, 0, 0)
COLOR_CELL = (255, 255, 255)
COLOR_GRID = (40, 40, 40)

4. 第二步：用二维数组表示生命世界

4.1 二维数组的含义

1
grid[y][x]

  • y：第几行

  • x：第几列

  • 值：0 或 1

4.2 随机初始化网格

1
2
3
4
5
6
7
8
9
10
11
def random_grid():
    grid = []  # 用来存放整个二维网格（每一行是一个列表）

    for y in range(GRID_HEIGHT):
        row = []  # 当前行
        for x in range(GRID_WIDTH):
            cell = random.choice([ALIVE, DEAD])
            row.append(cell)
        grid.append(row)

    return grid

更优雅的写法：

1
2
3
4
5
def random_grid():
    return [
        [random.choice([ALIVE, DEAD]) for _ in range(GRID_WIDTH)]
        for _ in range(GRID_HEIGHT)
    ]

教学要点：

  • 外层列表 → 行

  • 内层列表 → 列

  • 每个位置随机赋值

5. 第三步：统计邻居数量（核心逻辑之一）

5.1 邻域定义

每个细胞有 8 个邻居：

1
2
3
(-1,-1) (0,-1) (1,-1)
(-1, 0)   X    (1, 0)
(-1, 1) (0, 1) (1, 1)

5.2 邻居统计函数

1
2
3
4
5
6
7
8
9
10
def count_neighbors(grid, x, y):
    count = 0
    for dy in [-1, 0, 1]:
        for dx in [-1, 0, 1]:
            if dx == 0 and dy == 0:
                continue
            nx, ny = x + dx, y + dy
            if 0 <= nx < GRID_WIDTH and 0 <= ny < GRID_HEIGHT:
                count += grid[ny][nx]
    return count

逐行解释：

  • 两层循环遍历 3×3 区域

  • 排除自身

  • 检查是否越界

  • 累加存活细胞数

6. 第四步：计算下一代网格

6.1 为什么不能原地修改？

  • 当前状态 → 决定下一状态

  • 如果边算边改，会影响后续计算

解决方案：

使用一个新的二维数组

6.2 下一代计算函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def next_generation(grid):
    new_grid = [
        [DEAD for _ in range(GRID_WIDTH)]
        for _ in range(GRID_HEIGHT)
    ]

    for y in range(GRID_HEIGHT):
        for x in range(GRID_WIDTH):
            neighbors = count_neighbors(grid, x, y)
            if grid[y][x] == ALIVE:
                if neighbors in (2, 3):
                    new_grid[y][x] = ALIVE
            else:
                if neighbors == 3:
                    new_grid[y][x] = ALIVE

    return new_grid

教学重点：

  • 规则直接翻译成代码

  • 清晰的 if / else 逻辑

  • 新旧网格分离

7. 第五步：使用 Pygame 显示网格

7.1 显示原理

  • 一个细胞 → 一个矩形

  • 坐标换算：

1
2
屏幕 x = 网格列 * CELL_SIZE
屏幕 y = 网格行 * CELL_SIZE

7.2 绘制函数

1
2
3
4
5
6
7
8
9
10
11
12
13
def draw_grid(screen, grid):
    screen.fill(COLOR_BG)

    for y in range(GRID_HEIGHT):
        for x in range(GRID_WIDTH):
            if grid[y][x] == ALIVE:
                rect = pygame.Rect(
                    x * CELL_SIZE,
                    y * CELL_SIZE,
                    CELL_SIZE,
                    CELL_SIZE
                )
                pygame.draw.rect(screen, COLOR_CELL, rect)

7.3 （可选）绘制网格线

1
2
3
4
5
6
7
8
9
10
11
12
13
for x in range(GRID_WIDTH):
    pygame.draw.line(
        screen, COLOR_GRID,
        (x * CELL_SIZE, 0),
        (x * CELL_SIZE, SCREEN_HEIGHT)
    )

for y in range(GRID_HEIGHT):
    pygame.draw.line(
        screen, COLOR_GRID,
        (0, y * CELL_SIZE),
        (SCREEN_WIDTH, y * CELL_SIZE)
    )

8. 第六步：主循环与键盘控制

8.1 Pygame 程序基本结构

1
2
3
pygame.init()
screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
clock = pygame.time.Clock()

8.2 事件处理与主循环

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
grid = random_grid()
running = True

while running:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False

        elif event.type == pygame.KEYDOWN:
            if event.key == pygame.K_SPACE:
                grid = next_generation(grid)
            elif event.key == pygame.K_r:
                grid = random_grid()
            elif event.key == pygame.K_ESCAPE:
                running = False

    draw_grid(screen, grid)
    pygame.display.flip()
    clock.tick(60)

教学重点：

  • SPACE：推进一代

  • R：重新初始化

  • ESC：退出

9. 总结：完整程序的思想

  • 二维数组 = 世界状态

  • 局部规则 → 全局演化

  • 显示逻辑与计算逻辑分离

  • 主循环 = 事件 + 更新 + 绘制

完整程序：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
import pygame
import random

# ==================================================
# 1. 参数设置
# ==================================================
CELL_SIZE = 10          # 单个细胞显示的像素大小
GRID_WIDTH = 80         # 网格列数
GRID_HEIGHT = 60        # 网格行数

SCREEN_WIDTH = GRID_WIDTH * CELL_SIZE
SCREEN_HEIGHT = GRID_HEIGHT * CELL_SIZE

ALIVE = 1
DEAD = 0

# 颜色定义
COLOR_BG = (0, 0, 0)
COLOR_CELL = (255, 255, 255)
COLOR_GRID = (40, 40, 40)


# ==================================================
# 2. 网格初始化（双层 for 循环版）
# ==================================================
def random_grid():
    grid = []

    for y in range(GRID_HEIGHT):
        row = []
        for x in range(GRID_WIDTH):
            cell = random.choice([ALIVE, DEAD])
            row.append(cell)
        grid.append(row)

    return grid


# ==================================================
# 3. 邻居数量统计
# ==================================================
def count_neighbors(grid, x, y):
    count = 0

    for dy in [-1, 0, 1]:
        for dx in [-1, 0, 1]:
            # 跳过自身
            if dx == 0 and dy == 0:
                continue

            nx = x + dx
            ny = y + dy

            # 边界检查
            if 0 <= nx < GRID_WIDTH and 0 <= ny < GRID_HEIGHT:
                count += grid[ny][nx]

    return count


# ==================================================
# 4. 计算下一代
# ==================================================
def next_generation(grid):
    new_grid = []

    for y in range(GRID_HEIGHT):
        new_row = []
        for x in range(GRID_WIDTH):
            neighbors = count_neighbors(grid, x, y)

            if grid[y][x] == ALIVE:
                if neighbors == 2 or neighbors == 3:
                    new_row.append(ALIVE)
                else:
                    new_row.append(DEAD)
            else:
                if neighbors == 3:
                    new_row.append(ALIVE)
                else:
                    new_row.append(DEAD)

        new_grid.append(new_row)

    return new_grid


# ==================================================
# 5. 绘制网格
# ==================================================
def draw_grid(screen, grid):
    screen.fill(COLOR_BG)

    # 绘制细胞
    for y in range(GRID_HEIGHT):
        for x in range(GRID_WIDTH):
            if grid[y][x] == ALIVE:
                rect = pygame.Rect(
                    x * CELL_SIZE,
                    y * CELL_SIZE,
                    CELL_SIZE,
                    CELL_SIZE
                )
                pygame.draw.rect(screen, COLOR_CELL, rect)

    # 绘制网格线（教学可选）
    for x in range(GRID_WIDTH):
        pygame.draw.line(
            screen,
            COLOR_GRID,
            (x * CELL_SIZE, 0),
            (x * CELL_SIZE, SCREEN_HEIGHT)
        )

    for y in range(GRID_HEIGHT):
        pygame.draw.line(
            screen,
            COLOR_GRID,
            (0, y * CELL_SIZE),
            (SCREEN_WIDTH, y * CELL_SIZE)
        )


# ==================================================
# 6. 主程序
# ==================================================
def main():
    pygame.init()
    screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))
    pygame.display.set_caption("Conway's Game of Life (Teaching Version)")

    clock = pygame.time.Clock()

    grid = random_grid()
    running = True

    while running:
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                running = False

            elif event.type == pygame.KEYDOWN:
                if event.key == pygame.K_SPACE:
                    # 推进到下一代
                    grid = next_generation(grid)

                elif event.key == pygame.K_r:
                    # 随机重新初始化
                    grid = random_grid()

                elif event.key == pygame.K_ESCAPE:
                    running = False

        draw_grid(screen, grid)
        pygame.display.flip()
        clock.tick(60)

    pygame.quit()


# ==================================================
# 7. 程序入口
# ==================================================
if __name__ == "__main__":
    main()

#
Lec8
Posted Updated Note5 minutes read (About 677 words)

Lec8

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
# ===== 第一课时：函数的定义与使用 =====

# 1. 最简单的函数 - 无参数无返回值
print("--- 示例1：打招呼函数 ---")

def say_hello():
    """这是一个简单的打招呼函数"""
    print("你好！")
    print("欢迎学习Python！")

# 调用函数
say_hello()
say_hello()  # 可以多次调用

# 2. 带参数的函数
print("\n--- 示例2：带参数的函数 ---")

def greet(name):
    """向指定的人打招呼"""
    print("你好，", name, "！")
    print("很高兴见到你！")

greet("小明")
greet("小红")
greet("小刚")

# 3. 带多个参数的函数
print("\n--- 示例3：多个参数 ---")

def introduce(name, age, hobby):
    """自我介绍函数"""
    print(f"大家好，我叫{name}")
    print(f"我今年{age}岁")
    print(f"我喜欢{hobby}")
    print("-" * 30)

introduce("小明", 10, "打篮球")
introduce("小红", 9, "画画")

# 4. 带返回值的函数
print("\n--- 示例4：返回值 ---")

def add(a, b):
    """计算两个数的和"""
    result = a + b
    return result

sum1 = add(5, 3)
sum2 = add(10, 20)
print("5 + 3 =", sum1)
print("10 + 20 =", sum2)

# 5. 实用函数：计算矩形面积
print("\n--- 示例5：计算面积 ---")

def rectangle_area(length, width):
    """计算矩形面积"""
    area = length * width
    return area

area1 = rectangle_area(5, 3)
area2 = rectangle_area(10, 8)
print("长5宽3的矩形面积：", area1)
print("长10宽8的矩形面积：", area2)

# 6. 实用函数：判断奇偶
print("\n--- 示例6：判断奇偶数 ---")

def is_even(number):
    """判断一个数是否为偶数"""
    if number % 2 == 0:
        return True
    else:
        return False

print("4是偶数吗？", is_even(4))
print("7是偶数吗？", is_even(7))
print("10是偶数吗？", is_even(10))

# 7. 实用函数：计算平均分
print("\n--- 示例7：计算平均分 ---")

def calculate_average(score1, score2, score3):
    """计算三门课的平均分"""
    total = score1 + score2 + score3
    average = total / 3
    return average

avg = calculate_average(85, 90, 88)
print("平均分是：", avg)

# 8. 综合练习：温度转换
print("\n--- 示例8：温度转换 ---")

def celsius_to_fahrenheit(celsius):
    """摄氏度转华氏度"""
    fahrenheit = celsius * 9/5 + 32
    return fahrenheit

def fahrenheit_to_celsius(fahrenheit):
    """华氏度转摄氏度"""
    celsius = (fahrenheit - 32) * 5/9
    return celsius

temp_c = 25
temp_f = celsius_to_fahrenheit(temp_c)
print(f"{temp_c}°C = {temp_f}°F")

temp_f2 = 77
temp_c2 = fahrenheit_to_celsius(temp_f2)
print(f"{temp_f2}°F = {temp_c2}°C")

# 9. 带默认参数的函数
print("\n--- 示例9：默认参数 ---")

def power(number, exponent=2):
    """计算幂次方，默认是平方"""
    result = number ** exponent
    return result

print("5的平方：", power(5))
print("2的3次方：", power(2, 3))
print("3的4次方：", power(3, 4))

# 10. 实践项目：成绩管理系统
print("\n--- 综合实践：成绩管理 ---")

def get_grade(score):
    """根据分数返回等级"""
    if score >= 90:
        return "优秀"
    elif score >= 80:
        return "良好"
    elif score >= 60:
        return "及格"
    else:
        return "不及格"

def print_report(name, math, chinese, english):
    """打印成绩单"""
    print(f"\n===== {name}的成绩单 =====")
    print(f"数学：{math}分 - {get_grade(math)}")
    print(f"语文：{chinese}分 - {get_grade(chinese)}")
    print(f"英语：{english}分 - {get_grade(english)}")
    
    avg = calculate_average(math, chinese, english)
    print(f"平均分：{avg:.1f}分 - {get_grade(avg)}")
    print("=" * 30)

print_report("小明", 85, 92, 88)
print_report("小红", 95, 87, 91)
#
KAF Training
Posted Updated Note2 minutes read (About 281 words)

KAF Training

For the figure, the each step corresponds to a processed batch(16).

#
Pixtral 12B API Inference
Posted Updated Note3 minutes read (About 435 words)

Pixtral 12B API Inference

Repository:
https://github.com/PSGBOT/pixtral-12B-Inference

本地图片上传

1
2
3
4
5
6
7
8
9
10
11
def encode_image(image_path):
    """Encode the image to base64."""
    try:
        with open(image_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode('utf-8')
    except FileNotFoundError:
        print(f"Error: The file {image_path} was not found.")
        return None
    except Exception as e:  # Added general exception handling
        print(f"Error: {e}")
        return None

Prompt

VLM物体描述的prompt:

核心需要：准确定位物体所在方位，不把远景识别为物体，降低False Positive

1
2
3
4
5
6
7
8
9
10
11
12
13
Focus on the area highlighted in green in the image.

Step 1: Determine if the highlighted area represents a distinct, identifiable object or instance:
- If the highlighted area is clearly a distinct object, proceed to Step 2.
- If the highlighted area is abstract, ambiguous, or you cannot confidently identify it as a specific object (e.g., part of background, texture, partial view), respond with "Valid: No".

Step 2: If the highlighted area is a distinct object, provide:
1. The specific name of the object (be precise and use technical terms when appropriate)
2. The primary function or purpose of this object
3. Any notable features visible in the highlighted area (no color description)
4. If there is text visible on the object, include what it says

Remember, if you're uncertain about the highlighted area being a distinct object, respond only with "Valid: No".

输出结果：

  • Valid
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Valid: Yes

    1. The specific name of the object: Soap dispenser
    2. The primary function or purpose of this object: To dispense liquid soap or hand sanitizer.
    3. Notable features visible in the highlighted area:
    	- The dispenser has a pump mechanism at the top.
    	- The body of the dispenser is cylindrical.
    	- The material appears to be translucent plastic.
    4. There is no visible text on the object.
  • invalid
    1
    Valid: No

VLM输出->Structured Output

使用另一个LLM来对VLM输出的内容进行parse，转化成json文件, 通过mistral ai 提供的接口实现:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
class Instance(BaseModel):
    valid: str
    name: Optional[str] = None
    feature: Optional[List[str]] = Field(default_factory=list)
    usage: Optional[List[str]] = Field(default_factory=list)

def parse_description_msg(msg):
    message = [
        {"role": "system", "content": "Extract the description information."},
        {
            "role": "user",
            "content": msg,
        },
    ]
    return message

chat_response = self.client.chat.parse(
	model=self.llm,
	messages=msg,
	response_format=Instance,
	max_tokens=self.llm_max_tokens,
	temperature=self.llm_temperature,
)
return json.loads(chat_response.choices[0].message.content)
#
FCSGG Repo Explanation
Posted Updated Note6 minutes read (About 870 words)

FCSGG Repo Explanation

FCSGG Repository Summary

FCSGG (Fully Convolutional Scene Graph Generation) is a PyTorch implementation of the paper “Fully Convolutional Scene Graph Generation” published in CVPR 2021. The project focuses on scene graph generation, which is the task of detecting objects in an image and identifying the relationships between them.

Core Components:

  1. Architecture:

    • Built on Detectron2, a popular object detection framework by Facebook
    • Uses a one-stage detector approach (CenterNet) as the meta-architecture
    • Supports various backbones including ResNet, HRNet (High-Resolution Network), Hourglass networks, and DLA

  2. Key Features:

    • Fully convolutional approach to scene graph generation
    • Multiple backbone options with different feature pyramid networks (FPN, BiFPN, HRFPN)
    • Various head designs including multiscale heads and attention mechanisms
    • Support for different input resolutions and training strategies

  3. Dataset:

    • Primarily designed for the Visual Genome dataset, a large-scale dataset for scene understanding
    • Includes custom data loaders and preprocessing for scene graph generation

  4. Model Components:

    • Backbones: Various CNN architectures (ResNet, HRNet, Hourglass, DLA)
    • Necks: Feature pyramid networks and variants (FPN, BiFPN, HRFPN, Trident)
    • Heads: Detection and relationship prediction heads
    • Loss Functions: Custom losses for object detection and relationship prediction

  5. Utilities:

    • Visualization tools for scene graphs
    • Evaluation metrics for scene graph generation
    • Training and inference scripts

Project Structure:

  • fcsgg/: Main module containing model implementation

    • modeling/: Neural network architecture components
      • backbone/: Feature extraction networks
      • necks/: Feature pyramid networks
      • heads/: Detection and relationship prediction heads
      • meta_arch/: High-level model architecture (CenterNet)
    • data/: Dataset handling and preprocessing
    • evaluation/: Metrics and evaluation code
    • utils/: Helper functions and utilities
    • layers/: Custom neural network layers
    • structures/: Data structures for scene graphs

  • configs/: Configuration files for different model variants

  • tools/: Training, evaluation, and visualization scripts

  • GraphViz/: Visualization tools for scene graphs

Key Innovations:

The project implements a fully convolutional approach to scene graph generation, which differs from traditional two-stage methods. Instead of first detecting objects and then predicting relationships, it uses a one-stage detector to simultaneously predict objects and their relationships in a fully convolutional manner.

Benchmarks:

The repository provides several pre-trained models with different backbones:

  1. HRNetW32-1S
  2. ResNet50-4S-FPN×2
  3. HRNetW48-5S-FPN×2

These models achieve competitive performance on the Visual Genome dataset for scene graph generation tasks.

Usage:

The project provides tools for training, evaluation, and visualization of scene graphs. It requires the Visual Genome dataset and can be run using Docker or directly with PyTorch.

In summary, FCSGG is a comprehensive implementation of a state-of-the-art approach to scene graph generation using fully convolutional networks, offering various model architectures and training configurations.

How Detectron2 is Used in FCSGG

FCSGG is built on top of Detectron2, Facebook’s object detection framework, and leverages many of its components while extending it for scene graph generation. Here’s a detailed breakdown:

1. Core Architecture Integration

  • Meta Architecture: FCSGG registers a custom meta architecture called “CenterNet” with Detectron2’s META_ARCH_REGISTRY. This extends Detectron2’s modular architecture system while maintaining compatibility.

  • Backbone Networks: FCSGG uses Detectron2’s backbone networks (ResNet, etc.) directly and also implements custom backbones like HRNet while following Detectron2’s backbone interface.

  • Feature Pyramid Networks (FPN): The repository uses Detectron2’s FPN implementation and extends it with custom variants like BiFPN and HRFPN.

2. Configuration System

  • YAML Configuration: FCSGG adopts Detectron2’s YAML-based configuration system, extending it with custom configurations for scene graph generation through add_fcsgg_config().

  • Command Line Arguments: The training script uses Detectron2’s default_argument_parser() to maintain the same command-line interface.

3. Data Handling

  • Dataset Registration: Visual Genome dataset is registered with Detectron2’s DatasetCatalog and MetadataCatalog, making it available through Detectron2’s data loading pipeline.

  • Custom Dataset Mapper: FCSGG implements a custom DatasetMapper class that extends Detectron2’s mapper to handle scene graph annotations.

  • Data Loaders: The repository uses Detectron2’s build_detection_train_loader and build_detection_test_loader with custom mappers.

4. Training and Evaluation

  • Trainer Class: FCSGG extends Detectron2’s DefaultTrainer class to customize the training loop, evaluation metrics, and data loading.

  • Checkpointing: The repository uses Detectron2’s DetectionCheckpointer for model saving and loading.

  • Distributed Training: FCSGG leverages Detectron2’s distributed training utilities through detectron2.utils.comm and the launch function.

  • Custom Evaluators: The repository implements a custom VGEvaluator for scene graph evaluation while following Detectron2’s evaluator interface.

5. Visualization and Logging

  • Event Storage: FCSGG uses Detectron2’s event storage system for logging metrics during training.

  • Visualization Tools: The repository leverages Detectron2’s visualization utilities for debugging and result analysis.

6. Extensions for Scene Graph Generation

  • Custom Heads: While using Detectron2’s architecture, FCSGG implements custom prediction heads for relationship detection.

  • Scene Graph Structures: The repository defines custom data structures for scene graphs that integrate with Detectron2’s Instances class.

  • Loss Functions: FCSGG implements specialized loss functions for scene graph generation while maintaining compatibility with Detectron2’s loss computation framework.

7. Installation and Dependencies

  • Submodule Integration: Detectron2 is included as a Git submodule, ensuring version compatibility.

  • Build Process: The installation process includes building Detectron2 from source to ensure proper integration.

In summary, FCSGG uses Detectron2 as its foundation, leveraging its modular architecture, data handling, training infrastructure, and configuration system while extending it with custom components for scene graph generation. This approach allows FCSGG to benefit from Detectron2’s robust implementation and optimizations while adding specialized functionality for relationship detection between objects.

#
Detectron
FCSGG Repository Application
Posted Updated Note4 minutes read (About 570 words)

FCSGG Repository Application

Official repo:
https://github.com/liuhengyue/fcsgg
Our repo:
https://github.com/PSGBOT/KAF-Generation

My venv: fcsgg

Installation

Environment Preparation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
git clone git@github.com:liuhengyue/fcsgg.git
cd fcsgg
git submodule init
git submodule update
conda create --name fcsgg
conda create -n fcsgg python=3.10
conda install nvidia/label/cuda-11.8.0::cuda-toolkit -c nvidia/label/cuda-11.8.0
conda install cudatoolkit
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# for building detectron
conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/fcsgg/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/fcsgg/include/"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/fcsgg/"

conda deactivate
conda activate fcsgg

export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

pip install -r requirements.txt
python -m pip install -e detectron2

Downloads

Datasets:

1
2
3
4
5
cd ~/Reconst
wget https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip -P ./Data/vg/
wget https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip -P ./Data/vg/
unzip -j ./Data/vg/images.zip -d ./Data/vg/VG_100K
unzip -j ./Data/vg/images2.zip -d ./Data/vg/VG_100K

Download the scene graphs and extract them to datasets/vg/VG-SGG-with-attri.h5.

Issues

1

1
AttributeError: module 'PIL.Image' has no attribute 'LINEAR'. Did you mean: 'BILINEAR'?

LINEAR-> BILINEAR: commit

2

在尝试训练的过程中报错：

1
2
3
  File "/home/cyl/Reconst/fcsgg/fcsgg/data/detection_utils.py", line 432, in generate_score_map
    masked_fmap = torch.max(masked_fmap, gaussian_mask * k)
RuntimeError: The size of tensor a (55) must match the size of tensor b (56) at non-singleton dimension 1

modify detection_utils.py: commit

Training

首先更改训练的配置文件./config/quick_schedules/Quick-FCSGG-HRNet-W32.yaml, (原文件使用预训练的参数)

1
2
3
4
MODEL:
  META_ARCHITECTURE: "CenterNet"
  HRNET:
    WEIGHTS: "output/FasterR-CNN-HR32-3x.pth"

更改为train from scratch

1
2
3
4
MODEL:
  META_ARCHITECTURE: "CenterNet"
  HRNET:
    WEIGHTS: ""  # Empty string to train from scratch

再运行：

1
python tools/train_net.py --num-gpus 1 --config-file configs/quick_schedules/Quick-FCSGG-HRNet-W32.yaml

成功训练✌

1
2
3
4
5
6
...
[04/23 10:21:01] d2.utils.events INFO:  eta: 0:05:37  iter: 1159  total_loss: 1.042  loss_cls: 0.7593  loss_box_wh: 0.08827  loss_center_reg: 0.02485  loss_raf: 0.1754    time: 0.4042  last_time: 0.4519  data_time: 0.0043  last_data_time: 0.0044   lr: 0.001  max_mem: 4141M
[04/23 10:21:09] d2.utils.events INFO:  eta: 0:05:29  iter: 1179  total_loss: 1.028  loss_cls: 0.7208  loss_box_wh: 0.09246  loss_center_reg: 0.02669  loss_raf: 0.1625    time: 0.4042  last_time: 0.4035  data_time: 0.0041  last_data_time: 0.0044   lr: 0.001  max_mem: 4141M
[04/23 10:21:17] d2.utils.events INFO:  eta: 0:05:21  iter: 1199  total_loss: 1.01  loss_cls: 0.671  loss_box_wh: 0.1038  loss_center_reg: 0.02432  loss_raf: 0.1635    time: 0.4042  last_time: 0.3943  data_time: 0.0042  last_data_time: 0.0043   lr: 0.001  max_mem: 4141M
[04/23 10:21:25] d2.utils.events INFO:  eta: 0:05:13  iter: 1219  total_loss: 0.9737  loss_cls: 0.6887  loss_box_wh: 0.0929  loss_center_reg: 0.02574  loss_raf: 0.1749    time: 0.4041  last_time: 0.4101  data_time: 0.0041  last_data_time: 0.0042   lr: 0.001  max_mem: 4141M
...

Explanation

See [[FCSGG Repo Explanation]]

#
OmegaConf Python Package
Posted Updated Notea minute read (About 166 words)

OmegaConf Python Package

docs

Start

1
pip install omegaconf
1
from omegaconf import OmegaConf

Create

You can create OmegaConf objects from multiple sources.

From dict

1
2
3
4
5
6
7
8
9
conf = OmegaConf.create({"k" : "v", "list" : [1, {"a": "1", "b": "2", 3: "c"}]})
print(OmegaConf.to_yaml(conf))

k: v
list:
- 1
- a: '1'
  b: '2'
  3: c

From list

1
2
3
4
5
6
7
8
conf = OmegaConf.create([1, {"a":10, "b": {"a":10, 123: "int_key"}}])
print(OmegaConf.to_yaml(conf))

- 1
- a: 10
  b:
    a: 10
    123: int_key

From yaml

1
2
3
4
5
6
7
8
9
10
11
12
conf = OmegaConf.load('source/example.yaml')
# Output is identical to the YAML file
print(OmegaConf.to_yaml(conf))

server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2

From dot-list

1
2
3
4
5
6
7
8
9
10
11
dot_list = ["a.aa.aaa=1", "a.aa.bbb=2", "a.bb.aaa=3", "a.bb.bbb=4"]
conf = OmegaConf.from_dotlist(dot_list)
print(OmegaConf.to_yaml(conf))

a:
  aa:
    aaa: 1
    bbb: 2
  bb:
    aaa: 3
    bbb: 4

From command line arguments

1
2
3
4
5
6
7
8
sys.argv = ['your-program.py', 'server.port=82', 'log.file=log2.txt']
conf = OmegaConf.from_cli()
print(OmegaConf.to_yaml(conf))

server:
  port: 82
log:
  file: log2.txt
#
SJTU-HPC基本操作
DINOv2 Repository Application
Posted Updated Note2 minutes read (About 236 words)

DINOv2 Repository Application

Repository:

Installation

official:

1
conda env create -f conda.yaml

不建议使用official的conda.yaml, 使用更改后的conda_cyl.yaml。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1

conda install nvidia/label/cuda-11.7.0::cuda-toolkit -c nvidia/label/cuda-11.7.0
conda install cudatoolkit
conda install -c conda-forge gcc=11.2.0
conda install -c conda-forge gxx=11.2.0

conda env config vars set LD_LIBRARY_PATH="/home/cyl/miniconda3/envs/dinov2/lib/"
conda env config vars set CPATH="/home/cyl/miniconda3/envs/dinov2/include/"
conda env config vars set CUDA_HOME="/home/cyl/miniconda3/envs/dinov2/"

export CC=$CONDA_PREFIX/bin/gcc
export CXX=$CONDA_PREFIX/bin/g++

# check with `which g++`

conda env update -f conda_cyl.yaml

pip3 install -U xformers==0.0.18

conda env config vars set PYTHONPATH="/home/cyl/Reconst/dinov2/"

Demo 🐱

官方提供了 depth estimation 和 segmentation 的 notebook，可以找时间理解一下

Train

使用的数据集为Imagenet-mini

1
2
3
4
imagenet-mini
├── labels.txt
├── train
└── val

Note: 需要额外添加一个label.txt

使用脚本生产数据集的meta data:

1

#
Follow

Archives

Recents

Tags

3D-Scene17
6-D3
AI16
AIGC1
API1
AR2
Academic1
Algorithm1
Aliyun1
App2
Atlas1
BS41
Bayesian-Inference1
Beautify1
Behaviorism1
Business1
C1
CADC1
CD1
CLI1
CLIP11
CNN1
CV68
Camera1
Capstone10
Chemistry1
Claude1
Communication2
Contrastive-Learning5
Control3
Csharp9
Css1
Cuda3
DD1
DINO4
DT1
Dataframe1
Debate5
Debugger1
Deep-Learning1
Development-Tools1
Diffusion2
Diffusion-Policy1
DiffusionModel4
Discrete-Mathematics1
Disney1
Docker1
Docs2
Dynamic-programming1
ESP322
Education1
Embeded-System9
Embodied-AI19
Emoation1
Emotion13
Ethic1
Experiment2
FL1
FPN2
Family1
Federated-Learning1
Foundation1
FoundationModel4
Functional programming1
GPT3
Game5
Gated-NN3
Git7
Github1
Godot3
Graph1
HPC1
HRI2
Haskell1
Health2
Hexo10
Hierarchical4
Html5
Humanism1
Humanoid1
HumanoidRobot1
Hybrid-Control1
Hyprland2
IK1
Image-Grounding2
Image-Text4
Image-generation2
Image2Text7
ImgGen3
ImitationLearning5
Information-Theory1
Jolt1
Json1
LLM17
LSP2
LatentAction1
Latex2
Lego1
Life4
LinearAlgebra1
Linux22
Live2d1
Love4
Lua1
MBTI1
ML14
MPC2
MR/AR3
Machine-Learning3
Mason1
Math7
Meme1
Message-Passing2
MindPlus1
MoE2
Mod3
Model-Predictive-Control1
Motivation1
Moveit1
Movie1
Multi-Agent1
Multi-modal14
Multi-view1
MultiModal5
Music5
NLP6
NN12
Network2
Nodejs5
Numpy1
Nvim9
Object-Detection9
Open-Vocabulary11
OpenCV1
Oral1
PHD1
PSY5
Pandas2
Panoptic1
Path1
Philosophy3
PhysX1
Physical-Scene4
Physics-engine1
Pio2
Planning1
Plugin8
PoseEstimation3
Postgraduate1
Prefab1
Probability2
Python30
Pytorch1
QML1
Quantum1
RAG1
RL3
RNN4
ROS6
Reading19
Real2Sim2
Reconstruct13
Regex2
Reinforcement-Learning2
Reinforcement-learning1
Repository5
Representation-Learning5
Research-paper97
Robot5
RobotLearning13
Robotics38
SJTU-Lecture1
SQL2
SSH3
Scalability2
Scene-graph34
Scene-synthesis2
Science-fiction1
Scrap1
Script2
Segmentation8
Semantic15
Shader3
Shell4
Signals and Systems1
Sim2Real1
Sklearn1
Snippets1
Society4
Star-rail1
Statistics2
Subgraph1
Submodule1
Supervised-learning2
Survey4
TC1
TOEFL1
Task-Planning9
Tasks5
Tech Communication1
Torch5
Transformer20
Translation-Embedding2
Travel5
UI1
Unified-Multimodal1
Unity20
Unsupervised-learning1
VAE2
VLA4
VLM9
VLP5
VQ-VAE1
Variational-Inference1
Version-management1
ViT5
VideoEditing2
Vim1
Visual-Relation23
WSL1
Waybar1
Wayland1
Web1
Website1
Well-being1
Window-manager2
WorldModel2
YKLL3
Zen2
♥️2
实习1
🍢1
🍰1
🐱2
🧀1
×