因为oh-my-live2d官网崩了，所以需要自己私有托管一下live2d服务才能让博客网上的小埋复活。(bonus:可以借此尝试自己制作一个live2d形象)
更于9/25，拖着很久没搞，然后官网自己复活了，好好好，直接快进到customized live2d。

因为之前使用gitalk评论系统在国内用不了了，所以尝试更换为一些国内可用的方案。其中一个是国内开发者开发@imaegoo搭建的Twikoo系统

对，然后针对icarus系的主题，官网有提供详细例程
主要分为云函数部署部分和前端配置部分

云函数

归根结底就是为静态的博客网提供一个可以响应用户数据提交和显示的服务。
主要分为两块

• Mongodb: 用于云存储用户评论数据
• Huggingface: 用于部署一个云服务器，响应用户需求，与db交互。

那么首先就需要去mongodb申请一个数据库，依照这个例程： https://twikoo.js.org/mongodb-atlas.html

随后huggingface的服务就通过这串链接字符串来与这个数据库交互

随后去huggingface开启云服务器: https://twikoo.js.org/backend.html#hugging-face-%E9%83%A8%E7%BD%B2
我的云仓库为： https://huggingface.co/spaces/CallMeChen/BlogComment
启动服务成功后可以看到如下log:

前端配置

参照 https://www.anzifan.com/post/icarus_to_candy_2

关于评论头像

访客还可以通过输入数字 QQ 邮箱地址，使用 QQ 头像发表评论。

后续

添加图床方便上传图片(已解决)

维护

10/20发现评论系统不能用了，发现是mongodb暂停了数据库服务，想起来它应该是每个月都需要自己connect一下（也算是释放不活跃用户的资源吧）。

随后发现huggingface的仓库一直卡在build，遂更换为Netlify云函数平台。
https://app.netlify.com/sites/admirable-sunburst-4762e3/configuration/general

1
2
3

{
     into: { db: "test", coll: "comment" },
}

L2 (Embedded system overview)

Take a look at the differences between Microprocessor (in our personal computer or phone) and Microcontroller

In embedded system, harvard architecture is widely used.
Our board (STM32F103C8T6) use harvard architecture on physical level (refer to the block diagram in reference manual)

However, in the software level, we treat the instruction memory and data memory as a whole block of memory (therefore, it is more accurate to say that the stm32 uses a mixed Harvard and von Neumann architecture.).

In stm32, instruction memory, data memory, registers of peripherals/IO are all mapped to memory.

L3 (Programming)

Type Qualifiers

const

• implies that value not supposed to be written by program (read only) during run-time. Can be modified by others like hardware.

  If you want to save the limited RAM spaces (data memory) for other variables, you can use this keyword to store this variable in ROM (program memory). It’s important for harvard architecture.

volatile

• indicate the value can be changed by something other than program so it should be reexamined frequently.

  This means two things:

  • The compiler will not try to optimize the variable with volatile. See the two examples on slides.
  • Each time the program reads the volatile variable, the processor will not look into cached data memory, meaning that the program can always get the newest updated data in memory (which is very important when external hardware change the variable). However, this case is not relevant with STM32 MCU since it didn’t have cache.

What about const volatile ?

• The combination of the above two concepts. Usually used to declare pointers
  Example:
  const volatile char *a declares a pointer pointing to a value that cannot be changed by the program through *a, but the value of a can be changed (pointing to another value). *a = 0 is not allowed, a = &b is allowed.

Generally, we use const volatile to declare pointers that points to hardware registers or memory-mapped Input ports(read only).

Basic Program Structure

1
2
3
4
5
6
7
8
9
10
11

// import header file for the board (containing the declaration of SFR)
// declare global variables
int main(void){
	// initialization (system clock, peripherals)
	while(true) // super loop, the the program alive
	{
		// interact with peripherals
	}

}

How to interact with peripherals

We need to use C code to set the value of SFR.

SFR (special function registers)

These are registers that are embedded in peripherals, used for configuration and control of peripherals.

VERBOSE~
If we want to get the status of a peripheral, we read the value of SFR.
If we want to send something to peripheral, we write value to SFR.

Let’s take timer as an example:
SFR in block diagram of timer:

SFR declaration in code:

We change operate with these registers through Bit Operation.

L4 (IO)

All the modes of GPIO:

You may see an unfamiliar unit

Input

After we configured the GPIO to be input ports, the output driver is disabled (disconnected).

Pull down

Pull up

Floating

General Purpose Output

The input driver part is still enabled so that we can read the output status.

Open Drain

Can “generate” voltage higher than VDD at IO pin.

Push Pull

Most common one.

Alternative Function Output

Not covered.

L5 (Interrupts)

Why interrupt?

Peripherals inform the processor through external interrupt.

Where do interrupts come from

We mainly deal with the interrupts from peripherals.

How to handle interrupts

Through interrupt service routine (ISR)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

| Vector Number | Interrupt Number | Description                    | Vector Address   |
|---------------|------------------|--------------------------------|------------------|
| 0             | -                | Initial Stack Pointer          | 0x0800 0000      |
| 1             | -                | Reset Handler                  | 0x0800 0004      |
....
| 27            | 11               | DMA1 Channel1 global Interrupt | 0x0800 006C      |
| 28            | 12               | DMA1 Channel2 global Interrupt | 0x0800 0070      |
| 29            | 13               | DMA1 Channel3 global Interrupt | 0x0800 0074      |
| 30            | 14               | DMA1 Channel4 global Interrupt | 0x0800 0078      |
| 31            | 15               | DMA1 Channel5 global Interrupt | 0x0800 007C      |
| 32            | 16               | DMA1 Channel6 global Interrupt | 0x0800 0080      |
| 33            | 17               | DMA1 Channel7 global Interrupt | 0x0800 0084      |
...
| 41            | 25               | TIM1 Break Interrupt           | 0x0800 00A4      |
| 42            | 26               | TIM1 Update Interrupt          | 0x0800 00A8      |
| 43            | 27               | TIM1 Trigger and Commutation   | 0x0800 00AC      |
| 44            | 28               | TIM1 Capture Compare Interrupt | 0x0800 00B0      |
...

This is an arbitrary IVT that depicts the pattern of IVT, for detailed IVT of STM32, please refer to the reference manual.

What to do inside ISR

• Always remember to clear the interrupt flag in ISR.
• Because there is often more interrupt sources than interrupt vectors, you need to judge which source triggered this interrupt based on interrupt status register.
• Customized operation… (but be short, because if the operation take too many clock cycles, it may be interrupted by another interrupt source, which may not be on purpose)

Nested interrupts

L6 (Timer)

Let’s split Timer peripheral into 3 parts:

The reset frequency of the counter is $$\frac{f_{input}}{(Prescaler+1)\times(Counter Period+1)}$$

Red part: Timer-channels unit

The timer channels are the working elements of the timer.
They are the means by which a timer peripheral interacts with its external environment (through input capture or output compare).

How to use multiple timer for more counting digits (32-bit)

L7 (LCD)

Just refer to RC2_LCD. I think it’s not the focus of final exam.

L9 & L10 (Input Capture & Output Compare)

IC is a peripheral that can monitor the input signal changes (pos/neg edge) independent of the processor (Core).
OC is a peripheral that can generate precise output signal independent of the processor (Core).

In STM32, it is embedded in timer peripheral (together with output compare).

IC

You can consider IC as a timer value recorder. It will record the timer value each time the capture condition is met (you can see from the diagram, the timer value is from CNT counter).

These conditions can be:

• rising edge
• falling edge
• both
  With a prescaler, we trigger capture events every few edges.
  Similar for interrupt, we can trigger an interrupt every few captures.

Note:
The IC does not capture the edge immediately when a rising or falling edge happened. The capture event needs to be sync with PB_clk.
Further more, the module will capture the timer counter value that is valid 2-3 PB_clk cycles after the capture event.

For detailed configuration, please refer to Reference Manual.pdf on canvas, page 349-359, 382-385.

OC

Just refer to RC2_Output_Campare and RC3_Lab4 for the concepts and PWM configuration.
Also, the solution of hw2 has been uploaded to canvas, please take a look.

项目框架

以下为项目布局示意图

（左下角为深度相机）

以下为项目的架构图

具体解释一下这张图

• 首先用户佩戴 hololens，hololens 会先把可交互的物体（物体数据源于物体检测算法）渲染在 UI 上，之后用户基于这些视觉信息，给出交互指令。

• 随后系统会根据给到的指令和可交互物体的信息，规划机械臂的运动路径。

• 路径信息会首先传到数字孪生系统中。该系统中有三部分孪生，分别是人体（实时数据来自人体检测算法），环境（实时数据来自 hololens 的环境感知系统） 和 机械臂（来自于前面提到的路径信息）的孪生。

• 随后该 DT 系统会根据路径信息解算在机械臂运动过程中是否会发生碰撞。

  • 如果发生碰撞
    • 会把碰撞发生的位发还给 hololens 显示，用户可以选择取消指令或者选择移走遮挡的物体（）。
  • 如果不发生碰撞
    • 会把机械臂的预计运动轨迹发回给 hololens 显示，用于警戒用户。
    • 将运动轨迹发给机械臂控制器，控制器控制机械臂运动。

程序接口定义

Python->Unity

代码仓库：

https://github.com/Chen-Yulin/Unity-Python-UDP-Communication

传输的数据为字符串：

1
2
3

def SendData(self, strToSend):
    # Use this function to send string to C#
    self.udpSock.sendto(bytes(strToSend,'utf-8'), (self.udpIP, self.udpSendPort))

物体识别信息

需要包含的信息：物体种类，物体的三轴方位，三轴旋转，三轴尺寸。

格式：

1
2
3
4
5
6
7
8
9
10
11

{Object Detection}
category
position.x
position.y
position.z
eulerRotation.x
eulerRotation.y
eulerRotation.z
scale.x
scale.y
scale.z

示例：

1
2
3
4
5
6
7
8
9
10
11

{Object Detection}
Handle
0.1
0.052
0.026
0
90
0
0.5
0.3
1

机械臂实时角度

需要包含的信息：6 个 joint 角度(单位为度)

格式：

1
2
3
4
5
6
7
8

{Current Joint}
j0
j1
j2
j3
j4
j5
j6

Unity->Python

机械臂需要旋转的角度

需要包含的信息：6 个 joint 角度(单位为度)

格式：

1
2
3
4
5
6
7
8

{Target Joint}
j0
j1
j2
j3
j4
j5
j6

Python-Unity Network

Python 和 Unity 间通讯

https://github.com/Siliconifier/Python-Unity-Socket-Communication

Demo 设计

一、需要实现的功能：

• 机械臂根据工人指示移动物体
• 机械臂运动意图在 Hololens 中的渲染
• 人体/机械臂轨迹的记录，用于评估（？）

二、具体 demo

机械臂移动物体到指定位置

首先 RGB-D 相机识别出桌面上的某一个物体，反馈到 hololens 的界面中，用户会看到对应物体上渲染出来了一个边界框和表示位姿的坐标轴：

随后用户可以通过手部的射线来移动这个目标框（物体上的检测框依然会存在）到某个位置，然后目标框上会显示一个 UI 询问是否要把物体移动到这里，如果点击确认，机械臂就会把对应的物体到目标框的位置。

可以在路径中设置一些障碍，展示机械臂障碍识别的功能。

机械臂移动物体到工人手上

用户凝视一个物体时显示一个界面，即是否要将操作员注视的物体移动到他的手上，如果手腕翻起，即为确认。随后机械臂会将该物体移动到手上。

自定义 Gen6d 物体

仓库：https://github.com/liuyuan-pal/Gen6D

手册：https://github.com/liuyuan-pal/Gen6D/blob/main/custom_object.md

步骤指令：

1
2
3
4
5
6
7
8
9

python prepare.py --action video2image --input data/custom/part1/ref.mp4 --output data/custom/part1/images --frame_inter 10 --image_size 960  

python prepare.py --action sfm --database_name custom/part1 --colmap D:\COLMAP\COLMAP-3.7-windows-cuda\COLMAP.bat        

# do some cloudcompare thing

python predict.py --cfg configs/gen6d_pretrain.yaml --database custom/part1 --video data/custom/part1/test.mp4 --resolution 460 --output data/custom/part1/test --ffmpeg ffmpeg

ffmpeg -framerate 30 -i %d-bbox.jpg -c:v libx264 -r 30 -pix_fmt yuv420p output.mp4

关于判定不准确怎么解决：https://github.com/liuyuan-pal/Gen6D/issues/29

6d pose -> unity coordinate

unity 使用左手坐标系，普遍的 6d 算法使用右手坐标系，所以得出[R;t]后需要做一步针对 y 轴的反射变换

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def right_to_left_hand_pose_R(R):
    # 定义反射矩阵
    M = torch.tensor([
        [ 1,  0,  0],
        [ 0, -1,  0],
        [ 0,  0,  1]
    ], dtype=R.dtype)
    
    # 转换旋转矩阵
    R_prime = M @ R @ M
    
    return R_prime

def right_to_left_hand_pose_t(t):
    # 定义反射矩阵
    M = torch.tensor([
        [ 1,  0,  0],
        [ 0, -1,  0],
        [ 0,  0,  1]
    ], dtype=t.dtype)

    # 转换位移向量
    t_prime = M @ t

    return t_prime

可以看到效果很好：

6-D Pose Estimation Survey

Model based (CAD model)

State of The Art: Foundation Pose (https://github.com/NVlabs/FoundationPose)

RGB

CASAPose (https://github.com/fraunhoferhhi/casapose?tab=readme-ov-file)

MegaPose (https://github.com/megapose6d/megapose6d)

D-RGB

MegaPose (https://github.com/megapose6d/megapose6d)

OVE6D (https://github.com/dingdingcai/OVE6D-pose)

Non-model

Gen6D (https://github.com/liuyuan-pal/Gen6D)

Yolov5 (https://github.com/cviviers/YOLOv5-6D-Pose)