参考: https://forum-zh.obsidian.md/t/topic/292

文献搜集网站

Google Scholar 114
CNKI 80
Connected Papers

总结了各文章之间的 关联度，太直观了

文献管理软件

Zotero

参考: https://sspai.com/post/56724
安装了Firefox-Zotero插件，可以直接捕获网页内容并存到Zotero Liberary中（打开pdf界面后直接点击插件按钮即可保存）
保存后Zotero会自行抓取论文关键信息并重新生成标题

参考仓库: https://github.com/rzeldent/esp32cam-rtsp

Objective

Stream video through wifi using ESP-32-CAM.
The video sources can be accessed by an ip address.

Implemenation

Install PlatformIO plugin in vscode.
Clone the repository

1

git clone --recursive https://github.com/rzeldent/esp32cam-rtsp.git

Use vscode pio to open the project. Wait pio till its configuration is done.

Change the default_envs settings, here I use esp32cam_ai_thinker

If no default_envs is specified, pio will build the project for all platforms

Here we can build and upload the program to the board.

Connect to ESP**** WiFi, and visit http://192.168.4.1 to configure the wifi settings of the board. I choose to my phone’s hot spot.

Then open the monitor to check the ip address (you need to connect your computer to the same LAN (local area network) to visit the ip)

Visit the 192.168.23.142 and you will see the page (similar with the page of http://192.168.4.1):

Click rtsp://192.168.23.197:554/mjpeg/1 and you will see the streaming video:

生无可恋做横向.jpg

Standard Toolchain Setup

Guide: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/linux-macos-setup.html#

ESP_IDF

I directly download the archive with all the submodules included: https://github.com/espressif/esp-idf/releases/tag/v5.3.1

This archive can also be downloaded from Espressif’s download server: https://dl.espressif.com/github_assets/espressif/esp-idf/releases/download/v5.3.1/esp-idf-v5.3.1.zip

cd into the unzip folder (The installation will fail in conda venv)

1
2
3

conda deactivate
./install.sh
. ./export.sh

In ~/.zshrc

1

alias get_idf='. $HOME/esp/esp-idf-v5.3.1/export.sh'

Then each time I need to setup esp32 development environment, I only need to type get_idf.

Some useful commands

1
2
3
4
5
6

idf.py set-target esp32
idf.py menuconfig
idf.py build
idf.py -p /dev/ttyUSB0 -b 115200 flash
idf.py -p /dev/ttyUSB0 -b monitor
idf.py -p /dev/ttyUSB0 flash monitor # combine together

PlatformIO

事实证明pio是最方便的。。
事前安装过pio的vscode插件，直接打开pio的esp32项目就直接可以编译上传以及查看串口监视器。

一款线上还在更新中的shader教学，有可交互代码，爆赞
https://thebookofshaders.com/

一些dalao写的shader：
https://www.shadertoy.com/

Shader Tutorials by Ronja
https://www.ronja-tutorials.com/
https://github.com/ronja-tutorials/ShaderTutorials?tab=readme-ov-file

Shader Tutorial For Beginner
https://github.com/Xibanya/ShaderTutorials

因为oh-my-live2d官网崩了，所以需要自己私有托管一下live2d服务才能让博客网上的小埋复活。(bonus:可以借此尝试自己制作一个live2d形象)
更于9/25，拖着很久没搞，然后官网自己复活了，好好好，直接快进到customized live2d。

因为之前使用gitalk评论系统在国内用不了了，所以尝试更换为一些国内可用的方案。其中一个是国内开发者开发@imaegoo搭建的Twikoo系统

拜一下，我的博客网也是改自他的魔改版hexo-icarus主题

对，然后针对icarus系的主题，官网有提供详细例程
主要分为云函数部署部分和前端配置部分

云函数

归根结底就是为静态的博客网提供一个可以响应用户数据提交和显示的服务。
主要分为两块

• Mongodb: 用于云存储用户评论数据
• Huggingface: 用于部署一个云服务器，响应用户需求，与db交互。

那么首先就需要去mongodb申请一个数据库，依照这个例程：https://twikoo.js.org/mongodb-atlas.html

密码保密)

随后huggingface的服务就通过这串链接字符串来与这个数据库交互

随后去huggingface开启云服务器:https://twikoo.js.org/backend.html#hugging-face-%E9%83%A8%E7%BD%B2
我的云仓库为：https://huggingface.co/spaces/CallMeChen/BlogComment
启动服务成功后可以看到如下log:

前端配置

参照 https://www.anzifan.com/post/icarus_to_candy_2

关于评论头像

访客还可以通过输入数字 QQ 邮箱地址，使用 QQ 头像发表评论。

后续

添加图床方便上传图片(已解决)

L2 (Embedded system overview)

Take a look at the differences between Microprocessor (in our personal computer or phone) and Microcontroller

Von Neumann vs Harvard architecture

efficiency: Harvard architecture can avoid the “Von Neumann bottleneck”

VERBOSE~
Von Neumann bottleneck: when the bandwidth between CPU and RAM is much lower than the speed at which a typical CPU can process data, because the shared bus for instructions and data can cause competition.

In embedded system, harvard architecture is widely used.
Our board (STM32F103C8T6) use harvard architecture on physical level (refer to the block diagram in reference manual)

VERBOSE~
ICode bus:This bus connects the Instruction bus of the Cortex®-M3 core to the Flash memory instruction interface.
DCode bus:This bus connects the DCode bus (literal load and debug access) of the Cortex ®-M3 core to the Flash memory Data interface.

However, in the software level, we treat the instruction memory and data memory as a whole block of memory (therefore, it is more accurate to say that the stm32 uses a mixed Harvard and von Neumann architecture.).

In stm32, instruction memory, data memory, registers of peripherals/IO are all mapped to memory.

Table from https://embeddedsecurity.io/vendor-stm32

L3 (Programming)

Type Qualifiers

const

• implies that value not supposed to be written by program (read only) during run-time. Can be modified by others like hardware.

  If you want to save the limited RAM spaces (data memory) for other variables, you can use this keyword to store this variable in ROM (program memory). It’s important for harvard architecture.

volatile

• indicate the value can be changed by something other than program so it should be reexamined frequently.

  This means two things:

  • The compiler will not try to optimize the variable with volatile. See the two examples on slides.
  • Each time the program reads the volatile variable, the processor will not look into cached data memory, meaning that the program can always get the newest updated data in memory (which is very important when external hardware change the variable). However, this case is not relevant with STM32 MCU since it didn’t have cache.

What about const volatile ?

• The combination of the above two concepts. Usually used to declare pointers
  Example:
  const volatile char *a declares a pointer pointing to a value that cannot be changed by the program through *a, but the value of a can be changed (pointing to another value). *a = 0 is not allowed, a = &b is allowed.

Generally, we use const volatile to declare pointers that points to hardware registers or memory-mapped Input ports(read only).

Basic Program Structure

1
2
3
4
5
6
7
8
9
10
11

// import header file for the board (containing the declaration of SFR)
// declare global variables
int main(void){
	// initialization (system clock, peripherals)
	while(true) // super loop, the the program alive
	{
		// interact with peripherals
	}

}

How to interact with peripherals

We need to use C code to set the value of SFR.

SFR (special function registers)

These are registers that are embedded in peripherals, used for configuration and control of peripherals.

VERBOSE~
If we want to get the status of a peripheral, we read the value of SFR.
If we want to send something to peripheral, we write value to SFR.

Let’s take timer as an example:
SFR in block diagram of timer:

SFR declaration in code:

We change operate with these registers through Bit Operation.
(These are PIC32 codes but I think you have got it)

L4 (IO)

All the modes of GPIO:

The whole block diagram:

You may see an unfamiliar unit Here.
It’s used to convert the input analog voltage to digital voltage.
You can memorize it through this graph:
Transfer function of a Schmitt trigger.

The horizontal and vertical axes are input voltage and output voltage, respectively. T and −T are the switching thresholds, and M and −M are the output voltage levels.

Input

After we configured the GPIO to be input ports, the output driver is disabled (disconnected).

Pull down

• When the IO pin is connected to LOW (0V) or unconnected, the input data register will be 0.
• When the IO pin is connected to HIGH (3.3V/5V), the input data register will be 1.

Pull up

• When the IO pin is connected to LOW (0V), the input data register will be 0.
• When the IO pin is connected to HIGH (3.3V/5V) or unconnected, the input data register will be 1.

Floating

• When the IO pin is connected to LOW (0V), the input data register will be 0.
• When the IO pin is connected to HIGH (VDD), the input data register will be 1.
• When the IO pin is unconnected, the input data register will be unpredictable.

General Purpose Output

The input driver part is still enabled so that we can read the output status.

Open Drain

Can “generate” voltage higher than VDD at IO pin.

• “0” in the Output register activates the N-MOS (LOW (0V) at IO pin)
• “1” in the Output register leaves the port in Hi-Z (HIGH (V+) at IO pin)
  (the P-MOS is never activated)

Push Pull

Most common one.

• “0” in the Output register activates the N-MOS (LOW (0V) at IO pin)
• “1” in the Output register activates the P-MOS (HIGH (VDD) at IO pin)

Alternative Function Output

Not covered.

L5 (Interrupts)

Why interrupt?

Most of the peripherals take quite a few time to complete its task or trigger an event. Instead of instruct the processor to keep checking the status of these peripheral (polling), we want the peripherals to inform processor when there exists an event, so that the processor can focus on its own task.

Peripherals inform the processor through external interrupt.

Where do interrupts come from

We mainly deal with the interrupts from peripherals.

Each peripheral can have multiple interrupt sources, indicating different events.

How to handle interrupts

Through interrupt service routine (ISR)

Interrupt vectors

Interrupt vectors are addresses that inform the interrupt handler as to where to find the ISR

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

| Vector Number | Interrupt Number | Description                    | Vector Address   |
|---------------|------------------|--------------------------------|------------------|
| 0             | -                | Initial Stack Pointer          | 0x0800 0000      |
| 1             | -                | Reset Handler                  | 0x0800 0004      |
....
| 27            | 11               | DMA1 Channel1 global Interrupt | 0x0800 006C      |
| 28            | 12               | DMA1 Channel2 global Interrupt | 0x0800 0070      |
| 29            | 13               | DMA1 Channel3 global Interrupt | 0x0800 0074      |
| 30            | 14               | DMA1 Channel4 global Interrupt | 0x0800 0078      |
| 31            | 15               | DMA1 Channel5 global Interrupt | 0x0800 007C      |
| 32            | 16               | DMA1 Channel6 global Interrupt | 0x0800 0080      |
| 33            | 17               | DMA1 Channel7 global Interrupt | 0x0800 0084      |
...
| 41            | 25               | TIM1 Break Interrupt           | 0x0800 00A4      |
| 42            | 26               | TIM1 Update Interrupt          | 0x0800 00A8      |
| 43            | 27               | TIM1 Trigger and Commutation   | 0x0800 00AC      |
| 44            | 28               | TIM1 Capture Compare Interrupt | 0x0800 00B0      |
...

This is an arbitrary IVT that depicts the pattern of IVT, for detailed IVT of STM32, please refer to the reference manual.

What to do inside ISR

• Always remember to clear the interrupt flag in ISR.
• Because there is often more interrupt sources than interrupt vectors, you need to judge which source triggered this interrupt based on interrupt status register.
• Customized operation… (but be short, because if the operation take too many clock cycles, it may be interrupted by another interrupt source, which may not be on purpose)

Nested interrupts

When executing ISR, the processor can be interrupted by interrupts with higher priority.

Remainder: the lower the priority number, the higher the priority

L6 (Timer)

Let’s split Timer peripheral into 3 parts:

Blue part: Master/slave controller

The master/slave unit provides the time-base unit with the counting clock signal (for example the CK_PSC signal, PSC here means that it’s for the prescaler in time-base unit), as well as the counting direction (counting up/down) control signal.
This unit mainly provides the control signals for the time-base unit.

Yellow part: Time-base unit

The main block of the programmable timer is a 16-bit counter with its related auto-reload register.
The counter can count up, down or both up and down.
The counter clock can be divided by a prescaler (which is basically another counter).

On the reference manual there are many wave-forms for you to understand how these control registers take effects.

The reset frequency of the counter is $$\frac{f_{input}}{(Prescaler+1)\times(Counter Period+1)}$$

Red part: Timer-channels unit

The timer channels are the working elements of the timer.
They are the means by which a timer peripheral interacts with its external environment (through input capture or output compare).

How to use multiple timer for more counting digits (32-bit)

It is possible to configure one slave timer to increment its counter based on a master-timer events such as the timer update event. The master-timer event is signaled by the master timer master/slave controller unit. This controlling unit uses the master timer output-TRGO signal. The master timer output-TRGO signal is connected to the slave timer TRGI-input signal. The master/slave controller unit of the slave timer is configured to use the TRGI-input signal as clock source to increment the slave timer counter.

L7 (LCD)

Just refer to RC2_LCD. I think it’s not the focus of final exam.

L9 & L10 (Input Capture & Output Compare)

IC is a peripheral that can monitor the input signal changes (pos/neg edge) independent of the processor (Core).
OC is a peripheral that can generate precise output signal independent of the processor (Core).

In STM32, it is embedded in timer peripheral (together with output compare).

IC

You can consider IC as a timer value recorder. It will record the timer value each time the capture condition is met (you can see from the diagram, the timer value is from CNT counter).

These conditions can be:

• rising edge
• falling edge
• both
  With a prescaler, we trigger capture events every few edges.
  Similar for interrupt, we can trigger an interrupt every few captures.

Note:
The IC does not capture the edge immediately when a rising or falling edge happened. The capture event needs to be sync with PB_clk.
Further more, the module will capture the timer counter value that is valid 2-3 PB_clk cycles after the capture event.

For detailed configuration, please refer to Reference Manual.pdf on canvas, page 349-359, 382-385.

OC

Just refer to RC2_Output_Campare and RC3_Lab4 for the concepts and PWM configuration.
Also, the solution of hw2 has been uploaded to canvas, please take a look.

项目框架

以下为项目布局示意图

（左下角为深度相机）

以下为项目的架构图

具体解释一下这张图

• 首先用户佩戴 hololens，hololens 会先把可交互的物体（物体数据源于物体检测算法）渲染在 UI 上，之后用户基于这些视觉信息，给出交互指令。

• 随后系统会根据给到的指令和可交互物体的信息，规划机械臂的运动路径。

• 路径信息会首先传到数字孪生系统中。该系统中有三部分孪生，分别是人体（实时数据来自人体检测算法），环境（实时数据来自 hololens 的环境感知系统） 和 机械臂（来自于前面提到的路径信息）的孪生。

• 随后该 DT 系统会根据路径信息解算在机械臂运动过程中是否会发生碰撞。

  • 如果发生碰撞
    • 会把碰撞发生的位发还给 hololens 显示，用户可以选择取消指令或者选择移走遮挡的物体（）。
  • 如果不发生碰撞
    • 会把机械臂的预计运动轨迹发回给 hololens 显示，用于警戒用户。
    • 将运动轨迹发给机械臂控制器，控制器控制机械臂运动。

程序接口定义

Python->Unity

代码仓库：

https://github.com/Chen-Yulin/Unity-Python-UDP-Communication

传输的数据为字符串：

1
2
3

def SendData(self, strToSend):
    # Use this function to send string to C#
    self.udpSock.sendto(bytes(strToSend,'utf-8'), (self.udpIP, self.udpSendPort))

物体识别信息

需要包含的信息：物体种类，物体的三轴方位，三轴旋转，三轴尺寸。

格式：

1
2
3
4
5
6
7
8
9
10
11

{Object Detection}
category
position.x
position.y
position.z
eulerRotation.x
eulerRotation.y
eulerRotation.z
scale.x
scale.y
scale.z

示例：

1
2
3
4
5
6
7
8
9
10
11

{Object Detection}
Handle
0.1
0.052
0.026
0
90
0
0.5
0.3
1

机械臂实时角度

需要包含的信息：6 个 joint 角度(单位为度)

格式：

1
2
3
4
5
6
7
8

{Current Joint}
j0
j1
j2
j3
j4
j5
j6

Unity->Python

机械臂需要旋转的角度

需要包含的信息：6 个 joint 角度(单位为度)

格式：

1
2
3
4
5
6
7
8

{Target Joint}
j0
j1
j2
j3
j4
j5
j6

Python-Unity Network

Python 和 Unity 间通讯

https://github.com/Siliconifier/Python-Unity-Socket-Communication