• Creating a DataFrame
• Location
• type transforming
• Groupby and agg
• Example
  • Example 1

Creating a DataFrame

1
2
3
4

import pandas as pd
df = pd.read_csv("a.csv")
df = pd.DataFrame({dict_t}) #convert a dict to a dataframe
df = Series_t.to_frame()

Location

1
2
3
4
5
6
7

df.loc[0:4,"Year":"Party"]
df.loc[[1,2,5],["Year","Candidate"]]
df.loc[:,["Year","Candidate"]]
df.loc[:,"year"] # return series
df.loc[1,"year":"Party"] # return series
df.shape[0] # count of row
df[["Year","Candidate"]][0:5]

type transforming

1
2
3

df['column_name'] = df['column_name'].astype(int)
df['column_name'] = df['column_name'].astype(float)
df['column_name'] = df['column_name'].astype(str)

Groupby and agg

1
2
3

df.groupby('column_name').agg(['sum', 'mean', 'max', 'min'])
df.groupby(['column_name_1', 'column_name_2']).agg({'column_name_3': ['sum', 'mean'], 'column_name_4': ['max', 'min']})
df.groupby(['column_name_1', 'column_name_2']).agg(['sum', 'mean', 'max', 'min'])

Example

Example 1

Extracting the top 20 categories from a DataFrame using groupby:

1. Use the groupby and size methods to calculate the size of each group:

   1 2 3 4 5 6 7

   group_sizes = df.groupby('category').size() ```` 2. Use the `sort_values` method to sort the group sizes in descending order: ```python sorted_groups = group_sizes.sort_values(ascending=False)

2. Use the head method to extract the top 20 categories:

1

top_20 = sorted_groups.head(20)

4. Use the isin method to filter the original DataFrame to only include rows with the top 20 categories:

1

df_filtered = df[df['category'].isin(top_20.index)]

This will extract the top 20 categories from the ‘category’ column of the DataFrame and create a new DataFrame that only includes rows with those categories.

A Message to Myself

hi, how are you doing today?
When you wake up in the morning to the sound of your alarm clock,
when you join an online lecture and hear the hissing sound coming from your headphones,
when the flying catkins take away the smell of heather,
you always wonder why you are here. living.
This is a hard question to answer, you are an atheist, you have to find out the answer that belongs to yourself.
The past 12 months have been very memorable for you, right?
You have changed from an unknown game player to a personal mod developer with over 10,000 subscribers in the steam workshop;
You become an video publisher in bilibili and your first video gets tens of thousands of views;
You have successfully established a blog site of your own;
Based on nvim, you have customized your own fancy editor;
You build a personal knowledge base with obsidian, and you feel your knowledge connected like never before;
You have gained an unforgettable internship experience and got your first salary in life;
You have led many scientific and technological innovation teams and met many people of all kinds;
You got to know chatGPT, which makes you feel that all the knowledge in this world is at your fingertips.

It looks like everything is going well, and you’ve been moving in a positive direction, but it’s superficial, and you still haven’t figured out what you’re living for.
You often boast that you have a good mentality and a big heart, but many times you are not strong psychologically, but numb to respond to stressors.
By now, you have come to the second half of your junior year. Compared with before, the pressure on your shoulders is even greater, especially the pressure of choosing your future path.
Under such pressure, it is easy to get lost if you don’t have clear enough pointers.
So, now is the time to seriously think about what you are seeking for in life. In general, it’s smarter to infer what you’re seeking from what you’re not gonna seek.
First of all, what you are seeking is not a fake, which can be defined as concealment, prevarication, hypocrisy.
For you, personal wealth, schooling certification are all fakes. They can only guide how others judge our worth, but they cannot tell us what our own worth is.

More fundamentally, You don’t pursue the approval of others, and you don’t pursue the society’s expectations of you.
You hate the studies of how to become success, you hate cram school, and you hate these things that are concretized by distorted human nature. You think that people should have some higher purpose.
But those purpose are too high, far beyond your reach, and the devil’s voice of corruption is always in your ears, and your pain comes from it.
In fact, at this point, you already understand what your life goal is, just say it, don’t care what other people will say, at least I won’t laugh at you.
You hope that one day you will contribute, in whatever form, to the advancement of all humanity.
Just like your favorite singer, Michael Jackson, you aspire to be a man who have a world in mind like him.
But now you are too far away from this goal, and you don’t even know how to touch it, but the pressure of life will always follow, and the prejudice of society will not disappear because of your denial.
For example, now, your road to postgraduate education is not clear, what should you do if you encounter various setbacks?
I think what you should do most is to reconcile with yourself. An outstanding person may not necessarily succeed, but he will definitely recognize himself.
Often, you don’t know where the next step will lead you, and you don’t want to put yourself in a bad situation again and again, but you have to remember that I am always by your side,
No matter what you do, no matter how bad things happen, I will always be there for you and I will always approve of you.
As long as the two of us are here, we will find a way to make things better. Because we’re best friends from now on, and that’s what best friends do.

I know a guy who compiled the obsidian version of the Chinese-English thesaurus by himself. He lost his job, got myeloma, and his health and wealth were shattered, but he completed this Grand project under such extremely difficult conditions.
I would like to quote a passage he wrote in his preface:
Now looking back, it all seems like a dream. If I wasn’t poor, I wouldn’t have had the time and energy to do this; if I was rich enough, I might have done something else; if my talent was a little bigger, I wouldn’t have done this; if my talent was a little smaller, it wouldn’t have been enough to do this; the highest level of computer language I know is the most basic HTML language, which can only change the color and style of text; if I hadn’t gotten sick, I wouldn’t have expanded my vocabulary to over nine thousand words; if there was no Bilibili and no video I made myself, my work would not have been known so quickly. All of these seem predetermined, not too much or too little, just right.

When you stumble again and again on the road of life, I will always reconcile with you, and we will face the next challenge with full confidence.

Notice

Acting like my favourite friend.

What do I have (positive aspect)

• I am good at playing sandbox game, maybe a sign of innovative?
• I have learn many skills that can support me to do some individual development
• I have a healthy home environment
• I am not unhealth
• I took part in numoerous project teams and know many people, no matter good or bad
• I have the ability to improve my life through some technical skills

I have fear of what

• 我担心会把实在的当作虚妄的，把虚妄的当成实在的
• 我曾经认可所谓的人本主义教育，作为我对于应试教育的反抗，但现在看来，他也是错的。
• 我担心我一辈子都活在一个乐于对人评头论足的环境
  • 对于一个人的学历
  • 对于一个人的选择
• 我担心我做的事情没有意义
  • 对于我的意义
    • 如果我在人生的最后，回过头来看，我是否会觉得我现在做的毫无意义？
  • 对他人来说的意义
    • 这个已经克服了，但不能完全忽视
    • 比如我的父母，他们在某种程度上就是30年后的我，他们的意见应该参考
• 我担心我对于太多事情有不切实际的期待
  • 对于人的本性（善/恶）
  • 对于人的欲望
  • 对于社会舆论
    • 当我认为他人生活在信息茧房中，我首先需要自指，我自己是否生活在茧房中？
    • 当我认为他人被洗脑时，我是否有被洗脑

Replace fear with confidence

Future is full of possibility

What is my confusion?

• 个人的努力在社会发展的洪流下不值一提

• 关于个人道路的选择

  • 读博士/读硕士
    • 博士？需要大量时间，家庭日渐凋敝，读出来后专业落伍
    • 硕士？内卷，名额太少，学术成就不如博士
    • 我追求学术成就吗？我认为这是在人类史上留下个人价值的最好方式。但我了解到了太多的科研劝退内容。
    • 所谓的劝退，针对的是对于科研抱有不切实际的期望的人群
      • 我属于那个人群吗
  • 学术成就/家财万贯（成功人士）/实业兴邦（打螺丝？）
    • 中产的美梦？
    • 家财万贯不现实，目前社会的上升通道太少，创业失败即家破人亡，ai或许能凿壁偷光？
    • 学术更像是一场赌博，稳妥的办法是添砖加瓦，但这已经有许多人在做了。但许多划时代的研究都不是如此。
    • 我目前的经历更主要偏向应用，调包，调参。。。。
    • 如果是应用，我希望我是构建者而不是码农
    • 什么是构建者？
      • 例如自研一些开发软件

• 我选择的信念是否正确？

  • 我的信念是什么？
    • 在更小时候，我并不喜欢他的音乐风格
    • 在高考那段最磨练意志的时段，主要是MJ的音乐陪我走过，给了我充沛的精神力量
    • 为什么我会喜欢MJ的音乐？抗争的精神力量！
      • 对于肤色的抗争
      • 对于政治迫害的抗争
      • 对于人类欲望的抗争
      • 对于病痛的抗争
    • 那么在高考那段时间，我到底想抗争什么？
      • 极端狭窄的独木桥
      • 对学生的评价体系
      • 压抑的同龄人氛围
      • 将人物化

Generator file

File Path: catkin_ws/src/test/script/env_pkl_generator.py

Launch Ros

1
2

cd ~
roslaunch test demo_gazebo.launch

It will open Gazebo and RViz and display the robotic arm on both sofware.

Load Scene

1
2
3

cd ~
rosrun test moveit_setscene.py
rosrun test gazebo_setscene.py

远程链接

向日葵
识别吗
341 866 266
ge6v9Q

• Beautiful Soup
  • Generate bs4
  • find node
  • siblings
  • with Regular Expression

Beautiful Soup

Generate bs4

1

soup = bs4.BeautifulSoup(text)

find node

1
2
3

soup.find("div", attrs={"id":"..."})
soup.find_all("...")
soup.div

siblings

1
2
3
4

soup.find("...").next_sibling
soup.find("...").previous_sibling
for elm in soup.div.next_siblings:
    ...

with Regular Expression

1
2

soup(class_=re.compile('item-'))
soup(attrs={"class":re.compile("star-rating.*")})[0].get("class")[1]

• Syntax
• Usage
  • Use re
  • Use Pandas
• Example
  • Example 1

Syntax

regex101
regexr

Usage

Use re

1
2
3

re.sub(pattern,repl,str)
re.search(pattern, str)
re.findall(pattern,str)

Use Pandas

1
2
3

df['column_name'].str.contains('pattern')
df['column_name'].str.findall('pattern')
df['column_name'].str.replace('pattern', 'replacement')

Example

Example 1

Extracting room numbers from a ‘Description’ column in a DataFrame using regular expressions:

1. Import the re module:

   1 2 3 4 5 6 7 8 9 10 11 12

   import re ```` 2. Define a function to extract the room number from a description: ```python def extract_room_number(description): match = re.search(r'(\d+\.\d+|\d+)(?=\s+of which are bedrooms)', description) if match: return float(match.group(1)) else: return None

2. Use the apply() method to apply the function to the ‘Description’ column and create a new ‘RoomNumber’ column:

1

df['RoomNumber'] = df['Description'].apply(extract_room_number)

This will extract the room number from the ‘Description’ column and store it in the new ‘RoomNumber’ column.

Some basic operation

Shuffle the training set for hold out validation

1
2

from sklearn.utils import shuffle
training_set,dev_set = np.split(shuffle(data_sample_35),[25])

Also capable of spliting training set, validation(development) set, test set

Practicing website

sql-practice

• Environment
• Example

Environment

Install package for sql

1
2
3

pip install sqlalchemy
pip install ibm_db_sa
pip install ipython-sql

Example

Which are the build-in commands?

A shell built-in command is contained in the shell itself (implemented by shell author) and runs on the same shell without creating a new process, e.g. cd, pwd, exit, alias.

By contrast: For a regular(linux) command, it is run from a binary file (found in $PATH or specified file) by forking the existing process in a subshell, e.g. ls, cat, rm.

How to implement

1. Judge whether is build-in command

Assume we have already got the parsed arguments

char * parsedArgs[MAXPARSE].

For example, a input char * input = "cd .." can be parsed as char * parsedArgs[MAXPARSE] = {"cd", ".."}

Then, we just need to judge whether parsedArgs[0] is contained in our build-in command set {"exit", "cd", "pwd" ...} (by using strcmp)

2. Implementation

In this project (as of milestone 2) , we need to implement exit, pwd, cd.

exit

In Bash, exit is a built-in command used to terminate the current shell session (same in our mumsh).

Each running shell is a process, so we can use the exit() in stdlib.h to terminate the shell process, achieving the goal of exiting the shell.

1
2
3
4

void exec_exit(){
    printf("exit\n");
    exit(0); // 0 for exit normally, otherwise abnormally
}

pwd

Running a crude (or minimal?) shell and have no idea where you are? Just type pwd and it will print the name of the current/working directory.

In C, we can implement the similar functionality using the getcwd() in unistd.h.

if success, getcwd() will return the pointer to the result, else ( e.g. no enough space) return NULL

1
2
3
4

void exec_pwd(){
    char buf[1024]; // memory space need for storage
    printf("%s\n", getcwd(buf,sizeof(buf)));
}

However, the working directory is not guaranteed to be shorter than 1024, so we may need to dynamically allocate the buffer to suit the length.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

int size = 100;
char *buffer = malloc(size);

while (getcwd(buffer, size) == NULL) {
    if (errno == ERANGE) {  // buffer too small
        size *= 2;
        buffer = realloc(buffer, size);
    } else {  // other error
        ...
        free(buffer);
        return 1;
    }
}

printf("%s\n", buffer);

free(buffer);

cd

cd is used for changing working directory.

In C, we can implement the similar functionality using the chdir() in unistd.h.

if success, chdir() will return 0, else ( e.g. no such directory) return -1

1
2
3

void exec_cd(char * path){
    chdir(path);
}

Very straight forward, but if we want to implement more advanced features of the cd command such as changing to the home directory or the previous directory, we would need to handle these cases manually.

For example, you could check if the argument is ~ or -, and then call chdir() with the appropriate path

• 如何使用Shader实现语义分割
  • 什么是Shader
  • Shader代码的组成部分
  • 如何使用Shader实现语义分割

如何使用Shader实现语义分割

什么是Shader

Shader是一种计算机程序，它用于在图形渲染管线中处理顶点和像素。在Unity中，Shader可以用来控制物体的渲染方式，包括颜色、纹理、透明度等。Shader的工作流程是将材料（如纹理、数据、颜色等）通过加工展现在材质上的过程。

Unity中有多种类型的Shader，可以根据不同的情况使用。例如，在三维网格上使用的方式和图片上使用的方式会有区别。

Shader代码的组成部分

一个Unity Shader通常由以下几个部分组成：

• Properties：这部分定义了Shader中可供用户在材质检查器中编辑的属性。例如，可以定义一个颜色属性，让用户可以在材质检查器中更改物体的颜色。

• SubShader：每个Shader至少包含一个SubShader。SubShader定义了Shader的实际渲染操作。如果Shader需要支持多种不同的渲染硬件或渲染质量设置，可以在Shader中定义多个SubShader。

• Pass：每个SubShader至少包含一个Pass。Pass定义了一次渲染操作，包括顶点和片段着色器的代码。可以在一个SubShader中定义多个Pass来实现多次渲染操作。

• CGPROGRAM：这部分包含了顶点和片段着色器的代码。可以使用Cg/HLSL语言来编写着色器代码，并使用内置的变量和函数来访问物体的信息。

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

  Shader "Custom/SimpleColor" { Properties { _Color ("Color", Color) = (1,1,1,1) } SubShader { Tags { "RenderType"="Opaque" } LOD 100 Pass { CGPROGRAM #pragma vertex vert #pragma fragment frag fixed4 _Color; struct appdata { float4 vertex : POSITION; }; struct v2f { float4 vertex : SV_POSITION; }; v2f vert (appdata v) { v2f o; o.vertex = UnityObjectToClipPos(v.vertex); return o; } fixed4 frag (v2f i) : SV_Target { return _Color; } ENDCG } } }

如何使用Shader实现语义分割

如果希望根据物体的标签来控制物体的颜色，可以在Shader中使用材质属性来实现。首先，需要在脚本中获取物体的标签信息，并根据标签信息设置物体材质的颜色属性。然后，在Shader中，可以使用这个颜色属性来控制物体的渲染颜色。

下面是一个简单的示例，它展示了如何在脚本中设置材质的颜色属性，并在Shader中使用该属性来控制物体的颜色：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

// 脚本中设置材质的颜色属性
void Start()
{
    Renderer renderer = GetComponent<Renderer>();
    Material material = renderer.material;

    // 根据物体的标签设置材质的颜色属性
    if (CompareTag("Pedestrian"))
    {
        material.SetColor("_Color", Color.blue);
    }
    else if (CompareTag("Road"))
    {
        material.SetColor("_Color", Color.white);
    }
    else if (CompareTag("Building"))
    {
        material.SetColor("_Color", Color.black);
    }
}
````

```c
// Shader中使用颜色属性来控制物体的颜色
Shader "Custom/SemanticSegmentation" {
    Properties {
        _Color ("Color", Color) = (1,1,1,1)
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            fixed4 _Color;

            struct appdata {
                float4 vertex : POSITION;
            };

            struct v2f {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v) {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target {
                return _Color;
            }
            ENDCG
        }
    }
}