Track

Routine

爱心屋签到： aixinwu.sjtu.edu.cn/products/asw-store
每日二GRISSO💊

Starting from 2/20:

Non-routine

论文

Due

Moved to [[2025 Fall Schedule]]

No Due

Moved to [[2025 Fall Schedule]]

有生之年系列

Moved to [[2025 Fall Schedule]]

Archived

• AMD 2025冬令营 1/16 🔒Done! 2025-01-19 🕸️ Routine > Non-routine > Due
• 凉水彩 🔒Done! 2025-01-19 🕸️ Routine > Non-routine > Due > No Due
• wyf 生日 1/22 🔒Done! 2025-01-24 🕸️ Routine > Non-routine > Due
• JI VPN application 🔒Done! 2025-01-24 🕸️ Routine > Non-routine > Due > No Due
• 糕生日 2/11 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > Due
• WW2-Naval Mod 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
  • 潜艇
    • 深弹
    • 水箱毁伤
    • 水听
    • 视距外脱锁
    • 罗盘UI
    • 声纳
    • 发电机启停UI
    • 备用鱼雷
  • 舰载机修复
    • 水机回收
    • 机库
• 取邮政快递 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
• 《悉达多》[[《悉达多》 读书会p14]] 🔒Done! 2025-02-13 🕸️ Routine > Non-routine > 论文 > No Due
• Scene-LLm 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• 3D-LLM 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• Transformer from scratch 2/14 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > Due
• NSFC 环境重建大纲 2/18 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > Due
• 场景语义化的文献归纳 https://sjtu.feishu.cn/docx/DV4Sd0wM8oFGe9xFokJcHHJ9nlf 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文 > No Due
• DINO 🔒Done! 2025-02-19 🕸️ Routine > Non-routine > 论文
• PointLLM 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• ProgPrompt9 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• SayPlan 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• Grounding DINO 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• DELTA 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• MaskCLIP 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• ConceptFusion 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文
• 春季选课 Registration 2/28 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > Due
• 本科学分转换申请 文件 2/28 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > Due
• Holo-Arm PRP application sheet 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > No Due
• 《禅与摩托车的维修艺术》[[《禅与摩托车维修艺术》读书会p10]] 🔒Done! 2025-03-09 🕸️ Routine > Non-routine > 论文 > No Due
• 知识图谱表格细节完善 3/10 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > Due
• 褴褛飞旋电吉他 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > No Due
• DeepSeek from scratch 2/20 (deleted) 🔒Done! 2025-03-12 🕸️ Routine > Non-routine > 论文 > Due
• 场景语义理解/重建 文献综述，表格 3/13 🔒Done! 2025-03-21 🕸️ Routine > Non-routine > 论文 > Due
• 给陈天华发上海大学生机械工程创新大赛证书 3/22 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > Due
• 大众餐厅草莓 3/24 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > Due
• 《素食者》读书会p15 🔒Done! 2025-03-23 🕸️ Routine > Non-routine > 论文 > No Due
• 医保扫码补缴 3/29~3/30 🔒Done! 2025-03-31 🕸️ Routine > Non-routine > 论文 > Due
• PRP Application 3/30 🔒Done! 2025-03-31 🕸️ Routine > Non-routine > 论文 > Due
• 自然辩证法pre 4/2 🔒Done! 2025-04-03 🕸️ Routine > Non-routine > 论文 > Due
• 服务器shell配置 🔒Done! 2025-04-03 🕸️ Routine > Non-routine > 论文 > No Due
• 计算机视觉hw1 4/10 🔒Done! 2025-04-07 🕸️ Non-routine > 论文 > Due
• 数据挖掘pre registration 4/8 🔒Done! 2025-04-08 🕸️ Routine > Non-routine > 论文 > Due
• 加入雪糕的steam家庭成员 🔒Done! 2025-04-09 🕸️ Routine > Non-routine > 论文 > Due
• 确定490最终人选 4/10 (顾，孙一豪，安瑞童) 🔒Done! 2025-04-12 🕸️ Track > Routine > Non-routine > Due
• 数据挖掘pre 4/10 🔒Done! 2025-04-12 🕸️ Track > Routine > Non-routine > Due
• 博客三周年纪念 4/10 🔒Done! 2025-04-12 🕸️ Track > Routine > Non-routine > Due
• 4903 kickoff 信息通知 🔒Done! 2025-04-15 🕸️ Track > Routine > Non-routine > Due
• 《简单的激情》读书会p19 🔒Done! 2025-04-15 🕸️ Track > Routine > Non-routine > No Due
• 俞国雄 🔒Done! 2025-04-15 🕸️ Track
• meet 毕 11:00 4/17 🔒Done! 2025-04-17 🕸️ Track > Routine > Non-routine > Due
• 490 kickoff meeting 4/17 🔒Done! 2025-04-18 🕸️ Track > Routine > Non-routine > Due
  • background
  • 分工
    • 核心模块复现
    • 文献综述，针对要解决的问题(已经做了什么工作，我们怎么解决,inspiration)，针对大背景(Introduction)
      • 限定范围
    • 数据准备
• 中午羽毛球 4/18 🔒Done! 2025-04-18 🕸️ Track > Routine > Non-routine > Due
• meet wyf 4/19 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• CV meeting 4/19 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• 计算机视觉 project proposal 4/24 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• 数据挖掘hw1 4/20 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• 490 论文下发 ed.1 🔒Done! 2025-04-21 🕸️ Track > Routine > Non-routine > Due
• connected paper调研 🔒Done! 2025-04-25 🕸️ Track > Routine > Non-routine > Due
• Seminar 4/25 10:00 🔒Done! 2025-04-25 🕸️ Track > Routine > Non-routine > Due
• 夏季选课 (10:00 on April 25 to 13:00 on April 28) 🔒Done! 2025-04-25 🕸️ Track > Routine > Non-routine > Due
• CenterNet 🔒Done! 2025-05-01 🕸️ Track > Routine > Non-routine > 论文
• 划船 4/26 10:00 🔒Done! 2025-05-01 🕸️ Track > Routine > Non-routine > Due
• 孤独摇滚剧场版 4/26 11:35 🔒Done! 2025-05-01 🕸️ Track > Routine > Non-routine > Due
• 桶川读书会 4/26 19:00 🔒Done! 2025-05-01 🕸️ Track > Routine > Non-routine > Due
• 千岛湖 4/27 🔒Done! 2025-05-01 🕸️ Track > Routine > Non-routine > Due
  • 记得带便携烤架
  • 攻略
  • 酒店
  • 车票
• 数据挖掘hw2 5/5 (4/25) 🔒Done! 2025-05-01 🕸️ Track > Routine > Non-routine > Due
• 筹备老妈生日 5/26 🔒Done! 2025-05-03 🕸️ Track > Routine > Non-routine > Due
• 并行报告 5/3 🔒Done! 2025-05-03 🕸️ Track > Routine > Non-routine > Due
• 计算机视觉hw2 5/1 🔒Done! 2025-05-03 🕸️ Track > Routine > Non-routine > Due
• 老妈生日礼物：运动耳机 🔒Done! 2025-05-03 🕸️ Track > Routine > Non-routine > No Due
• 乒乓球拍 🔒Done! 2025-05-03 🕸️ Track > Routine > Non-routine > No Due
• 《负负得正》 🔒Done! 2025-05-03 🕸️ Track > Routine > Non-routine > No Due
• 《桶川跟踪狂杀人事件》读书会p17 🔒Done! 2025-05-05 🕸️ Track > Routine > Non-routine > No Due
• 自然辩证法语音转录 🔒Done! 2025-05-08 🕸️ Track > Routine > Non-routine > Due
• 小仓鼠衣服洗烘 🔒Done! 2025-05-10 🕸️ Track > Routine > Non-routine > Due
• 自然辩证法考试 5/9 18:00 dxy102 🔒Done! 2025-05-10 🕸️ Track > Routine > Non-routine > Due
• AR论文修改 5/10 🔒Done! 2025-05-10 🕸️ Track > Routine > Non-routine > Due

伪代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# f_q, f_k: encoder networks for query and key 
# queue: dictionary as a queue of K keys (CxK) 
# m: momentum 
# t: temperature  

f_k.params = f_q.params # initialize 
for x in loader: # load a minibatch x with N samples 
	x_q = aug(x) # a randomly augmented version 
	x_k = aug(x) # another randomly augmented version  
	q = f_q.forward(x_q) # queries: NxC 
	k = f_k.forward(x_k) # keys: NxC 
	k = k.detach() # no gradient to keys  
	
	# positive logits: Nx1 
	l_pos = bmm(q.view(N,1,C), k.view(N,C,1))  # 相当于把batch中每个正样本对之间求了cosine临近
	
	# negative logits: NxK 
	l_neg = mm(q.view(N,C), queue.view(C,K))  
	
	# logits: Nx(1+K) 
	logits = cat([l_pos, l_neg], dim=1)  
	
	# contrastive loss, Eqn.(1) 
		labels = zeros(N) # positives are the 0-th，将识别的类别视为0,可以直接使用CrossEntropyLoss
	loss = CrossEntropyLoss(logits/t, labels)  
	
	# SGD update: query network 
	loss.backward() 
	update(f_q.params)  
	
	# momentum update: key network 
	f_k.params = m*f_k.params+(1-m)*f_q.params  
	
	# update dictionary 
	enqueue(queue, k) # enqueue the current minibatch 
	dequeue(queue) # dequeue the earliest minibatch

亮点

Dictionary as a queue

在使用key encoder(momentum encoder)创建负样本，并把encode过的负样本存在一个queue（FIFO）中方便后续对比时直接使用，每次训练都会使用一个新的mini batch，此时会将此mini batch中的样本encode之后加入queue并删除存在最久的那个mini batch的样本（因为考虑到最老的mini batch使用的encoder是最过时的，所以FIFO是非常合理的），这样可以有效控制负样本的数量，也就是公式中的K。

• 节省字典的计算开销
• 而且mini batch大小可以直接和负样本脱钩

Momentum update

因为负样本数量（字典/队列）很大，所以没办法给key encoder回传梯度，所以可以考虑把query encoder的参数直接复制给key encoder，但过快改变的key encoder会导致样本字典的特征不一致，所以使用动量更新的方式。

过往工作对比

b)
只有一个编码器进行学习。Memory bank存下了所有样本的key。每当梯度回传后，会把memory bank被本次训练中被采样过的key使用新的encoder进行更新。

• 缺乏特帧一致性
• 需要训练一阵个epoch才能更新一遍memory bank

MoCo和memory bank 更接近，但是使用了queue dictionary和momentum update

https://www.youtube.com/watch?v=j3VNqtJUoz0&t=16s

核心思想：

• 将图像分为patches, 线性映射, 再加上图片的position embeding来输入transformer encoder
• 额外使用一个cls token用于占位（ViT的输出就是这个cls input token对应的output token）

https://github.com/facebookresearch/dino/tree/main

https://juejin.cn/post/7224738994825789496
https://www.youtube.com/watch?v=h3ij3F3cPIk&t=1005s
DI+NO（蒸馏+No Label）
具体来说，DINO 是使用一种称为“无监督自蒸馏”的方法，该方法通过自监督学习来学习模型的知识表示。在这个方法中，模型使用自身的输出来生成“伪标签”，然后使用这些伪标签来重新训练模型，从而进一步提高模型的性能和泛化能力。

知识蒸馏

https://blog.csdn.net/xbinworld/article/details/83063726

重点idea就是提出用soft target来辅助hard target一起训练，而soft target来自于大模型的预测输出。这里有人会问，明明true label（hard target）是完全正确的，为什么还要soft target呢？
hard target 包含的信息量（信息熵）很低，soft target包含的信息量大，拥有不同类之间关系的信息（比如同时分类驴和马的时候，尽管某张图片是马，但是soft target就不会像hard target 那样只有马的index处的值为1，其余为0，而是在驴的部分也会有概率。）[5]
这样的好处是，这个图像可能更像驴，而不会去像汽车或者狗之类的，而这样的soft信息存在于概率中，以及label之间的高低相似性都存在于soft target中。但是如果soft targe是像这样的信息[0.98 0.01 0.01]，就意义不大了，所以需要在softmax中增加温度参数T（这个设置在最终训练完之后的推理中是不需要的）

ViT

DINO

关于DINO中发生的涌现
https://juejin.cn/post/7280436457142501388

DINO之前的工作

We have also seen emerged two properties that can be leveraged in future applications: the quality of the features in k-NN classification has a potential for image retrieval. The presence of information about the scene layout in the features can also benefit weakly supervised image segmentation.

https://blog.csdn.net/h661975/article/details/135116957

loss: ITC (Image Text Contrastive)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# image_encoder - ResNet or Vision Transformer 
# text_encoder - CBOW or Text Transformer 
# I[n, h, w, c] - minibatch of aligned images 
# T[n, l] - minibatch of aligned texts 
# W_i[d_i, d_e] - learned proj of image to embed 
# W_t[d_t, d_e] - learned proj of text to embed 
# t - learned temperature parameter  

# extract feature representations of each modality 
I_f = image_encoder(I) #[n, d_i] 
T_f = text_encoder(T) #[n, d_t]  

# joint multimodal embedding [n, d_e] 
I_e = l2_normalize(np.dot(I_f, W_i), axis=1) T
_e = l2_normalize(np.dot(T_f, W_t), axis=1)  

# scaled pairwise cosine similarities [n, n] 
logits = np.dot(I_e, T_e.T) * np.exp(t)  

# symmetric loss function 
labels = np.arange(n) 
loss_i = cross_entropy_loss(logits, labels, axis=0) 
loss_t = cross_entropy_loss(logits, labels, axis=1) 
loss = (loss_i + loss_t)/2

Cross_entropy_loss:

CLIP 本质上是全局图像嵌入，不利于像素对齐特征提取。