Repository:
https://github.com/PSGBOT/pixtral-12B-Inference

本地图片上传

1
2
3
4
5
6
7
8
9
10
11

def encode_image(image_path):
    """Encode the image to base64."""
    try:
        with open(image_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode('utf-8')
    except FileNotFoundError:
        print(f"Error: The file {image_path} was not found.")
        return None
    except Exception as e:  # Added general exception handling
        print(f"Error: {e}")
        return None

Prompt

VLM物体描述的prompt:

核心需要：准确定位物体所在方位，不把远景识别为物体，降低False Positive

1
2
3
4
5
6
7
8
9
10
11
12
13

Focus on the area highlighted in green in the image.

Step 1: Determine if the highlighted area represents a distinct, identifiable object or instance:
- If the highlighted area is clearly a distinct object, proceed to Step 2.
- If the highlighted area is abstract, ambiguous, or you cannot confidently identify it as a specific object (e.g., part of background, texture, partial view), respond with "Valid: No".

Step 2: If the highlighted area is a distinct object, provide:
1. The specific name of the object (be precise and use technical terms when appropriate)
2. The primary function or purpose of this object
3. Any notable features visible in the highlighted area (no color description)
4. If there is text visible on the object, include what it says

Remember, if you're uncertain about the highlighted area being a distinct object, respond only with "Valid: No".

输出结果：

• Valid

  1 2 3 4 5 6 7 8 9

  Valid: Yes 1. The specific name of the object: Soap dispenser 2. The primary function or purpose of this object: To dispense liquid soap or hand sanitizer. 3. Notable features visible in the highlighted area: - The dispenser has a pump mechanism at the top. - The body of the dispenser is cylindrical. - The material appears to be translucent plastic. 4. There is no visible text on the object.

• invalid

  1

  Valid: No

VLM输出->Structured Output

使用另一个LLM来对VLM输出的内容进行parse，转化成json文件, 通过mistral ai 提供的接口实现:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

class Instance(BaseModel):
    valid: str
    name: Optional[str] = None
    feature: Optional[List[str]] = Field(default_factory=list)
    usage: Optional[List[str]] = Field(default_factory=list)

def parse_description_msg(msg):
    message = [
        {"role": "system", "content": "Extract the description information."},
        {
            "role": "user",
            "content": msg,
        },
    ]
    return message

chat_response = self.client.chat.parse(
	model=self.llm,
	messages=msg,
	response_format=Instance,
	max_tokens=self.llm_max_tokens,
	temperature=self.llm_temperature,
)
return json.loads(chat_response.choices[0].message.content)

https://pan.sjtu.edu.cn/web/desktop/personalSpace?path=Disney

https://pan.sjtu.edu.cn/web/share/268e334a73ed7a82daca26aecf2bfd67

使用mpm安装(https://wiki.archlinux.org/title/MATLAB)
Download mpm from https://www.mathworks.com/mpm/glnxa64/mpm and make it executable.

安装：

1

./mpm install --release=R2024b --destination=/home/cyl/matlab MATLAB

安装后启动完成lisense注册后使用patch(https://bbs.archlinux.org/viewtopic.php?id=303177)

1
2

patchelf --clear-execstack /home/user/.MathWorks/ServiceHost/-mw_shared_installs/v2024.13.0.2/bin/glnxa64/libmwfoundation_crash_handling.so
patchelf --clear-execstack /home/user/.MathWorks/ServiceHost/-mw_shared_installs/v2024.13.0.2/bin/glnxa64/mathworksservicehost/rcf/matlabconnector/serviceprocess/rcf/service/libmwmshrcfservice.so # may not needed

如果出现空白窗口左下显示ready，那么参考(www.reddit.com/r/matlab/comments/1dhejp5/matlab_gui_not_loading_properly_on_arch/)，设置环境变量

1

export _JAVA_AWT_WM_NONREPARENTING=1

Sources

Single Instance

• Image Classification - 32 Classes - Fourniture: About 10K

Complicated Scene

• Real
  • Indoor Training Set (ITS) [RESIDE-Standard]: 1.4K
  • MIT Indoor Scenes: 15K
• Synthetic
  • InteriorVerse

chair_dataset

• image_1~image_3000: kaggle furniture image dataset
• image_3001~image_4887: DeepFurniture

desk_dataset

• image_1~image_700: pix3d

home_appliance_dataset

• image_1~image_3000: kaggle furniture image dataset fridge only
• image_3001~image_3429: DeepFurniture home-appliance category

shelves_dataset

• image_1~image_3000: kaggle furniture image dataset
• image_3001~image_3243: pix3d wardrobe category
• image_3244~image_3604: pix3d bookcase category

sofa_dataset

• image_1~image_1947: pix3d
• image_1498~image_3888: DeepFurniture

table_dataset

• image_1~image_3000: kaggle furniture image dataset
• image_3001~image_4870: pix3d
• image_4871~image_7293: DeepFurniture

tool_dataset

• image_1~image_115: pix3d
• image_116~image_1441:kaggle mechanical tool dataset hammer
• image_1442~image_1812:kaggle mechanical tool dataset plier
• image_1813~image_3138:kaggle mechanical tool dataset screw driver
• image_3139~image_4469:kaggle mechanical tool dataset wrench

tv_dataset

• image_1~image_3000: kaggle furniture image dataset

PSR Dataset

Cabinet

from shelves_dataset 3000 images -> ? train & ? val samples

Desk (Processing)

from desk_dataset 699 images -> ? train & ? val samples

Tool (Processing)

from tool_dataset (simplified) 1335 images -> 2729 train & 911 val samples

Furniture (Processing)

from 130k Images/furniture 1983 images -> ? train & ? val samples

Coco (Processing)

from coco2017 2212 images -> ? train & ? val samples

current total:

• train: 12,415-686(bg)=11729
• val: 3,028-182(bg)=2846

https://www.youtube.com/watch?app=desktop&v=HVcTPeitxmw