01:

01:

Exercise 1

(a) Polynomial Interpolation

To approximate the function $$f(x) = \frac{1}{1 + \exp(4x)} $$on the interval [-5, 5], polynomial interpolation was performed using polynomials of degree n=6 and n=14. Two types of nodes were considered: equally spaced nodes and Clenshaw-Curtis nodes, the latter computed using the formula

$$
x_i = \frac{a + b}{2} - \frac{b - a}{2} \cos\left(\pi \frac{i}{n}\right), \quad i = 0, \dots, n.
$$

Using polyfit, the interpolation polynomials were computed based on the function values at the chosen nodes. The polynomials were evaluated using polyval on a fine grid over [-5, 5] and plotted alongside the true function f(x). The results showed that interpolation with Clenshaw-Curtis nodes provided better approximation near the boundaries, especially for n=14, reducing the Runge phenomenon that is prominent with equally spaced nodes.

(b) Piecewise Linear Interpolation

For the same values of n=6 and n=14, piecewise linear interpolation was carried out using interp1 with equally spaced nodes. The linear interpolant $$p_{1,h}$$ was constructed over n subintervals of length $$h = \frac{10}{n}$$. The resulting piecewise linear functions were evaluated on a fine grid and compared graphically with the original function f(x). While less smooth than the high-degree polynomials, the linear interpolation captured the overall trend of the function and showed more stable behavior near the interval edges.

(c) Error Analysis

To quantitatively assess the interpolation accuracy, the maximum approximation error

$$
E_n = \max_{x \in [-5, 5]} |f(x) - p_n(x)|
$$

was computed for n=2 to 40 for both equally spaced and Clenshaw-Curtis nodes. Similarly, the maximum error for piecewise linear interpolation, $$E_{1,h}$$, was evaluated for the same range of n. A loop was used to iterate over n, computing and storing the errors. The results were visualized on a log-scale plot with respect to n. The plots clearly demonstrated that Clenshaw-Curtis interpolation achieves significantly better accuracy with increasing n, while equally spaced nodes lead to instability and large errors as n grows. Piecewise linear interpolation, although low-order, exhibited stable and predictable convergence.

Exercise 2

(a) Equally Distributed Nodes

The function $f(x) = \sin(x) + x$ was interpolated on the interval [0,10] using Lagrange polynomial interpolation. Two degrees were considered: $n = 4$ and $n = 15$, corresponding to $n + 1$ equally spaced nodes. For each value of n, two datasets were created: one using the exact function values $y_i = f(x_i)$, and the other using perturbed values $z_i = y_i + \epsilon_i$, where $\epsilon_i \sim \text{Uniform}(-0.1, 0.1)$. Using polyfit and polyval, the interpolation polynomials were computed and evaluated on a dense grid for visualization.

The results show that for n = 4, both the unperturbed and perturbed interpolants approximate the true function reasonably well, though the perturbed one deviates slightly. For n = 15, the interpolant using exact values exhibits oscillatory behavior near the boundaries, a clear manifestation of the Runge phenomenon. The polynomial interpolant using noisy data becomes highly unstable, demonstrating extreme sensitivity to small perturbations in the input values.

(b) Clenshaw-Curtis Nodes

The same procedure was repeated using Clenshaw-Curtis nodes computed using the cosine-based formula. For both n = 4 and n = 15, interpolation polynomials were fitted using the exact and perturbed data values. When visualized, the interpolants using Clenshaw-Curtis nodes performed significantly better near the boundaries compared to those using equally spaced nodes. Particularly for n = 15, the polynomial interpolant with exact values closely follows the true function, with reduced oscillations. Even when perturbations are introduced, the polynomial remains relatively stable and does not exhibit the extreme sensitivity observed in the equally spaced case. This confirms that Clenshaw-Curtis nodes enhance numerical stability in high-degree interpolation.

(c) Piecewise Linear Interpolation

Piecewise linear interpolation was implemented using interp1 on equally spaced nodes for both n = 4 and n = 15. As before, both the exact and perturbed datasets were used to construct the interpolants. The linear interpolants followed the general shape of the original function and remained visually close to the true curve even when noise was added. This behavior was consistent across both values of n. While less accurate than high-degree polynomial interpolation in smooth regions, the piecewise linear approach showed strong robustness to noise and did not exhibit oscillations or instability, making it a reliable method for interpolating noisy data.

Exercise 3

(a) Least Squares Polynomial of Degree 4

The file data1.mat was loaded to obtain the dataset $(x_i, y_i)$, where $y_i = \sin(x_i) + x_i + \epsilon_i$, and $epsilon_i$ represents Gaussian noise with zero mean and standard deviation $σ=0.1$. A degree-4 least squares polynomial $p^{LS}_4$ was fitted using polyfit(x, y, 4). On the same plot, the original function $f(x) = \sin(x) + x$, the noisy data points $(x_i, y_i)$, and the polynomial $p^{LS}_4(x)$ were displayed. The polynomial provided a smooth approximation that followed the general trend of the function, mitigating the effect of noise.

(b) Higher-Degree Least Squares Polynomials (m = 7, 15)

The procedure was repeated for polynomial degrees $m = 7$ and $m = 15$. For $m = 7$, the approximation improved slightly, with the polynomial capturing more curvature of the underlying function. However, at $m = 15$, overfitting became evident—the polynomial began to follow the noise, especially near the edges of the interval. This illustrates the typical variance-bias tradeoff in polynomial regression: higher degrees reduce bias but increase sensitivity to noise.

(c) Approximation Error vs. Degree

To assess the accuracy of the least squares fit, the error $E(p^{LS}m, f) = \max{x \in [0, 10]} |f(x) - p^{LS}_m(x)|$

was computed for degrees m = 1 to 1515, evaluated over a fine grid. The error was then plotted as a function of m on a semi-logarithmic scale. The plot showed a decreasing trend in the error for small m, reaching a minimum at some intermediate degree. Beyond that, the error increased, reflecting overfitting. This confirms that, for noisy data, the optimal degree is not necessarily high.

(d) Noise Variance Estimation and Error Comparison

The variance of the noise was estimated using the formula

$\hat{\sigma}^2 = \frac{1}{n - m} \sum_{i=1}^{n+1} \left( y_i - p^{LS}_m(x_i) \right)^2,$

with m = 4 and n + 1 = 20. The estimated variance $\hat{\sigma}$ was then used to compute the expected approximation error scale

$\hat{\sigma} \cdot \sqrt{\frac{m + 1}{n + 1}}.$

The previously observed minimal error matched this estimate in magnitude, validating the analytical approximation. A comparison with the theoretical value $\sigma \cdot \sqrt{(m+1)/(n+1)}$ was also performed, showing good agreement.

(e) Large-Scale Data: n + 1 = 20000

The analysis from parts (a) to (d) was repeated using the dataset in data2.mat, which contains n + 1 = 20000 samples. The results demonstrated a dramatic improvement in the stability and accuracy of least squares fitting. Overfitting effects were less pronounced even at higher polynomial degrees due to the large number of samples. The error curve in (c) showed a more gradual and consistent decrease, and the noise influence was better averaged out. Variance: 0.04814, Bound ≈ 0.00347

(f) Error Comparison Between Small and Large Datasets

The minimal approximation error achieved using the larger dataset was significantly smaller than that obtained from the 20-point dataset. This reflects the benefit of larger sample sizes in reducing noise impact and improving model robustness. Moreover, the estimated noise error scale $\hat{\sigma} \cdot \sqrt{(m+1)/(n+1)}$ decreased accordingly due to the much larger denominator, reinforcing the statistical intuition.

Exercise 4

the Lagrange interpolation error at any point x in the interval containing the nodes is given by

$f(x) - p_n(x) = \frac{f^{(n+1)}(\xi)}{(n+1)!} \prod_{k=0}^n (x - x_k)$
Taking the absolute value yields the error bound
$|f(x) - p_n(x)| \leq \frac{M}{(n+1)!} \left| \prod_{k=0}^n (x - x_k) \right|$

where $p_n(x)$ is the interpolating polynomial of degree n through n+1 points. For both functions, the interpolation degree was set to n=4, which requires the fifth derivative of the target function.

For the function $f_1(x) = \cosh(x)$, the interpolation nodes were defined as $x_k = -1 + \frac{k}{2}$for $k = 0, 1, \dots$, giving equally spaced points in the interval [−1,1]. The fifth derivative of $\cosh(x)$ is $\sinh(x)$, since derivatives of hyperbolic functions follow a repeating pattern. The maximum of $|\sinh(x)|$ over[−1,1] occurs at x = 1, where $\sinh(1) \approx 1.175$. Substituting into the error formula, the upper bound depends on this derivative value and the absolute value of the product of linear terms $(x - x_k),$ divided by $5! = 120$.

For the function $f_2(x) = \cos(x) + \sin(x)$, the interpolation nodes were chosen as $x_k = -\frac{\pi}{2} + \frac{\pi k}{4}$ for $k = 0, \dots, 4$, covering the interval [−2π,2π]. The fifth derivative of $f_2$ is $-\cos(x) + \sin(x)$, based on the periodic differentiation cycle of sine and cosine. The maximum of the absolute value of this expression was found numerically over the interpolation interval. This value was then used in the same error formula as above, with the factorial and the product of distances from the interpolation nodes.

In both cases, the error bound reflects how the function’s higher-order smoothness and the choice of interpolation nodes influence the potential deviation between the true function and its polynomial approximation. The approach confirms that even for smooth functions, the interpolation error can grow significantly away from the nodes if the derivatives are large or if the node distribution leads to large oscillations in the interpolating polynomial.

Repository:
https://github.com/PSGBOT/pixtral-12B-Inference

本地图片上传

1
2
3
4
5
6
7
8
9
10
11

def encode_image(image_path):
    """Encode the image to base64."""
    try:
        with open(image_path, "rb") as image_file:
            return base64.b64encode(image_file.read()).decode('utf-8')
    except FileNotFoundError:
        print(f"Error: The file {image_path} was not found.")
        return None
    except Exception as e:  # Added general exception handling
        print(f"Error: {e}")
        return None

Prompt

VLM物体描述的prompt:

核心需要：准确定位物体所在方位，不把远景识别为物体，降低False Positive

1
2
3
4
5
6
7
8
9
10
11
12
13

Focus on the area highlighted in green in the image.

Step 1: Determine if the highlighted area represents a distinct, identifiable object or instance:
- If the highlighted area is clearly a distinct object, proceed to Step 2.
- If the highlighted area is abstract, ambiguous, or you cannot confidently identify it as a specific object (e.g., part of background, texture, partial view), respond with "Valid: No".

Step 2: If the highlighted area is a distinct object, provide:
1. The specific name of the object (be precise and use technical terms when appropriate)
2. The primary function or purpose of this object
3. Any notable features visible in the highlighted area (no color description)
4. If there is text visible on the object, include what it says

Remember, if you're uncertain about the highlighted area being a distinct object, respond only with "Valid: No".

输出结果：

• Valid

  1 2 3 4 5 6 7 8 9

  Valid: Yes 1. The specific name of the object: Soap dispenser 2. The primary function or purpose of this object: To dispense liquid soap or hand sanitizer. 3. Notable features visible in the highlighted area: - The dispenser has a pump mechanism at the top. - The body of the dispenser is cylindrical. - The material appears to be translucent plastic. 4. There is no visible text on the object.

• invalid

  1

  Valid: No

VLM输出->Structured Output

使用另一个LLM来对VLM输出的内容进行parse，转化成json文件, 通过mistral ai 提供的接口实现:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

class Instance(BaseModel):
    valid: str
    name: Optional[str] = None
    feature: Optional[List[str]] = Field(default_factory=list)
    usage: Optional[List[str]] = Field(default_factory=list)

def parse_description_msg(msg):
    message = [
        {"role": "system", "content": "Extract the description information."},
        {
            "role": "user",
            "content": msg,
        },
    ]
    return message

chat_response = self.client.chat.parse(
	model=self.llm,
	messages=msg,
	response_format=Instance,
	max_tokens=self.llm_max_tokens,
	temperature=self.llm_temperature,
)
return json.loads(chat_response.choices[0].message.content)

https://pan.sjtu.edu.cn/web/desktop/personalSpace?path=Disney

https://pan.sjtu.edu.cn/web/share/268e334a73ed7a82daca26aecf2bfd67

https://bbs.archlinux.org/viewtopic.php?id=303177

使用mpm安装
安装后需要使用patch

1
2

patchelf --clear-execstack /home/user/.MathWorks/ServiceHost/-mw_shared_installs/v2024.13.0.2/bin/glnxa64/libmwfoundation_crash_handling.so
patchelf --clear-execstack /home/user/.MathWorks/ServiceHost/-mw_shared_installs/v2024.13.0.2/bin/glnxa64/mathworksservicehost/rcf/matlabconnector/serviceprocess/rcf/service/libmwmshrcfservice.so

Single Instance

• Image Classification - 32 Classes - Fourniture: About 10K

Complicated Scene

• Real
  • Indoor Training Set (ITS) [RESIDE-Standard]: 1.4K
  • MIT Indoor Scenes: 15K
• Synthetic
  • InteriorVerse

https://www.youtube.com/watch?app=desktop&v=HVcTPeitxmw

Download pkg from https://apps.cloud.blackmagicdesign.com/davinci-resolve

1
2
3
4

git clone https://aur.archlinux.org/davinci-resolve.git
cd davinci-resolve
mv ~/Downloads/DaVinci_Resolve_19.1.4_Linux.zip ./
makepkg -si

FCSGG Repository Summary

FCSGG (Fully Convolutional Scene Graph Generation) is a PyTorch implementation of the paper “Fully Convolutional Scene Graph Generation” published in CVPR 2021. The project focuses on scene graph generation, which is the task of detecting objects in an image and identifying the relationships between them.

Core Components:

1. Architecture:

   • Built on Detectron2, a popular object detection framework by Facebook
   • Uses a one-stage detector approach (CenterNet) as the meta-architecture
   • Supports various backbones including ResNet, HRNet (High-Resolution Network), Hourglass networks, and DLA

2. Key Features:

   • Fully convolutional approach to scene graph generation
   • Multiple backbone options with different feature pyramid networks (FPN, BiFPN, HRFPN)
   • Various head designs including multiscale heads and attention mechanisms
   • Support for different input resolutions and training strategies

3. Dataset:

   • Primarily designed for the Visual Genome dataset, a large-scale dataset for scene understanding
   • Includes custom data loaders and preprocessing for scene graph generation

4. Model Components:

   • Backbones: Various CNN architectures (ResNet, HRNet, Hourglass, DLA)
   • Necks: Feature pyramid networks and variants (FPN, BiFPN, HRFPN, Trident)
   • Heads: Detection and relationship prediction heads
   • Loss Functions: Custom losses for object detection and relationship prediction

5. Utilities:

   • Visualization tools for scene graphs
   • Evaluation metrics for scene graph generation
   • Training and inference scripts

Project Structure:

• fcsgg/: Main module containing model implementation

  • modeling/: Neural network architecture components
    • backbone/: Feature extraction networks
    • necks/: Feature pyramid networks
    • heads/: Detection and relationship prediction heads
    • meta_arch/: High-level model architecture (CenterNet)
  • data/: Dataset handling and preprocessing
  • evaluation/: Metrics and evaluation code
  • utils/: Helper functions and utilities
  • layers/: Custom neural network layers
  • structures/: Data structures for scene graphs

• configs/: Configuration files for different model variants

• tools/: Training, evaluation, and visualization scripts

• GraphViz/: Visualization tools for scene graphs

Key Innovations:

The project implements a fully convolutional approach to scene graph generation, which differs from traditional two-stage methods. Instead of first detecting objects and then predicting relationships, it uses a one-stage detector to simultaneously predict objects and their relationships in a fully convolutional manner.

Benchmarks:

The repository provides several pre-trained models with different backbones:

1. HRNetW32-1S
2. ResNet50-4S-FPN×2
3. HRNetW48-5S-FPN×2

These models achieve competitive performance on the Visual Genome dataset for scene graph generation tasks.

Usage:

The project provides tools for training, evaluation, and visualization of scene graphs. It requires the Visual Genome dataset and can be run using Docker or directly with PyTorch.

In summary, FCSGG is a comprehensive implementation of a state-of-the-art approach to scene graph generation using fully convolutional networks, offering various model architectures and training configurations.

How Detectron2 is Used in FCSGG

FCSGG is built on top of Detectron2, Facebook’s object detection framework, and leverages many of its components while extending it for scene graph generation. Here’s a detailed breakdown:

1. Core Architecture Integration

• Meta Architecture: FCSGG registers a custom meta architecture called “CenterNet” with Detectron2’s META_ARCH_REGISTRY. This extends Detectron2’s modular architecture system while maintaining compatibility.

• Backbone Networks: FCSGG uses Detectron2’s backbone networks (ResNet, etc.) directly and also implements custom backbones like HRNet while following Detectron2’s backbone interface.

• Feature Pyramid Networks (FPN): The repository uses Detectron2’s FPN implementation and extends it with custom variants like BiFPN and HRFPN.

2. Configuration System

• YAML Configuration: FCSGG adopts Detectron2’s YAML-based configuration system, extending it with custom configurations for scene graph generation through add_fcsgg_config().

• Command Line Arguments: The training script uses Detectron2’s default_argument_parser() to maintain the same command-line interface.

3. Data Handling

• Dataset Registration: Visual Genome dataset is registered with Detectron2’s DatasetCatalog and MetadataCatalog, making it available through Detectron2’s data loading pipeline.

• Custom Dataset Mapper: FCSGG implements a custom DatasetMapper class that extends Detectron2’s mapper to handle scene graph annotations.

• Data Loaders: The repository uses Detectron2’s build_detection_train_loader and build_detection_test_loader with custom mappers.

4. Training and Evaluation

• Trainer Class: FCSGG extends Detectron2’s DefaultTrainer class to customize the training loop, evaluation metrics, and data loading.

• Checkpointing: The repository uses Detectron2’s DetectionCheckpointer for model saving and loading.

• Distributed Training: FCSGG leverages Detectron2’s distributed training utilities through detectron2.utils.comm and the launch function.

• Custom Evaluators: The repository implements a custom VGEvaluator for scene graph evaluation while following Detectron2’s evaluator interface.

5. Visualization and Logging

• Event Storage: FCSGG uses Detectron2’s event storage system for logging metrics during training.

• Visualization Tools: The repository leverages Detectron2’s visualization utilities for debugging and result analysis.

6. Extensions for Scene Graph Generation

• Custom Heads: While using Detectron2’s architecture, FCSGG implements custom prediction heads for relationship detection.

• Scene Graph Structures: The repository defines custom data structures for scene graphs that integrate with Detectron2’s Instances class.

• Loss Functions: FCSGG implements specialized loss functions for scene graph generation while maintaining compatibility with Detectron2’s loss computation framework.

7. Installation and Dependencies

• Submodule Integration: Detectron2 is included as a Git submodule, ensuring version compatibility.

• Build Process: The installation process includes building Detectron2 from source to ensure proper integration.

In summary, FCSGG uses Detectron2 as its foundation, leveraging its modular architecture, data handling, training infrastructure, and configuration system while extending it with custom components for scene graph generation. This approach allows FCSGG to benefit from Detectron2’s robust implementation and optimizations while adding specialized functionality for relationship detection between objects.