The formula recognition pipeline includes the following four modules. Each module can be trained and inferred independently and contains multiple models. For more details, please click on the respective module to view the documentation.
- [Formula Recognition Module](../module_usage/formula_recognition.en.md)
- [Layout Detection Module](../module_usage/layout_detection.en.md)（Optional）
- [Document Image Orientation Classification Module](../module_usage/doc_img_orientation_classification.en.md) （Optional）
- [Text Image Correction Module](../module_usage/text_image_unwarping.en.md) （Optional）
In this pipeline, you can choose the model you want to use based on the benchmark data provided below.
> The inference time only includes the model inference time and does not include the time for pre- or post-processing.
| Model | Model Download Link | Top-1 Acc (%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction |
|---|---|---|---|---|---|---|
| PP-LCNet_x1_0_doc_ori | Inference Model/Training Model | 99.06 | 2.62 / 0.59 | 3.24 / 1.19 | 7 | A document image classification model based on PP-LCNet_x1_0, with four categories: 0 degrees, 90 degrees, 180 degrees, and 270 degrees. |
| Model | Model Download Link | CER | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Description |
|---|---|---|---|---|---|---|
| UVDoc | Inference Model/Training Model | 0.179 | 19.05 / 19.05 | - / 869.82 | 30.3 | High-precision text image correction model |
| Model | Model Download Link | mAP(0.5) (%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction |
|---|---|---|---|---|---|---|
| PP-DocLayout_plus-L | Inference Model/Training Model | 83.2 | 53.03 / 17.23 | 634.62 / 378.32 | 126.01 | A higher-precision layout area localization model trained on a self-built dataset containing Chinese and English papers, PPT, multi-layout magazines, contracts, books, exams, ancient books and research reports using RT-DETR-L |
| Model | Model Download Link | mAP(0.5) (%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction |
|---|---|---|---|---|---|---|
| PP-DocLayout-L | Inference Model/Training Model | 90.4 | 33.59 / 33.59 | 503.01 / 251.08 | 123.76 | A high-precision layout area localization model trained on a self-built dataset containing Chinese and English papers, magazines, contracts, books, exams, and research reports using RT-DETR-L. |
| PP-DocLayout-M | Inference Model/Training Model | 75.2 | 13.03 / 4.72 | 43.39 / 24.44 | 22.578 | A layout area localization model with balanced precision and efficiency, trained on a self-built dataset containing Chinese and English papers, magazines, contracts, books, exams, and research reports using PicoDet-L. |
| PP-DocLayout-S | Inference Model/Training Model | 70.9 | 11.54 / 3.86 | 18.53 / 6.29 | 4.834 | A high-efficiency layout area localization model trained on a self-built dataset containing Chinese and English papers, magazines, contracts, books, exams, and research reports using PicoDet-S. |
| Model | Model Download Link | mAP(0.5) (%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction |
|---|---|---|---|---|---|---|
| PicoDet-S_layout_17cls | Inference Model/Training Model | 87.4 | 8.80 / 3.62 | 17.51 / 6.35 | 4.8 | A high-efficiency layout area localization model trained on a self-built dataset of Chinese and English papers, magazines, and research reports using PicoDet-S. |
| PicoDet-L_layout_17cls | Inference Model/Training Model | 89.0 | 12.60 / 10.27 | 43.70 / 24.42 | 22.6 | A balanced efficiency and precision layout area localization model trained on a self-built dataset of Chinese and English papers, magazines, and research reports using PicoDet-L. |
| RT-DETR-H_layout_17cls | Inference Model/Training Model | 98.3 | 115.29 / 101.18 | 964.75 / 964.75 | 470.2 | A high-precision layout area localization model trained on a self-built dataset of Chinese and English papers, magazines, and research reports using RT-DETR-H. |
| Model | Model Download Link | mAP(0.5) (%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction |
|---|---|---|---|---|---|---|
| PP-DocLayout-L | Inference Model/Training Model | 90.4 | 33.59 / 33.59 | 503.01 / 251.08 | 123.76 | A high-precision layout area localization model trained on a self-built dataset containing Chinese and English papers, magazines, contracts, books, exams, and research reports using RT-DETR-L. |
| PP-DocLayout-M | Inference Model/Training Model | 75.2 | 13.03 / 4.72 | 43.39 / 24.44 | 22.578 | A layout area localization model with balanced precision and efficiency, trained on a self-built dataset containing Chinese and English papers, magazines, contracts, books, exams, and research reports using PicoDet-L. |
| PP-DocLayout-S | Inference Model/Training Model | 70.9 | 11.54 / 3.86 | 18.53 / 6.29 | 4.834 | A high-efficiency layout area localization model trained on a self-built dataset containing Chinese and English papers, magazines, contracts, books, exams, and research reports using PicoDet-S. |
| Model | Model Download Link | mAP(0.5) (%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction |
|---|---|---|---|---|---|---|
| PP-DocLayout_plus-L | Inference Model/Training Model | 83.2 | 53.03 / 17.23 | 634.62 / 378.32 | 126.01 | A higher-precision layout area localization model trained on a self-built dataset containing Chinese and English papers, PPT, multi-layout magazines, contracts, books, exams, ancient books and research reports using RT-DETR-L |
| Model | Model Download Link | En-BLEU(%) | Zh-BLEU(%) | GPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
CPU Inference Time (ms) [Normal Mode / High-Performance Mode] |
Model Storage Size (MB) | Introduction | UniMERNet | Inference Model/Training Model | 85.91 | 43.50 | 1311.84 / 1311.84 | - / 8288.07 | 1530 | UniMERNet is a formula recognition model developed by Shanghai AI Lab. It uses Donut Swin as the encoder and MBartDecoder as the decoder. The model is trained on a dataset of one million samples, including simple formulas, complex formulas, scanned formulas, and handwritten formulas, significantly improving the recognition accuracy of real-world formulas. | PP-FormulaNet-S | Inference Model/Training Model | 87.00 | 45.71 | 182.25 / 182.25 | - / 254.39 | 224 | PP-FormulaNet is an advanced formula recognition model developed by the Baidu PaddlePaddle Vision Team. The PP-FormulaNet-S version uses PP-HGNetV2-B4 as its backbone network. Through parallel masking and model distillation techniques, it significantly improves inference speed while maintaining high recognition accuracy, making it suitable for applications requiring fast inference. The PP-FormulaNet-L version, on the other hand, uses Vary_VIT_B as its backbone network and is trained on a large-scale formula dataset, showing significant improvements in recognizing complex formulas compared to PP-FormulaNet-S. | PP-FormulaNet-L | Inference Model/Training Model | 90.36 | 45.78 | 1482.03 / 1482.03 | - / 3131.54 | 695 | PP-FormulaNet_plus-S | Inference Model/Training Model | 88.71 | 53.32 | 179.20 / 179.20 | - / 260.99 | 248 | PP-FormulaNet_plus is an enhanced version of the formula recognition model developed by the Baidu PaddlePaddle Vision Team, building upon the original PP-FormulaNet. Compared to the original version, PP-FormulaNet_plus utilizes a more diverse formula dataset during training, including sources such as Chinese dissertations, professional books, textbooks, exam papers, and mathematics journals. This expansion significantly improves the model’s recognition capabilities. Among the models, PP-FormulaNet_plus-M and PP-FormulaNet_plus-L have added support for Chinese formulas and increased the maximum number of predicted tokens for formulas from 1,024 to 2,560, greatly enhancing the recognition performance for complex formulas. Meanwhile, the PP-FormulaNet_plus-S model focuses on improving the recognition of English formulas. With these improvements, the PP-FormulaNet_plus series models perform exceptionally well in handling complex and diverse formula recognition tasks. |
|---|---|---|---|---|---|---|---|
| PP-FormulaNet_plus-M | Inference Model/Training Model | 91.45 | 89.76 | 1040.27 / 1040.27 | - / 1615.80 | 592 | |
| PP-FormulaNet_plus-L | Inference Model/Training Model | 92.22 | 90.64 | 1476.07 / 1476.07 | - / 3125.58 | 698 | |
| LaTeX_OCR_rec | Inference Model/Training Model | 74.55 | 39.96 | 1088.89 / 1088.89 | - / - | 99 | LaTeX-OCR is a formula recognition algorithm based on an autoregressive large model. It uses Hybrid ViT as the backbone network and a transformer as the decoder, significantly improving the accuracy of formula recognition. |
| Mode | GPU Configuration | CPU Configuration | Acceleration Technology Combination |
|---|---|---|---|
| Normal Mode | FP32 Precision / No TRT Acceleration | FP32 Precision / 8 Threads | PaddleInference |
| High-Performance Mode | Optimal combination of pre-selected precision types and acceleration strategies | FP32 Precision / 8 Threads | Pre-selected optimal backend (Paddle/OpenVINO/TRT, etc.) |
doc-parser. Once installed, you can experience it locally via the command line or integrate it with Python.
Please note: If you encounter issues such as the program becoming unresponsive, unexpected program termination, running out of memory resources, or extremely slow inference during execution, please try adjusting the configuration according to the documentation, such as disabling unnecessary features or using lighter-weight models.
### 2.1 Command Line Experience
You can quickly experience the effect of the formula recognition pipeline with one command. Before running the code below, please download the [example image](https://paddle-model-ecology.bj.bcebos.com/paddlex/demo_image/pipelines/general_formula_recognition_001.png) locally：
```bash
paddleocr formula_recognition_pipeline -i https://paddle-model-ecology.bj.bcebos.com/paddlex/demo_image/pipelines/general_formula_recognition_001.png
# Specify whether to use the document orientation classification model with --use_doc_orientation_classify.
paddleocr formula_recognition_pipeline -i ./general_formula_recognition_001.png --use_doc_orientation_classify True
# Specify whether to use the text image unwarping module with --use_doc_unwarping.
paddleocr formula_recognition_pipeline -i ./general_formula_recognition_001.png --use_doc_unwarping True
# Specify the use of GPU for model inference with --device.
paddleocr formula_recognition_pipeline -i ./general_formula_recognition_001.png --device gpu
```
| Parameter | Description | Type | Default |
|---|---|---|---|
input |
Meaning:Data to be predicted, required. Description: Local path of image or PDF file, e.g., /root/data/img.jpg; URL link, e.g., network URL of image or PDF file: Example; Local directory, the directory should contain images to be predicted, e.g., local path: /root/data/ (currently does not support prediction of PDF files in directories; PDF files must be specified with a specific file path).
|
str |
|
save_path |
Meaning:Specify the path to save the inference results file. Description: If not set, the inference results will not be saved locally. |
str |
|
doc_orientation_classify_model_name |
Meaning:The name of the document orientation classification model. Description: If not set, the default model in pipeline will be used. |
str |
|
doc_orientation_classify_model_dir |
Meaning:The directory path of the document orientation classification model. Description: If not set, the official model will be downloaded. |
str |
|
doc_orientation_classify_batch_size |
Meaning:The batch size of the document orientation classification model. Description: If not set, the default batch size will be set to 1.
|
int |
|
doc_unwarping_model_name |
Meaning: The name of the text image unwarping model. Description: If not set, the default model in pipeline will be used. |
str |
|
doc_unwarping_model_dir |
Meaning:The directory path of the text image unwarping model. Description: If not set, the official model will be downloaded. |
str |
|
doc_unwarping_batch_size |
Meaning:The batch size of the text image unwarping model. Description: If not set, the default batch size will be set to 1. |
int |
|
use_doc_orientation_classify |
Meaning:Whether to load and use the document orientation classification module. Description: If not set, the parameter will be set to the value initialized in the pipeline, which is True by default. |
bool |
|
use_doc_unwarping |
Meaning:Whether to load and use the text image unwarping module. Description: If not set, the parameter will be set to the value initialized in the pipeline, which is True by default. |
bool |
|
layout_detection_model_name |
Meaning:The name of the layout detection model. Description: If not set, the default model in pipeline will be used. |
str |
|
layout_detection_model_dir |
Meaning:The directory path of the layout detection model. Description: If not set, the official model will be downloaded. |
str |
|
layout_threshold |
Meaning:Score threshold for the layout model. Description: Any value between 0-1. If not set, the default value is used, which is 0.5.
|
float |
|
layout_nms |
Meaning:Whether to use Non-Maximum Suppression (NMS) as post-processing for layout detection. Description: If not set, the parameter will be set to the value initialized in the pipeline, which is True by default.
|
bool |
|
layout_unclip_ratio |
Meaning:Unclip ratio for detected boxes in layout detection model. Description: Any float > 0. If not set, the default is 1.0.
|
float |
|
layout_merge_bboxes_mode |
Meaning:The merging mode for the detection boxes output by the model in layout region detection. Description:
large.
|
str |
|
layout_detection_batch_size |
Meaning:The batch size for the layout region detection model. Description: If not set, the default batch size will be set to 1. |
int |
|
use_layout_detection |
Meaning:Whether to load and use the layout detection module. Description: If not set, the parameter will be set to the value initialized in the pipeline, which is True by default. |
bool |
|
formula_recognition_model_name |
Meaning:The name of the formula recognition model. Description: If not set, the default model from the pipeline will be used. |
str |
|
formula_recognition_model_dir |
Meaning:The directory path of the formula recognition model. Description: If not set, the official model will be downloaded. |
str |
|
formula_recognition_batch_size |
Meaning:The batch size for the formula recognition model. Description: If not set, the batch size will default to 1. |
int |
|
device |
Meaning:The device used for inference. Description: You can specify a particular card number:
|
str |
|
enable_hpi |
Meaning:Whether to enable the high-performance inference plugin. | bool |
False |
use_tensorrt |
Meaning:Whether to use the Paddle Inference TensorRT subgraph engine. If the model does not support acceleration through TensorRT, setting this flag will not enable acceleration.bool |
False |
|
precision |
Meaning:Compute precision, such as FP32 or FP16. | str |
fp32 |
enable_mkldnn |
Meaning:Whether to enable MKL-DNN acceleration for inference. Description: If MKL-DNN is unavailable or the model does not support it, acceleration will not be used even if this flag is set. |
bool |
True |
mkldnn_cache_capacity |
Meaning:MKL-DNN cache capacity. | int |
10 |
cpu_threads |
Meaning:The number of threads to use when performing inference on the CPU. | int |
8 |
paddlex_config |
Meaning:Path to PaddleX pipeline configuration file. | str |
save_path, where the visualization result of formula recognition is as follows:
If you need to visualize the formula recognition pipeline, you need to run the following command to install the LaTeX rendering environment. Currently, visualization of the formula recognition pipeline only supports the Ubuntu environment, and other environments are not supported. For complex formulas, the LaTeX result may contain some advanced representations that may not be successfully displayed in environments such as Markdown:
```bash
sudo apt-get update
sudo apt-get install texlive texlive-latex-base texlive-xetex latex-cjk-all texlive-latex-extra -y
```
Note: Due to the need to render each formula image during the formula recognition visualization process, the process takes a long time. Please be patient.
### 2.2 Python Script Integration
Using the command line is a quick way to experience and check the results. Generally, in a project, you often need to integrate it through code. You can perform quick inference with just a few lines of code. The inference code is as follows:
```python
from paddleocr import FormulaRecognitionPipeline
pipeline = FormulaRecognitionPipeline()
# ocr = FormulaRecognitionPipeline(use_doc_orientation_classify=True) # Specify whether to use the document orientation classification model with use_doc_orientation_classify.
# ocr = FormulaRecognitionPipeline(use_doc_unwarping=True) # Specify whether to use the text image unwarping module with use_doc_unwarping.
# ocr = FormulaRecognitionPipeline(device="gpu") # Specify the use of GPU for model inference with device.
output = pipeline.predict("./general_formula_recognition_001.png")
for res in output:
res.print() ## Print the structured output of the prediction
res.save_to_img(save_path="output") ## Save the formula visualization result of the current image.
res.save_to_json(save_path="output") ## Save the structured JSON result of the current image
```
In the above Python script, the following steps are executed:
（1）Instantiate the formula recognition pipeline object through create_pipeline(), with specific parameters as follows:
| Parameter | Description | Type | Default |
|---|---|---|---|
doc_orientation_classify_model_name |
Meaning:The name of the document orientation classification model. Description: If set to None, the default model in pipeline will be used. |
str|None |
None |
doc_orientation_classify_model_dir |
Meaning:The directory path of the document orientation classification model. Description: If set to None, the official model will be downloaded. |
str|None |
None |
doc_orientation_classify_batch_size |
Meaning:The batch size of the document orientation classification model. Description: If set to None, the default batch size will be set to 1. |
int|None |
None |
doc_unwarping_model_name |
Meaning:The name of the text image unwarping model. Description: If set to None, the default model in pipeline will be used. |
str|None |
None |
doc_unwarping_model_dir |
Meaning:The directory path of the text image unwarping model. Description: If set to None, the official model will be downloaded. |
str|None |
None |
doc_unwarping_batch_size |
Meaning:The batch size of the text image unwarping model. Description: If set to None, the default batch size will be set to 1. |
int|None |
None |
use_doc_orientation_classify |
Meaning:Whether to load and use the document orientation classification module. Description: If set to None, the parameter will be set to the value initialized in the pipeline, which is True by default. |
bool|None |
None |
use_doc_unwarping |
Meaning:Whether to load and use the text image unwarping module. Description: If set to None, the parameter will be set to the value initialized in the pipeline, which is True by default. |
bool|None |
None |
layout_detection_model_name |
Meaning:The name of the layout detection model. Description: If set to None, the default model in pipeline will be used. |
str|None |
None |
layout_detection_model_dir |
Meaning:The directory path of the layout detection model. Description: If set to None, the official model will be downloaded. |
str|None |
None |
layout_threshold |
Meaning:Threshold for layout detection, used to filter out predictions with low confidence. Description:
|
float|dict|None |
None |
layout_nms |
Meaning:Whether to use Non-Maximum Suppression (NMS) as post-processing for layout detection. Description: If set to None, the parameter will be set to the value initialized in the pipeline, which is set to True by default. |
bool|None |
None |
layout_unclip_ratio |
Meaning:Expansion factor for the detection boxes of the layout region detection model. Description:
|
float|Tuple[float,float]|dict|None |
None |
layout_merge_bboxes_mode |
Meaning:Filtering method for overlapping boxes in layout detection. Description:
|
str|dict|None |
None |
layout_detection_batch_size |
Meaning:The batch size for the layout region detection model. Description: If set to None, the default batch size will be set to 1. |
int|None |
None |
use_layout_detection |
Meaning:Whether to load and use the layout detection module. Description: If set to None, the parameter will be set to the value initialized in the pipeline, which is True by default. |
bool|None |
None |
formula_recognition_model_name |
Meaning:The name of the formula recognition model. Description: If set to None, the default model from the pipeline will be used. |
str|None |
None |
formula_recognition_model_dir |
Meaning:The directory path of the formula recognition model. Description: If set to None, the official model will be downloaded. |
str|None |
None |
formula_recognition_batch_size |
Meaning:The batch size for the formula recognition model. Description: If set to None, the batch size will default to 1. |
int|None |
None |
device |
Meaning:The device used for inference. Description: You can specify a particular card number:
|
str|None |
None |
enable_hpi |
Meaning:Whether to enable the high-performance inference plugin. | bool |
False |
use_tensorrt |
Meaning:Whether to use the Paddle Inference TensorRT subgraph engine. If the model does not support acceleration through TensorRT, setting this flag will not enable acceleration. Description: For Paddle with CUDA version 11.8, the compatible TensorRT version is 8.x (x>=6), and it is recommended to install TensorRT 8.6.1.6. |
bool |
False |
precision |
Meaning:Compute precision, such as FP32 or FP16. | str |
"fp32" |
enable_mkldnn |
Meaning:Whether to enable MKL-DNN acceleration for inference. Description: If MKL-DNN is unavailable or the model does not support it, acceleration will not be used even if this flag is set. |
bool |
True |
mkldnn_cache_capacity |
Meaning:MKL-DNN cache capacity. | int |
10 |
cpu_threads |
Meaning:The number of threads to use when performing inference on the CPU. | int |
8 |
paddlex_config |
Meaning:Path to PaddleX pipeline configuration file. | str|None |
None |
predict() method of the formula recognition pipeline object to perform inference prediction. This method will return a list of results.
Additionally, the pipeline also provides the predict_iter() method. Both methods are completely consistent in terms of parameter acceptance and result return. The difference is that predict_iter() returns a generator, which allows for step-by-step processing and retrieval of prediction results. This is suitable for handling large datasets or scenarios where memory saving is desired. You can choose to use either of these methods based on your actual needs.
Here are the parameters of the predict() method and their descriptions:
| Parameter | Description | Type | Default Value |
|---|---|---|---|
input |
Meaning:Data to be predicted, supporting multiple input types, required. Description:
|
Python Var|str|list |
|
use_layout_detection |
Meaning:Whether to use the layout detection module during inference. | bool|None |
None |
use_doc_orientation_classify |
Meaning:Whether to use the document orientation classification module during inference. | bool|None |
None |
use_doc_unwarping |
Meaning:Whether to use the text image unwarping module during inference. | bool|None |
None |
layout_threshold |
Meaning:Same meaning as the instantiation parameters. Description: If set to None, the instantiation value is used; otherwise, this parameter takes precedence. |
float|dict|None |
None |
layout_nms |
Meaning:Same meaning as the instantiation parameters. Description: If set to None, the instantiation value is used; otherwise, this parameter takes precedence. |
bool|None |
None |
layout_unclip_ratio |
Meaning:Same meaning as the instantiation parameters. Description: If set to None, the instantiation value is used; otherwise, this parameter takes precedence. |
float|Tuple[float,float]|dict|None |
None |
layout_merge_bboxes_mode |
Meaning:Same meaning as the instantiation parameters. Description: If set to None, the instantiation value is used; otherwise, this parameter takes precedence. |
string |
None |
json file:
| Method | Description | Parameter | Parameter Type | Parameter Description | Default Value |
|---|---|---|---|---|---|
print() |
Print results to terminal | format_json |
bool |
Whether to format the output content using JSON indentation. |
True |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True. |
4 | ||
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False retains the original characters. Effective only when format_json is True. |
False |
||
save_to_json() |
Save results as a JSON file | save_path |
str |
Path to save the file. If it is a directory, the saved file will be named the same as the input file type. | 无 |
indent |
int |
Specify the indentation level to beautify the output JSON data, making it more readable. Effective only when format_json is True. |
4 | ||
ensure_ascii |
bool |
Control whether to escape non-ASCII characters to Unicode. When set to True, all non-ASCII characters will be escaped; False retains the original characters. Effective only when format_json is True. |
False |
||
save_to_img() |
Save results as an image file | save_path |
str |
Path to save the file, supports directory or file path. | 无 |
print() method will print the results to the terminal. The content printed to the terminal is explained as follows:
input_path: (str) The input path of the image to be predicted.page_index: (Union[int, None]) If the input is a PDF file, this indicates the current page number of the PDF. Otherwise, it is Nonemodel_settings: (Dict[str, bool]) The model parameters required for the pipeline configuration.
use_doc_preprocessor: (bool) Controls whether to enable the document preprocessing sub-pipeline.use_layout_detection: (bool) Controls whether to enable the layout detection module.doc_preprocessor_res: (Dict[str, Union[str, Dict[str, bool], int]]) The output result of the document preprocessing sub-pipeline. It exists only when use_doc_preprocessor=True
input_path: (Union[str, None]) T numpy.ndarray, it is saved as None.model_settings: (Dict) he model configuration parameters of the preprocessing sub-pipeline.
use_doc_orientation_classify: (bool) Controls whether to enable document orientation classification.use_doc_unwarping: (bool) Controls whether to enable document distortion correction.angle: (int) The prediction result of document orientation classification. When enabled, it takes values from [0,1,2,3], corresponding to [0°,90°,180°,270°]; when disabled, it is -1.layout_det_res: (Dict[str, List[Dict]]) The output result of the layout area detection module. It exists only when use_layout_detection=Trueinput_path: (Union[str, None]) The image path accepted by the layout area detection module. When the input is a numpy.ndarray, it is saved as None.boxes: (List[Dict[int, str, float, List[float]]]) A list of layout area detection prediction results.cls_id: (int) The class ID predicted by layout area detection.label: (str) The class label predicted by layout area detection.score: (float) The confidence score of the predicted class.coordinate: (List[float]) The bounding box coordinates predicted by layout area detection, in the format [x_min, y_min, x_max, y_max], where (x_min, y_min) is the top-left corner and (x_max, y_max) is the bottom-right corner.formula_res_list: (List[Dict[str, int, List[float]]]) A list of formula recognition prediction results.rec_formula: (str) The LaTeX source code predicted by formula recognition.formula_region_id: (int) The ID number predicted by formula recognition.dt_polys: (List[float]) The bounding box coordinates predicted by formula recognition, in the format [x_min, y_min, x_max, y_max], where (x_min, y_min) is the top-left corner and (x_max, y_max) is the bottom-right corner.save_to_json() method will save the above content to the specified save_path. If a directory is specified, the saved path will be save_path/{your_img_basename}_res.json. If a file is specified, it will be saved directly to that file. Since JSON files do not support saving numpy arrays, numpy.array types will be converted to list format.save_to_img() method will save the visualization results to the specified save_path. If a directory is specified, the saved path will be save_path/{your_img_basename}_formula_res_img.{your_img_extension}. If a file is specified, it will be saved directly to that file. (The pipeline usually contains many result images, so it is not recommended to specify a specific file path directly, otherwise multiple images will be overwritten and only the last one will be retained.)| Attribute | Attribute Description |
|---|---|
json |
Get the prediction results in json format |
img |
Get the visualization image in dict format |
json attribute is of the dict type, and its content is consistent with what is saved by calling the save_to_json() method.img attribute is a dictionary-type data. The keys are preprocessed_img、 layout_det_res and formula_res_img, and the corresponding values are three Image.Image objects: the first one is used to display the visualization of image preprocessing, the second one is for displaying the visualization of layout region detection, and the third one is for displaying the visualization of formula recognition. If the image preprocessing submodule is not used, the dictionary will not contain the preprocessed_img key. Similarly, if the layout region detection submodule is not used, the dictionary will not contain the layout_det_res key.For the main operations provided by the service:
200, and the attributes of the response body are as follows:| Name | Type | Meaning |
|---|---|---|
logId |
string |
The UUID of the request. |
errorCode |
integer |
Error code. Fixed as 0. |
errorMsg |
string |
Error message. Fixed as "Success". |
result |
object |
The result of the operation. |
| Name | Type | Meaning |
|---|---|---|
logId |
string |
The UUID of the request. |
errorCode |
integer |
Error code. Same as the response status code. |
errorMsg |
string |
Error message. |
The main operations provided by the service are as follows:
inferObtain the formula recognition results from images.
POST /formula-recognition
| Name | Type | Meaning | Required |
|---|---|---|---|
file |
string |
The URL of an image or PDF file accessible by the server, or the Base64-encoded content of the file. By default, for PDF files exceeding 10 pages, only the first 10 pages will be processed. To remove the page limit, please add the following configuration to the pipeline configuration file: |
Yes |
fileType |
integer | null |
The type of the file. 0 for PDF files, 1 for image files. If this attribute is missing, the file type will be inferred from the URL. |
No |
useDocOrientationClassify |
boolean | null |
Please refer to the description of the use_doc_orientation_classify parameter of the pipeline object's predict method. |
No |
useDocUnwarping |
boolean | null |
Please refer to the description of the use_doc_unwarping parameter of the pipeline object's predict method. |
No |
useLayoutDetection |
boolean | null |
Please refer to the description of the use_layout_detection parameter of the pipeline object's predict method. |
No |
layoutThreshold |
number | null |
Please refer to the description of the layout_threshold parameter of the pipeline object's predict method. |
No |
layoutNms |
boolean | null |
Please refer to the description of the layout_nms parameter of the pipeline object's predict method. |
No |
layoutUnclipRatio |
number | array | null |
Please refer to the description of the layout_unclip_ratio parameter of the pipeline object's predict method. |
No |
layoutMergeBboxesMode |
string | null |
Please refer to the description of the layout_merge_bboxes_mode parameter of the pipeline object's predict method. |
No |
visualize |
boolean | null |
Whether to return the final visualization image and intermediate images during the processing.
For example, adding the following setting to the pipeline config file: visualize parameter in the request.If neither the request body nor the configuration file is set (If visualize is set to null in the request and not defined in the configuration file), the image is returned by default.
|
No |
result in the response body has the following attributes:| 名称 | 类型 | 含义 |
|---|---|---|
formulaRecResults |
object |
The formula recognition results. The array length is 1 (for image input) or the actual number of document pages processed (for PDF input). For PDF input, each element in the array represents the result of each page actually processed in the PDF file. |
dataInfo |
object |
Information about the input data. |
Each element in formulaRecResults is an object with the following attributes:
| Name | Type | Meaning |
|---|---|---|
prunedResult |
object |
A simplified version of the res field in the JSON representation of the result generated by the pipeline object's predict method, excluding the input_path and the page_index fields. |
outputImages |
object | null |
See the description of the img attribute of the result of the pipeline prediction. The images are in JPEG format and are Base64-encoded. |
inputImage | null |
string |
The input image. The image is in JPEG format and is Base64-encoded. |
import base64
import requests
API_URL = "http://localhost:8080/formula-recognition"
file_path = "./demo.jpg"
with open(file_path, "rb") as file:
file_bytes = file.read()
file_data = base64.b64encode(file_bytes).decode("ascii")
payload = {"file": file_data, "fileType": 1}
response = requests.post(API_URL, json=payload)
assert response.status_code == 200
result = response.json()["result"]
for i, res in enumerate(result["formulaRecResults"]):
print(res["prunedResult"])
for img_name, img in res["outputImages"].items():
img_path = f"{img_name}_{i}.jpg"
with open(img_path, "wb") as f:
f.write(base64.b64decode(img))
print(f"Output image saved at {img_path}")
#include <iostream>
#include <fstream>
#include <vector>
#include <string>
#include "cpp-httplib/httplib.h" // https://github.com/Huiyicc/cpp-httplib
#include "nlohmann/json.hpp" // https://github.com/nlohmann/json
#include "base64.hpp" // https://github.com/tobiaslocker/base64
int main() {
httplib::Client client("localhost", 8080);
const std::string filePath = "./demo.jpg";
std::ifstream file(filePath, std::ios::binary | std::ios::ate);
if (!file) {
std::cerr << "Error opening file: " << filePath << std::endl;
return 1;
}
std::streamsize size = file.tellg();
file.seekg(0, std::ios::beg);
std::vector buffer(size);
if (!file.read(buffer.data(), size)) {
std::cerr << "Error reading file." << std::endl;
return 1;
}
std::string bufferStr(buffer.data(), static_cast(size));
std::string encodedFile = base64::to_base64(bufferStr);
nlohmann::json jsonObj;
jsonObj["file"] = encodedFile;
jsonObj["fileType"] = 1;
auto response = client.Post("/formula-recognition", jsonObj.dump(), "application/json");
if (response && response->status == 200) {
nlohmann::json jsonResponse = nlohmann::json::parse(response->body);
auto result = jsonResponse["result"];
if (!result.is_object() || !result["formulaRecResults"].is_array()) {
std::cerr << "Unexpected response format." << std::endl;
return 1;
}
for (size_t i = 0; i < result["formulaRecResults"].size(); ++i) {
auto res = result["formulaRecResults"][i];
if (res.contains("prunedResult")) {
std::cout << "Recognized formula: " << res["prunedResult"].dump() << std::endl;
}
if (res.contains("outputImages") && res["outputImages"].is_object()) {
for (auto& [imgName, imgData] : res["outputImages"].items()) {
std::string outputPath = imgName + "_" + std::to_string(i) + ".jpg";
std::string decodedImage = base64::from_base64(imgData.get());
std::ofstream outFile(outputPath, std::ios::binary);
if (outFile.is_open()) {
outFile.write(decodedImage.c_str(), decodedImage.size());
outFile.close();
std::cout << "Saved image: " << outputPath << std::endl;
} else {
std::cerr << "Failed to write image: " << outputPath << std::endl;
}
}
}
}
} else {
std::cerr << "Request failed." << std::endl;
if (response) {
std::cerr << "HTTP status: " << response->status << std::endl;
std::cerr << "Response body: " << response->body << std::endl;
}
return 1;
}
return 0;
}
import okhttp3.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Base64;
public class Main {
public static void main(String[] args) throws IOException {
String API_URL = "http://localhost:8080/formula-recognition";
String imagePath = "./demo.jpg";
File file = new File(imagePath);
byte[] fileContent = java.nio.file.Files.readAllBytes(file.toPath());
String base64Image = Base64.getEncoder().encodeToString(fileContent);
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode payload = objectMapper.createObjectNode();
payload.put("file", base64Image);
payload.put("fileType", 1);
OkHttpClient client = new OkHttpClient();
MediaType JSON = MediaType.get("application/json; charset=utf-8");
RequestBody body = RequestBody.create(JSON, payload.toString());
Request request = new Request.Builder()
.url(API_URL)
.post(body)
.build();
try (Response response = client.newCall(request).execute()) {
if (response.isSuccessful()) {
String responseBody = response.body().string();
JsonNode root = objectMapper.readTree(responseBody);
JsonNode result = root.get("result");
JsonNode formulaRecResults = result.get("formulaRecResults");
for (int i = 0; i < formulaRecResults.size(); i++) {
JsonNode item = formulaRecResults.get(i);
int finalI = i;
JsonNode prunedResult = item.get("prunedResult");
System.out.println("Pruned Result [" + i + "]: " + prunedResult.toString());
JsonNode outputImages = item.get("outputImages");
if (outputImages != null && outputImages.isObject()) {
outputImages.fieldNames().forEachRemaining(imgName -> {
try {
String imgBase64 = outputImages.get(imgName).asText();
byte[] imgBytes = Base64.getDecoder().decode(imgBase64);
String imgPath = imgName + "_" + finalI + ".jpg";
try (FileOutputStream fos = new FileOutputStream(imgPath)) {
fos.write(imgBytes);
System.out.println("Saved image: " + imgPath);
}
} catch (IOException e) {
System.err.println("Failed to save image: " + e.getMessage());
}
});
}
}
} else {
System.err.println("Request failed with HTTP code: " + response.code());
}
}
}
}
package main
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
API_URL := "http://localhost:8080/formula-recognition"
filePath := "./demo.jpg"
fileBytes, err := ioutil.ReadFile(filePath)
if err != nil {
fmt.Printf("Error reading file: %v\n", err)
return
}
fileData := base64.StdEncoding.EncodeToString(fileBytes)
payload := map[string]interface{}{
"file": fileData,
"fileType": 1,
}
payloadBytes, err := json.Marshal(payload)
if err != nil {
fmt.Printf("Error marshaling payload: %v\n", err)
return
}
client := &http.Client{}
req, err := http.NewRequest("POST", API_URL, bytes.NewBuffer(payloadBytes))
if err != nil {
fmt.Printf("Error creating request: %v\n", err)
return
}
req.Header.Set("Content-Type", "application/json")
res, err := client.Do(req)
if err != nil {
fmt.Printf("Error sending request: %v\n", err)
return
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
fmt.Printf("Unexpected status code: %d\n", res.StatusCode)
return
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Printf("Error reading response body: %v\n", err)
return
}
type FormulaRecResult struct {
PrunedResult map[string]interface{} `json:"prunedResult"`
OutputImages map[string]string `json:"outputImages"`
InputImage *string `json:"inputImage"`
}
type Response struct {
Result struct {
FormulaRecResults []FormulaRecResult `json:"formulaRecResults"`
DataInfo interface{} `json:"dataInfo"`
} `json:"result"`
}
var respData Response
if err := json.Unmarshal(body, &respData); err != nil {
fmt.Printf("Error unmarshaling response: %v\n", err)
return
}
for i, res := range respData.Result.FormulaRecResults {
fmt.Printf("Result %d - prunedResult: %+v\n", i, res.PrunedResult)
for imgName, imgData := range res.OutputImages {
imgBytes, err := base64.StdEncoding.DecodeString(imgData)
if err != nil {
fmt.Printf("Error decoding image %s_%d: %v\n", imgName, i, err)
continue
}
filename := fmt.Sprintf("%s_%d.jpg", imgName, i)
if err := os.WriteFile(filename, imgBytes, 0644); err != nil {
fmt.Printf("Error saving image %s: %v\n", filename, err)
continue
}
fmt.Printf("Saved image to %s\n", filename)
}
}
}
using System;
using System.IO;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json.Linq;
class Program
{
static readonly string API_URL = "http://localhost:8080/formula-recognition";
static readonly string inputFilePath = "./demo.jpg";
static async Task Main(string[] args)
{
var httpClient = new HttpClient();
byte[] fileBytes = File.ReadAllBytes(inputFilePath);
string fileData = Convert.ToBase64String(fileBytes);
var payload = new JObject
{
{ "file", fileData },
{ "fileType", 1 }
};
var content = new StringContent(payload.ToString(), Encoding.UTF8, "application/json");
HttpResponseMessage response = await httpClient.PostAsync(API_URL, content);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
JObject jsonResponse = JObject.Parse(responseBody);
JArray formulaRecResults = (JArray)jsonResponse["result"]["formulaRecResults"];
for (int i = 0; i < formulaRecResults.Count; i++)
{
var res = formulaRecResults[i];
Console.WriteLine($"[{i}] prunedResult:\n{res["prunedResult"]}");
JObject outputImages = res["outputImages"] as JObject;
if (outputImages != null)
{
foreach (var img in outputImages)
{
string imgName = img.Key;
string base64Img = img.Value?.ToString();
if (!string.IsNullOrEmpty(base64Img))
{
string imgPath = $"{imgName}_{i}.jpg";
byte[] imageBytes = Convert.FromBase64String(base64Img);
File.WriteAllBytes(imgPath, imageBytes);
Console.WriteLine($"Output image saved at {imgPath}");
}
}
}
}
}
}
const axios = require('axios');
const fs = require('fs');
const path = require('path');
const API_URL = 'http://localhost:8080/formula-recognition';
const inputFilePath = './demo.jpg';
const fileType = 1;
function encodeImageToBase64(filePath) {
const bitmap = fs.readFileSync(filePath);
return Buffer.from(bitmap).toString('base64');
}
const payload = {
file: encodeImageToBase64(inputFilePath),
fileType: fileType
};
axios.post(API_URL, payload)
.then((response) => {
const resultArray = response.data.result.formulaRecResults;
resultArray.forEach((res, index) => {
console.log(`\n[${index}] prunedResult:`);
console.log(res.prunedResult);
const outputImages = res.outputImages;
if (outputImages) {
Object.entries(outputImages).forEach(([imgName, base64Img]) => {
const outputPath = `${imgName}_${index}.jpg`;
fs.writeFileSync(outputPath, Buffer.from(base64Img, 'base64'));
console.log(`Saved output image: ${outputPath}`);
});
} else {
console.log(`[${index}] outputImages is null`);
}
});
})
.catch((error) => {
console.error('API error：', error.message);
});
<?php
$API_URL = "http://localhost:8080/formula-recognition";
$image_path = "./demo.jpg";
$image_data = base64_encode(file_get_contents($image_path));
$payload = array("file" => $image_data, "fileType" => 1);
$ch = curl_init($API_URL);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($payload));
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
$result = json_decode($response, true)["result"]["formulaRecResults"];
foreach ($result as $i => $item) {
echo "[$i] prunedResult:\n";
print_r($item["prunedResult"]);
if (!empty($item["outputImages"])) {
foreach ($item["outputImages"] as $img_name => $base64_img) {
$img_path = "{$img_name}_{$i}.jpg";
file_put_contents($img_path, base64_decode($base64_img));
echo "Output image saved at $img_path\n";
}
} else {
echo "No outputImages found for item $i\n";
}
}
?>
| Scenario | Fine-Tuning Module | Reference Link |
|---|---|---|
| Formulas are missing | Layout Detection Module | Link |
| Formula content is inaccurate | Formula Recognition Module | Link |
| Whole-image rotation correction is inaccurate | Document Image Orientation Classification Module | Link |
| Image distortion correction is inaccurate | Text Image Correction Module | Fine-tuning not supported |