âMathBERT: A Pre-Trained Model for Mathematical Formula Understandingâ, 2021-05-02 (; similar)â :
Large-scale pre-trained models like BERT, have obtained a great success in various Natural Language Processing (NLP) tasks, while it is still a challenge to adapt them to the math-related tasks. Current pre-trained models neglect the structural features and the semantic correspondence between formula and its context.
To address these issues, we propose a novel pre-trained model, namely MathBERT, which is jointly trained with mathematical formulas and their corresponding contexts. In addition, in order to further capture the semantic-level structural features of formulas, a new pre-training task is designed to predict the masked formula substructures extracted from the Operator Tree (OPT), which is the semantic structural representation of formulas.
We conduct various experiments on 3 downstream tasks to evaluate the performance of MathBERT, including mathematical information retrieval, formula topic classification and formula headline generation. Experimental results demonstrate that MathBERT outperforms existing methods on all those 3 tasks.
Moreover, we qualitatively show that this pre-trained model effectively captures the semantic-level structural information of formulas.
To the best of our knowledge, MathBERT is the first pre-trained model for mathematical formula understanding.