The Experts below are selected from a list of 48432 Experts worldwide ranked by ideXlab platform
Lirong Dai - One of the best experts on this subject based on the ideXlab platform.
-
srd a tree structure based decoder for online handwritten Mathematical Expression recognition
IEEE Transactions on Multimedia, 2021Co-Authors: Jianshu Zhang, Yongxin Yang, Yizhe Song, Lirong DaiAbstract:Recently, recognition of online handwritten mathe- matical Expression has been greatly improved by employing encoder-decoder based methods. Existing encoder-decoder models use string decoders to generate LaTeX strings for Mathematical Expression recognition. However, in this paper, we importantly argue that string representations might not be the most natural for Mathematical Expressions – Mathematical Expressions are inherently tree structures other than flat strings. For this purpose, we propose a novel sequential relation decoder (SRD) that aims to decode Expressions into tree structures for online handwritten Mathematical Expression recognition. At each step of tree construction, a sub-tree structure composed of a relation node and two symbol nodes is computed based on previous sub-tree structures. This is the first work that builds a tree structure based decoder for encoder-decoder based Mathematical Expression recognition. Compared with string decoders, a decoder that better understands tree structures is crucial for Mathematical Expression recognition as it brings a more reasonable learning objective and improves overall generalization ability. We demonstrate how the proposed SRD outperforms state-of-the-art string decoders through a set of experiments on CROHME database, which is currently the largest benchmark for online handwritten Mathematical Expression recognition.
-
track attend and parse tap an end to end framework for online handwritten Mathematical Expression recognition
IEEE Transactions on Multimedia, 2019Co-Authors: Jianshu Zhang, Lirong DaiAbstract:In this paper, we introduce Track, Attend, and Parse (TAP), an end-to-end approach based on neural networks for online handwritten Mathematical Expression recognition (OHMER). The architecture of TAP consists of a tracker and a parser. The tracker employs a stack of bidirectional recurrent neural networks with gated recurrent units (GRU) to model the input handwritten traces, which can fully utilize the dynamic trajectory information in OHMER. Followed by the tracker, the parser adopts a GRU equipped with guided hybrid attention (GHA) to generate notations. The proposed GHA is composed of a coverage-based spatial attention, a temporal attention, and an attention guider. Moreover, we demonstrate the strong complementarity between offline information with static-image input and online information with ink-trajectory input by blending a fully convolutional networks-based watcher into TAP. Inherently, unlike traditional methods, this end-to-end framework does not require the explicit symbol segmentation and a predefined Expression grammar for parsing. Validated on a benchmark published by the CROHME competition, the proposed approach outperforms the state-of-the-art methods and achieves the best reported results with an Expression recognition accuracy of 61.16% on CROHME 2014 and 57.02% on CROHME 2016, using only official training dataset.
-
multi scale attention with dense encoder for handwritten Mathematical Expression recognition
International Conference on Pattern Recognition, 2018Co-Authors: Jianshu Zhang, Lirong DaiAbstract:Handwritten Mathematical Expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, recently we propose the attention based encoder-decoder model that recognizes Mathematical Expression images from two-dimensional layouts to one-dimensional LaTeX strings. In this study, we improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and restore the fine-grained details dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an Expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.
-
multi scale attention with dense encoder for handwritten Mathematical Expression recognition
arXiv: Computer Vision and Pattern Recognition, 2018Co-Authors: Jianshu Zhang, Lirong DaiAbstract:Handwritten Mathematical Expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, we utilize the attention based encoder-decoder model that recognizes Mathematical Expression images from two-dimensional layouts to one-dimensional LaTeX strings. We improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and save the fine-grained details that will be dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an Expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.
-
a gru based encoder decoder approach with attention for online handwritten Mathematical Expression recognition
International Conference on Document Analysis and Recognition, 2017Co-Authors: Jianshu Zhang, Lirong DaiAbstract:In this study, we present a novel end-to-end approach based on the encoder-decoder framework with the attention mechanism for online handwritten Mathematical Expression recognition (OHMER). First, the input two-dimensional ink trajectory information of handwritten Expression is encoded via the gated recurrent unit based recurrent neural network (GRU-RNN). Then the decoder is also implemented by the GRU-RNN with a coverage-based attention model. The proposed approach can simultaneously accomplish the symbol recognition and structural analysis to output a character sequence in LaTeX format. Validated on the CROHME 2014 competition task, our approach significantly outperforms the state-of-the-art with an Expression recognition accuracy of 52.43% by only using the official training dataset. Furthermore, the alignments between the input trajectories of handwritten Expressions and the output LaTeX sequences are visualized by the attention mechanism to show the effectiveness of the proposed method.
Jianshu Zhang - One of the best experts on this subject based on the ideXlab platform.
-
stroke constrained attention network for online handwritten Mathematical Expression recognition
Pattern Recognition, 2021Co-Authors: Jiaming Wang, Jianshu Zhang, Bin Wang, Bo RenAbstract:Abstract In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten Mathematical Expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately. Besides, we propose an approach combining the encoder fusion and decoder fusion, namely encoder-decoder fusion, which can further improve the performance. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance. Furthermore, by conducting experiments on an additional task: online handwritten Chinese character recognition (HCCR), we demonstrate the generality of our proposed method.
-
srd a tree structure based decoder for online handwritten Mathematical Expression recognition
IEEE Transactions on Multimedia, 2021Co-Authors: Jianshu Zhang, Yongxin Yang, Yizhe Song, Lirong DaiAbstract:Recently, recognition of online handwritten mathe- matical Expression has been greatly improved by employing encoder-decoder based methods. Existing encoder-decoder models use string decoders to generate LaTeX strings for Mathematical Expression recognition. However, in this paper, we importantly argue that string representations might not be the most natural for Mathematical Expressions – Mathematical Expressions are inherently tree structures other than flat strings. For this purpose, we propose a novel sequential relation decoder (SRD) that aims to decode Expressions into tree structures for online handwritten Mathematical Expression recognition. At each step of tree construction, a sub-tree structure composed of a relation node and two symbol nodes is computed based on previous sub-tree structures. This is the first work that builds a tree structure based decoder for encoder-decoder based Mathematical Expression recognition. Compared with string decoders, a decoder that better understands tree structures is crucial for Mathematical Expression recognition as it brings a more reasonable learning objective and improves overall generalization ability. We demonstrate how the proposed SRD outperforms state-of-the-art string decoders through a set of experiments on CROHME database, which is currently the largest benchmark for online handwritten Mathematical Expression recognition.
-
multi modal attention network for handwritten Mathematical Expression recognition
International Conference on Document Analysis and Recognition, 2019Co-Authors: Jiaming Wang, Jianshu Zhang, Zirui WangAbstract:In this paper, we propose a novel multi-modal attention network (MAN), which is based on encoder-decoder framework, for handwritten Mathematical Expression recognition (HMER). Here, multi-modal means two specific modalities: online and offline, where online modality employs dynamic trajectories as input and offline modality employs static images as input. More specifically, the proposed method first feeds dynamic trajectories and static images into online and offline channels of the multi-modal encoder respectively. The output of the encoder is then transferred to the multi-modal decoder to generate a LaTeX sequence as the Mathematical Expression recognition result. To make full use of the complementary information that comes from the two modalities, we propose a re-attention mechanism as an enhanced version of the multi-modal attention mechanism which can further improve the recognition performance. Evaluated on a benchmark published by CROHME competition, the proposed approach achieves an Expression recognition accuracy of 54.05% on CROHME 2014 and 50.56% on CROHME 2016 which substantially outperforms the state-of-the-arts using the single online or offline modality.
-
track attend and parse tap an end to end framework for online handwritten Mathematical Expression recognition
IEEE Transactions on Multimedia, 2019Co-Authors: Jianshu Zhang, Lirong DaiAbstract:In this paper, we introduce Track, Attend, and Parse (TAP), an end-to-end approach based on neural networks for online handwritten Mathematical Expression recognition (OHMER). The architecture of TAP consists of a tracker and a parser. The tracker employs a stack of bidirectional recurrent neural networks with gated recurrent units (GRU) to model the input handwritten traces, which can fully utilize the dynamic trajectory information in OHMER. Followed by the tracker, the parser adopts a GRU equipped with guided hybrid attention (GHA) to generate notations. The proposed GHA is composed of a coverage-based spatial attention, a temporal attention, and an attention guider. Moreover, we demonstrate the strong complementarity between offline information with static-image input and online information with ink-trajectory input by blending a fully convolutional networks-based watcher into TAP. Inherently, unlike traditional methods, this end-to-end framework does not require the explicit symbol segmentation and a predefined Expression grammar for parsing. Validated on a benchmark published by the CROHME competition, the proposed approach outperforms the state-of-the-art methods and achieves the best reported results with an Expression recognition accuracy of 61.16% on CROHME 2014 and 57.02% on CROHME 2016, using only official training dataset.
-
multi scale attention with dense encoder for handwritten Mathematical Expression recognition
International Conference on Pattern Recognition, 2018Co-Authors: Jianshu Zhang, Lirong DaiAbstract:Handwritten Mathematical Expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, recently we propose the attention based encoder-decoder model that recognizes Mathematical Expression images from two-dimensional layouts to one-dimensional LaTeX strings. In this study, we improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and restore the fine-grained details dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an Expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.
Christian Viard-gaudin - One of the best experts on this subject based on the ideXlab platform.
-
Top-Down Online Handwritten Mathematical Expression Parsing with Graph Grammar
2015Co-Authors: Frank Julca-aguilar, Harold Mouchère, Christian Viard-gaudin, Nina HirataAbstract:In recognition of online handwritten Mathematical Expressions, symbol segmentation and classification and recognition of relations among symbols is managed through a parsing technique. Most parsing techniques follow a bottom-up approach and adapt grammars typically used to parse strings. However, in contrast to top-down approaches, pure bottom-up approaches do not exploit grammar information to avoid parsing of invalid subExpressions. Moreover, modeling math Expressions by string grammars makes difficult to extend it to include new structures. We propose a new parsing technique that models Mathematical Expressions as languages generated by graph grammars, and parses Expressions following a top-down approach. The method is general in the sense that it can be easily extended to parse other multidimensional languages, as chemical Expressions, or diagrams. We evaluate the method using the (publicly available) CROHME-2013 dataset.
-
A global learning approach for an online handwritten Mathematical Expression recognition system
Pattern Recognition Letters, 2014Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian Viard-gaudinAbstract:Despite the recent advances in handwriting recognition, handwritten two-dimensional (2D) languages are still a challenge. Electrical schemas, chemical equations and Mathematical Expressions (MEs) are examples of such 2D languages. In this case, the recognition problem is particularly difficult due to the two dimensional layout of the language. This paper presents an online handwritten Mathematical Expression recognition system that handles Mathematical Expression recognition as a simultaneous optimization of Expression segmentation, symbol recognition, and 2D structure recognition under the restriction of a Mathematical Expression grammar. The originality of the approach is a global strategy allowing learning Mathematical symbols and spatial relations directly from complete Expressions. A new contextual modeling is proposed for combining syntactic and structural information. Those models are used to find the most likely combination of segmentation/recognition hypotheses proposed by a 2D segmentation scheme. Thus, models are based on structural information concerning the symbol layout. The system is tested with a new public database of Mathematical Expressions which was used in the CHROME competition. We have also produced a large base of semi-synthetic Expressions which are used to train and test the global learning approach. We obtain very promising results on both synthetic and real Expressions databases, as well as in the recent CHROME competition.
-
Towards Handwritten Mathematical Expression Recognition
2009 10th International Conference on Document Analysis and Recognition, 2009Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian Viard-gaudinAbstract:In this paper, we propose a new framework for online handwritten Mathematical Expression recognition. The proposed architecture aims at handling Mathematical Expression recognition as a simultaneous optimization of symbol segmentation, symbol recognition, and 2D structure recognition under the restriction of a Mathematical Expression grammar. To achieve this goal, we consider a hypothesis generation mechanism supporting a 2D grouping of elementary strokes, a cost function defining the global likelihood of a solution, and a dynamic programming scheme giving at the end the best global solution according to a 2D grammar and a classifier. As a classifier, a neural network architecture is used; it is trained within the overall architecture allowing rejecting incorrect segmented patterns. The proposed system is trained with a set of synthetic online handwritten Mathematical Expressions. When tested on a set of real complex Expressions, the system achieves promising results at both symbol and Expression interpretation levels.
Ahmad-montaser Awal - One of the best experts on this subject based on the ideXlab platform.
-
A global learning approach for an online handwritten Mathematical Expression recognition system
Pattern Recognition Letters, 2014Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian Viard-gaudinAbstract:Despite the recent advances in handwriting recognition, handwritten two-dimensional (2D) languages are still a challenge. Electrical schemas, chemical equations and Mathematical Expressions (MEs) are examples of such 2D languages. In this case, the recognition problem is particularly difficult due to the two dimensional layout of the language. This paper presents an online handwritten Mathematical Expression recognition system that handles Mathematical Expression recognition as a simultaneous optimization of Expression segmentation, symbol recognition, and 2D structure recognition under the restriction of a Mathematical Expression grammar. The originality of the approach is a global strategy allowing learning Mathematical symbols and spatial relations directly from complete Expressions. A new contextual modeling is proposed for combining syntactic and structural information. Those models are used to find the most likely combination of segmentation/recognition hypotheses proposed by a 2D segmentation scheme. Thus, models are based on structural information concerning the symbol layout. The system is tested with a new public database of Mathematical Expressions which was used in the CHROME competition. We have also produced a large base of semi-synthetic Expressions which are used to train and test the global learning approach. We obtain very promising results on both synthetic and real Expressions databases, as well as in the recent CHROME competition.
-
the problem of handwritten Mathematical Expression recognition evaluation
International Conference on Frontiers in Handwriting Recognition, 2010Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian ViardgaudinAbstract:We discuss in this paper some issues related to the problem of Mathematical Expression recognition. The very first important issue is to define how to ground truth a dataset of handwritten Mathematical Expressions, and next we have to face the problem of benchmarking systems. We propose to define some indicators and the way to compute them so as they reflect the actual performances of a given system.
-
Towards Handwritten Mathematical Expression Recognition
2009 10th International Conference on Document Analysis and Recognition, 2009Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian Viard-gaudinAbstract:In this paper, we propose a new framework for online handwritten Mathematical Expression recognition. The proposed architecture aims at handling Mathematical Expression recognition as a simultaneous optimization of symbol segmentation, symbol recognition, and 2D structure recognition under the restriction of a Mathematical Expression grammar. To achieve this goal, we consider a hypothesis generation mechanism supporting a 2D grouping of elementary strokes, a cost function defining the global likelihood of a solution, and a dynamic programming scheme giving at the end the best global solution according to a 2D grammar and a classifier. As a classifier, a neural network architecture is used; it is trained within the overall architecture allowing rejecting incorrect segmented patterns. The proposed system is trained with a set of synthetic online handwritten Mathematical Expressions. When tested on a set of real complex Expressions, the system achieves promising results at both symbol and Expression interpretation levels.
Harold Mouchère - One of the best experts on this subject based on the ideXlab platform.
-
Top-Down Online Handwritten Mathematical Expression Parsing with Graph Grammar
2015Co-Authors: Frank Julca-aguilar, Harold Mouchère, Christian Viard-gaudin, Nina HirataAbstract:In recognition of online handwritten Mathematical Expressions, symbol segmentation and classification and recognition of relations among symbols is managed through a parsing technique. Most parsing techniques follow a bottom-up approach and adapt grammars typically used to parse strings. However, in contrast to top-down approaches, pure bottom-up approaches do not exploit grammar information to avoid parsing of invalid subExpressions. Moreover, modeling math Expressions by string grammars makes difficult to extend it to include new structures. We propose a new parsing technique that models Mathematical Expressions as languages generated by graph grammars, and parses Expressions following a top-down approach. The method is general in the sense that it can be easily extended to parse other multidimensional languages, as chemical Expressions, or diagrams. We evaluate the method using the (publicly available) CROHME-2013 dataset.
-
A global learning approach for an online handwritten Mathematical Expression recognition system
Pattern Recognition Letters, 2014Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian Viard-gaudinAbstract:Despite the recent advances in handwriting recognition, handwritten two-dimensional (2D) languages are still a challenge. Electrical schemas, chemical equations and Mathematical Expressions (MEs) are examples of such 2D languages. In this case, the recognition problem is particularly difficult due to the two dimensional layout of the language. This paper presents an online handwritten Mathematical Expression recognition system that handles Mathematical Expression recognition as a simultaneous optimization of Expression segmentation, symbol recognition, and 2D structure recognition under the restriction of a Mathematical Expression grammar. The originality of the approach is a global strategy allowing learning Mathematical symbols and spatial relations directly from complete Expressions. A new contextual modeling is proposed for combining syntactic and structural information. Those models are used to find the most likely combination of segmentation/recognition hypotheses proposed by a 2D segmentation scheme. Thus, models are based on structural information concerning the symbol layout. The system is tested with a new public database of Mathematical Expressions which was used in the CHROME competition. We have also produced a large base of semi-synthetic Expressions which are used to train and test the global learning approach. We obtain very promising results on both synthetic and real Expressions databases, as well as in the recent CHROME competition.
-
the problem of handwritten Mathematical Expression recognition evaluation
International Conference on Frontiers in Handwriting Recognition, 2010Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian ViardgaudinAbstract:We discuss in this paper some issues related to the problem of Mathematical Expression recognition. The very first important issue is to define how to ground truth a dataset of handwritten Mathematical Expressions, and next we have to face the problem of benchmarking systems. We propose to define some indicators and the way to compute them so as they reflect the actual performances of a given system.
-
Towards Handwritten Mathematical Expression Recognition
2009 10th International Conference on Document Analysis and Recognition, 2009Co-Authors: Ahmad-montaser Awal, Harold Mouchère, Christian Viard-gaudinAbstract:In this paper, we propose a new framework for online handwritten Mathematical Expression recognition. The proposed architecture aims at handling Mathematical Expression recognition as a simultaneous optimization of symbol segmentation, symbol recognition, and 2D structure recognition under the restriction of a Mathematical Expression grammar. To achieve this goal, we consider a hypothesis generation mechanism supporting a 2D grouping of elementary strokes, a cost function defining the global likelihood of a solution, and a dynamic programming scheme giving at the end the best global solution according to a 2D grammar and a classifier. As a classifier, a neural network architecture is used; it is trained within the overall architecture allowing rejecting incorrect segmented patterns. The proposed system is trained with a set of synthetic online handwritten Mathematical Expressions. When tested on a set of real complex Expressions, the system achieves promising results at both symbol and Expression interpretation levels.