Publications
Multimodal Search in Chemical Documents and Reactions
This paper introduces a multimodal search tool for retrieving chemical reactions, molecular structures, and associated text from scientific literature, linking visual and textual representations of chemical information.
Recommended citation
A. K. Shah, A. Dey, L. Luo, B. M. Amador, P. Philippy, M. Zhong, S. Ouyang, D. M. Friday, D. Bianchi, N. Jackson, R. Zanibbi, and J. Han, "Multimodal Search in Chemical Documents and Reactions", in Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, in SIGIR ’25. New York, NY, USA: Association for Computing Machinery, July 2025, pp. 4030–4034. doi: 10.1145/3726302.3730152.
ChemScraper: Leveraging PDF Graphics Instructions for Molecular Diagram Parsing
ChemScraper is a molecular diagram parser that directly extracts characters and graphical elements from PDFs without using OCR or GPUs. The extracted data is used for training neural models for molecular recognition in raster images.
Recommended citation
A. K. Shah, B. M. Amador, A. Dey, M. Creekmore, B. Ocampo, S. Denmark, and R. Zanibbi, “ChemScraper: Leveraging PDF Graphics Instructions for Molecular Diagram Parsing,” in Document Analysis and Recognition (Journal) - IJDAR 2024, vol. 27, Sep. 2024, pp. 395-414, doi: 10.1007/s10032-024-00486-7.
Line-of-sight with Graph Attention Parser (LGAP) for Math Formulas
The Line-of-Sight with Graph Attention Parser (LGAP) is a graph-based approach to recognizing mathematical notation. It improves interpretability over encoder-decoder models while enhancing accuracy through context-aware graph pooling.
Recommended citation
A. K. Shah and R. Zanibbi, “Line-of-Sight with Graph Attention Parser (LGAP) for Math Formulas,” in Document Analysis and Recognition - ICDAR 2023, G. A. Fink, R. Jain, K. Kise, and R. Zanibbi, Eds., in Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 2023, pp. 401–419. doi: 10.1007/978-3-031-41734-4_25.
Searching the ACL Anthology with Math Formulas and Text
MathDeck enables math formula search in the ACL Anthology PDF collection, integrating text and formula-based queries. The system introduces formula "chips" for intuitive formula creation, reuse, and annotation.
Recommended citation
B. Amador, M. Langsenkamp, A. Dey, A. K. Shah, and R. Zanibbi, “Searching the ACL Anthology with Math Formulas and Text,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, in SIGIR ’23. New York, NY, USA: Association for Computing Machinery, Jul. 2023, pp. 3110–3114. doi: 10.1145/3539618.3591803.
A Math Formula Extraction and Evaluation Framework for PDF Documents
This paper presents a math formula extraction pipeline for PDF documents, leveraging character-based information to support math-aware search engines. The system includes a novel formula detector and a graph-based parser for structure recognition.
Recommended citation
A. K. Shah, A. Dey, and R. Zanibbi, “A Math Formula Extraction and Evaluation Framework for PDF Documents,” in Document Analysis and Recognition – ICDAR 2021, Cham, 2021, pp. 19–34. doi: 10.1007/978-3-030-86331-9_2.