About me
🔍 I am a Research Scientist at Meta in the Applied AI (AAI) org, working on the post-training of Meta’s frontier large language models, the Muse family. My contributions span:
- 🧠 LLM research: post-training, data, and evaluation methods that improve reasoning, instruction-following, and overall model quality.
- 🎙️ Speech, audio & music: my current focus, advancing Muse’s understanding and generation across the audio domain.
- 💻 Coding: expert-level tasks, benchmarks, and training data that strengthen the models’ software engineering abilities.
Before joining AAI in June 2026, I was on Meta’s Media Foundation Video team (AI), building transformer- and diffusion-based models for video enhancement, compression, and quality optimization across Meta’s Family of Apps (Facebook, Instagram, WhatsApp, Messenger), and collaborating with the Meta Superintelligence Lab on large vision models for video generation and multimodal understanding.
🎓 I completed my Ph.D. at Rochester Institute of Technology (RIT) in 2025, conducting research at the Document and Pattern Recognition Lab (DPRL), under the mentorship of Dr. Richard Zanibbi. My dissertation, Parsing of Math Formulas and Chemical Diagrams using Graph-Based Representation and Attention Models, designed fast, efficient, and interpretable parsers for recognizing math formulas and chemical diagrams across PDFs, typeset images, and handwritten strokes, using graph attention in a multi-task learning framework. Earlier, I was an Applied Scientist Intern at Amazon (Alexa Speaker Understanding AI), improving speaker identification with semi-supervised learning.
🧑⚖️ I serve as a peer reviewer for Pattern Recognition and IJDAR, and on the ICDAR program committee (2023, 2025, 2026), where I was named an IAPR Outstanding Reviewer in 2025. I am also a Full Member of Sigma Xi. See my review service.
💻 Ph.D. Projects:
- Multimodal Chemical Search: A multimodal search tool for retrieving chemical reactions, molecular structures, and associated text from scientific literature, linking visual and textual representations of chemical information.
- ChemScraper: A molecule diagram parser for extracting molecular graphics from PDFs.
- MathDeck: A math-aware search system supporting formulas & text.
🌐 Research interests: Large language models (post-training and evaluation), speech, audio and music AI, multimodal deep learning, code generation, computer vision, video understanding and generation, pattern recognition, and recognition of graphical structures.
📰 News
2026 4 updates
Gave a guest lecture, “Agentic AI in Industry: Context, Autonomy, and What Works”, for the Fusemachines AI Fellowship 2026 (online, Kathmandu, Nepal), covering context engineering, autonomy and what actually works when deploying AI agents. See all my talks.
Moved to Meta’s Applied AI (AAI) org to work on the post-training of Meta’s frontier large language models, the Muse family, contributing to LLM research, speech, audio and music, and coding capabilities.
Recognized by the editors of Pattern Recognition (Elsevier) for 7 reviews (Nov 2024 – May 2026), and served on the ICDAR 2026 Program Committee. See my review service.
Elected a Full Member of Sigma Xi, The Scientific Research Honor Society.
2025 9 updates
Named an IAPR Outstanding Reviewer at the 19th International Conference on Document Analysis and Recognition, ICDAR 2025, Wuhan, China.
Started as Research Scientist at Meta, working on the Media Foundation Video team (AI).
Successfully defended and passed my Ph.D. dissertation on “Parsing of Math Formulas and Chemical Diagrams using Graph-Based Representation and Attention Models“.
Featured in the April-May 2025 MMLI Spotlights (Student Spotlight). News published on the DPRL’s recent highlights: Dr. Zanibbi received RIT’s highest research award and gave an invited talk at the Perimeter Institute for Theoretical Physics, which featured AI scientists including 2024 Nobel Laureate, Geoffrey Hinton.
Likewise, DPRL students achieved major milestones including top internships and industry placements—demonstrating the lab’s growing impact through MMLI collaborations.
Presented a poster on “ChemScraper: Pipeline for Parsing Raster and Vector Molecule Diagrams from PDFs” at the Molecule Maker Lab Institute (MMLI) Symposium 2025 at University of Illinois Urbana-Champaign (UIUC).
Our paper on “Multimodal Search in Chemical Documents and Reactions” has been accepted for publication at Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, in SIGIR ’25, Padua, Italy.
The system is available online at ReactionMiner. ReactionMiner searches reaction descriptions in PDF documents using text and chemical structures provided as SMILES/SMARTS strings. Extracted reaction records associated with passages can also be browsed and exported.
A step-by-step video tutorial on the AlphaSynthesis tool, ChemScraper, is available on YouTube. ChemScraper is a fast and accurate molecule diagram parser using characters and graphics extracted from PDF files and raster images.
Submitted our paper on the “Multimodal Search in Chemical Documents and Reactions” for publication at Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, in SIGIR ’25, Padua, Italy. The system is available online at ReactionMiner.
Gave a talk on “Reaction Extraction from PDF: ChemScraper and ReactionMiner Integration” at the inaugural Molecule Maker’s Playhouse (MMLI all-institute meeting), Champaign, IL. Slides
2024 4 updates
Successfully defended and passed my Ph.D. dissertation proposal on “Parsing of Math Formulas and Chemical Diagrams using Graph-Based Representation and Attention Models”.
Gave an oral presentation on “ChemScraper: Leveraging PDF Graphics Instructions for Molecular Diagram Parsing” at the 18th International Conference on Document Analysis and Recognition, ICDAR 2024, Athens, Greece.
Presented “Integration of ReactionMiner and ChemScraper: Mining Chemical Reactions from Literature” at the MMLI All-Institute cross-thrust meeting, and co-presented the poster “ChemScraper: Extracting and Parsing Molecular Diagrams from PDFs and Images” at the MMLI NSF Site Visit poster session, UIUC. Also gave an online webinar, “Intro to AI: Exploring Fundamentals, Research, and Career Opportunities”, hosted by PANA Academy, Kathmandu, Nepal.
A revised paper on ChemScraper has been published at the 18th International Conference on Document Analysis and Recognition, ICDAR 2024, Athens, Greece – Journal Track. The paper describes (1) a fast and accurate technique for parsing born-digital (vector) PDF images, and (2) its use to create training data for a new approach to visual parsing of molecule diagrams in raster images (i.e., pixel-based such as from PNGs). Code is available and the system is online at ChemScraper.
2023 6 updates
A paper describing ChemScraper parser for molecular diagrams in PDF drawing instructions (‘born-digital’) is available on arXiv here. The system can also generate annotated training data for visual parsers that recognize raster images (i.e., pixel-based, such as PNG). A link to associated code is provided in a footnote in the paper.
Co-presented a poster on “ChemScraper: Extracting Molecule Diagrams from PDF Vector and Raster Images with CDXML and SMILES Output” at the Molecule Maker Lab Institute (MMLI) All-Institute Retreat at University of Illinois Urbana-Champaign (UIUC).
Gave a poster presentation talk at Poster session 1 and doctoral consortium at the 17th International Conference on Document Analysis and Recognition, ICDAR 2023, San José, California.
Co-presented a poster on “ChemScraper: Extracting Molecule Diagrams from PDF Vector Images with Page-Level CDXML (ChemDraw) and SMILES Output” at the NSF Annual Review Meeting at University of Illinois Urbana-Champaign (UIUC).
Gave a Research Idea Ring (RIR) talk on “Line-of-sight with Graph Attention Parser (LGAP) for Math Formulas” at RIT.
Our paper on the “Line-of-sight with Graph Attention Parser (LGAP) for Math Formulas” has been accepted for publication at the 17th International Conference on Document Analysis and Recognition, ICDAR 2023, San José, California.
2022 5 updates
Co-presented a poster on “Reconstructing the Structure of Molecular Diagrams in PDF Documents using a CNN-Attention-Based Parsing Model” at the Molecule Maker Lab Institute (MMLI) All-Institute Retreat at University of Illinois Urbana-Champaign (UIUC).
Gave a guest lecture on “Bayesian Decision Theory” for RIT’s undergraduate course - Intro to Machine Learning (40 students).
Successfully completed Applied Scientist Intern at Amazon (Alexa AI). Started as Graduate Teaching Assistant (GTA) for the undergraduate course CSCI-335 Machine Learning.
Started as Applied Scientist Intern at Amazon (Alexa AI). Worked on the Alexa Perceptual Technologies - Speaker Understanding team to improve speaker identification in Alexa devices.
Gave a Research Idea Ring (RIR) talk on “A Fast and Interpretable Context-aware Parser for Isolated Formulas and Chemical Diagrams” at RIT.
2021 4 updates
Gave a poster presentation talk on the MathSeer extraction pipeline at the 16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland virtually.
MathSeer extraction pipeline released. This tool extracts formula locations and content in PDF documents. The pipeline is available from GitLab, and includes improved versions of SymbolScraper, ScanSSD (now, ScanSSD-XYc), and QD-GGA. The pipeline was prepared by Ayush K. Shah, Abhisek Dey, Matt Langsenkamp, and Prof. Zanibbi.
Successfully defended and passed my Ph.D. Research Potential Assessment (RPA) on “Mathematical Formula Recognition”. Slides
Our paper on the “MathSeer formula extraction and evaluation pipeline” has been accepted for publication at the 16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland.
2020 2 updates
Joined Rochester Institute of Technology (RIT) for Ph.D. in Computing and Information Sciences. Started as Graduate Research Assistant (GRA) at the Document and Pattern Recognition Lab (DPRL) under Prof. Richard Zanibbi.
Promoted to Machine Learning Engineer Level 1 at Fusemachines Nepal.
2019 3 updates
Promoted to Machine Learning Engineer Associate at Fusemachines Nepal.
Graduated from Kathmandu University as a Computer Engineer.
Started working as a Machine Learning Engineer Trainee at Fusemachines Nepal.
