Projects
Multimodal Chemical Search
A system for searching chemical reactions, molecular structures, and text in scientific literature. It integrates text, SMILES, and reaction-based queries, linking extracted reaction details with molecular diagrams and textual descriptions. The interface provides structured reaction and molecule cards for easy navigation and retrieval, supporting chemists in literature exploration and data extraction.
ChemScraper
A fast and accurate molecule diagram parser using characters and graphics extracted from born-digital (vector) PDF files—without the need for OCR, GPU, or vectorization. It uses these outputs to create training data for a new approach to visual parsing of molecule diagrams in raster files (i.e., pixel-based formats like PNGs) using a multi-task, segmentation-aware convolutional neural network (CNN).
MathDeck
A system for searching PDF documents in a portion of the ACL Anthology, incorporating both formulas and text, displaying matched words and formulas in context. Its user-friendly interface includes formula 'chips' for easy formula creation, search, reuse, and annotation. MathDeck supports both LaTeX and visual formula editing.
Math Formula Extraction
A tool for detecting and recognizing math formulas from PDF documents using both PDF information (without OCR) and visual features through a multi-task learning framework based on ResNet-50 with line-of-sight (LOS) graph-based attention.
LgEval
A tool for graph-based evaluation of recognition results, with visualization tools for fine-grained error analyses at the level of input primitives.
Guitar chord recognizer
An application that predicts the chords when the Mel spectrograms of guitar sound are fed into a CNN.
Nepali Plagiarism Detector (NLP)
An application that detects plagiarized Devanagari text files using a self-built rule-based stemming algorithm and Cosine similarity.
AI Plays GTA 5: Simulating self-driving vehicles
A bike-riding agent in a virtual environment (GTA5), built using CNN, used for simulating self-driving vehicles.
Sarangi: Nepali lyrics emotions extraction (NLP)
A framework that categorizes songs written in the Devanagari script into four emotions using Naive Bayes.
Autocar
A self-driving car that can detect lanes, stop signs, traffic lights, and avoid collisions, built using Canny edge detection, Hough transform, Haar cascade classifier, and Arduino programming.
MathMate – advanced mathematical calculator
An android app that solves different types of mathematical equations, numerical computations, and calculus problems showing involved steps.