Point a phone at printed sheet music and get back playable MIDI.
Stack
Python · PyTorch · OpenCV · CRNN+CTC · PrIMuS
Purpose
Optical music recognition already works on flatbed scans. It does not work on photographs. A picture taken by hand adds perspective distortion, shadows, and JPEG compression noise, and existing readers drop or misread notes because of it. That is why the technology has stayed on desktop scanners instead of phones.
How it works
The first version, built for CS 131, cut the page into individual symbols and classified them one at a time. That approach failed on photographs. One bad cut deletes a note, and every later stage works from the broken input, so a single error spreads through the whole pipeline. Exact pitch accuracy on real photographs was 10 percent.
The rebuild removes segmentation. A convolutional encoder turns a staff into a sequence of feature columns. A two-layer BiLSTM reads that sequence from left to right. A CTC head outputs the notes. Because the recurrent layers can see the spacing between noteheads, the model predicts rhythm and pitch in the same pass.
Impact
Exact pitch accuracy on real photographs rose from 10 percent to 95 percent. Duration accuracy reached 85 percent; neither earlier version could read durations at all. Holding the architecture and the data fixed and changing only the augmentation reduced the accuracy gap between clean and photographed input by 99.4 percent. The model has 3.1 million parameters, trains on 15,354 staves from the PrIMuS corpus, and finishes training in minutes on a laptop GPU.
