Introduction
Introduction
Sheet music is the musical notation used to indicate pitches, rhythms, dynamics, and more for a piece of music. Part of any musician’s training is to become fluent in sheet music and maximize the efficiency of one’s reading-to-playing time and accuracy. While music is an art that is best played by people, a computer reading and playing a melody could be very useful for practicing and assistance. Our project aimed to create an interactive module that can recognize notes and rhythms, take user input, and play the music.
image=
;
In order to create a function to read and play sheet music, we worked in three main distinct steps: creating an image recognition algorithm for important symbols and their location, adding manual user input to correct any mistakes, and turning the final data into audio.
Image Recognition
Image Recognition
The first step to recognizing sheet music is understanding the staff. Sheet music is made up of staffs, five parallel line segments. In order to recognize the lines of the staff, we first used the Blur, Binarize, and RemoveBackground functions to create a more readable image:
In[]:=
RemoveBackground@Binarize@Blur[image,{2,5}]
Out[]=
From there, we found that the function ImageLines can recognize the spaces in between lines more easily than the lines themselves, so we created this line:
In[]:=
ImageLines[RemoveBackground@Binarize@Blur[image,{2,5}],.1,.01,MaxFeatures->4,Method->"RANSAC"]HighlightImage[image,ImageLines[RemoveBackground@Binarize@Blur[image,{2,5}],.1,.01,MaxFeatures->4,Method->"RANSAC"]]
Out[]=
{Line[{{0.,100.21},{906.,82.09}}],Line[{{0.,115.007},{906.,90.4}}],Line[{{0.,126.053},{906.,103.775}}],Line[{{0.,90.0216},{906.,68.2428}}]}
Out[]=
The second step was getting the computer to recognize the dots of notes. Most notes (e.g. quarter notes and eighth notes) contain a filled - in dot whose latitude determines its pitch. By using ComponentMeasurements along with image manipulation functions, we were able to get the computer to recognize the coordinates of most notes. In order to maximize the image's efficiency with ComponentMeasurements, we used Binarize, TotalVariationFilter, and ColorNegate on the image:
In[]:=
newimage=ColorNegate@TotalVariationFilter[Binarize@image,.6]
Out[]=
Using ComponentMeasurements, we get:
In[]:=
notelocations=ComponentMeasurements[newimage,{"Centroid","EquivalentDiskRadius"},#AdjacentBorderCount==0&&50<#Area<1200&&#Elongation<.5&]
Out[]=
{2{{35.9953,113.607},8.25338},5{{86.25,106.294},7.5694},6{{321.861,105.971},8.13686},7{{599.221,105.658},7.24715},8{{135.633,99.3489},8.46284},9{{664.6,99.1459},8.15639},10{{389.935,98.3631},7.31273},12{{710.876,92.3},7.35613},13{{437.586,91.6818},7.71518},16{{757.63,86.7487},7.83797},21{{204.757,74.3503},7.71518},22{{253.62,73.3177},7.81764},24{{870.107,62.5578},7.42076},25{{822.308,63.0028},7.50605}}
In[]:=
HighlightImage[image,Circle@@@notelocations[[All,2]]]
Out[]=
As seen here, the function accurately recognizes all the quarter and eighth notes, while failing to recognize the half note due to its open center.
These combine to make our function DotDetector, which detects dots.
In[]:=
DotDetector[image_]:=Sort[1/2*#&/@#[[2]][[1]]&/@ComponentMeasurements[Blur[ColorNegate@TotalVariationFilter[Binarize@image,1.5],{0,.5}],{"Centroid","EquivalentDiskRadius"},#AdjacentBorderCount==0&&50<#Area<1200&&#Elongation<.5&],#1[[1]]<#2[[1]]&]
Turning Data into Audio
Turning Data into Audio
To start, we defined a couple association lists to allow for easier calculations:
In[]:=
notesLines = <|0->0,1->2,2->4, 3->5, 4->7,5->9,6->11 |>;dynamics=<|"pppp"->.1,"ppp"->.2,"pp"->.3,"p"->.4,"mp"->.5,"mf"->.6,"f"->.7,"ff"->.8,"fff"->.9,"ffff"->1|>;
To understand the data we must start with understanding the locations within the staff. With locations of the white spaces as found earlier, we take the average of those lines to find the middle line.
To find a specific object, we:
- use RegionDistance to calculate how far from the middle line the point is
- check whether it's above or below the middle line
- change to the vertical distance to the middle line using Pythagorean theorem
- adjust based on the average vertical distance between lines (using Geometric Mean)
It also includes an optional highlighted image of where on the staff it is looking.
- use RegionDistance to calculate how far from the middle line the point is
- check whether it's above or below the middle line
- change to the vertical distance to the middle line using Pythagorean theorem
- adjust based on the average vertical distance between lines (using Geometric Mean)
It also includes an optional highlighted image of where on the staff it is looking.
In[]:=
LedgerLines[image_,point_,echos_:True,spaces_:False]:=Module[{staffspaces,midline ,distance},If[spaces ===False,staffspaces = (*spaces*)Sort[#[[1]]&/@ImageLines[RemoveBackground@Binarize@ Blur[image,{2,5}],.1,.01, MaxFeatures ->4, Method -> "RANSAC"]],staffspaces =Sort[spaces,#1[[1]][[2]]<#2[[1]][[2]]&]];midline = (*3rd line of staff*)Line[{{Mean[#[[1]][[1]]&/@staffspaces],Mean[#[[1]][[2]]&/@staffspaces]},{Mean[#[[2]][[1]]&/@staffspaces],Mean[#[[2]][[2]]&/@staffspaces]}}];If[Length[staffspaces]!=4,"Uh Oh" (*if we don't have 4 spaces, slight issue*),distance =(*how many spaces/notes the note is from mid*)RegionDistance[midline,point](*distance to midline*)*If[RegionMember[BoundaryMeshRegion[Join[midline[[1]],{{ImageDimensions[image][[1]],0},{0,0}}],{Line[{1,2}],Line[{2,3}],Line[{3,4}],Line[{4,1}]}],point],-1,1](*above or below*)/(GeometricMean[Flatten@Table[staffspaces[[m+1]][[n]][[2]]-staffspaces[[m]][[n]][[2]],{m,3},{n,2}]](*vertical space between a line and a space*)*Sqrt[(midline[[1]][[2]][[1]]-midline[[1]][[1]][[1]])^2+(midline[[1]][[2]][[2]]-midline[[1]][[1]][[2]])^2]/(2*(midline[[1]][[2]][[1]]-midline[[1]][[1]][[1]]))(*adjust for vertical instead of direct*));{distance,If[echos,Echo@HighlightImage[image,point],HighlightImage[image,point]]}]]
Objects can either not take up any time (think accidental, measure bar, etc), take up time with silence (rests), or take up time with a noise (notes). For our purposes, we defined objects that do not take up time as having length zero, rests as having negative length, and notes as having positive length. We then run through the cases - an object with length zero will play no sound for no time, an object with negative length will play no sound for a set amount of time, and an object with positive length will play the corresponding note. The corresponding note is determined by adjusting the input - how many steps above the middle line, adjusted for what note the middle line is (based on clef), converted to a letter note, and then adjusted to be in the correct octave, with the corresponding sharp or flat tacked on. It makes a SoundNote of a note with that pitch, for the given length, instrument, and volume.
In[]:=
PlayNote[distance_, clef_:"Treble",length_:1,key_:<|0->0,1->0,2->0, 3->0, 4->0,5->0,6->0 |>,instrument_:"Piano",dynamic_:"f",accidental_:False]:=Which[length == 0,SoundNote[None, 0],length<0,SoundNote[None,-1*length],True ,SoundNote[notesLines[[Key[Mod[(Round[distance] + Which[clef == "Bass",-6,clef=="Alto",0,True,6]),7]]]](*which note within octave*)+12*Floor[(Round[distance] + Which[clef == "Bass",-6,clef=="Alto",0,True,6])/7]+(*octaves*)If[accidental == False,(*accidental*)key[Mod[distance ,7]],accidental],length,instrument,SoundVolume->If[NumberQ[dynamic],dynamic,dynamics[[dynamic]]]]]
We now conglomerate all the points and their sounds together into one Sound output. We take in information about the whole line (clef, instrument, the image, key, the tempo, where the spaces are), as well as an association between the points and their "lengths". We then go through all points check what type of object each note is by looking at the length. If it is a list with the first entry 0, that corresponds to an accidental, and thus redefines that entry of the key for that measure (given by its distance using LedgerLines). If it is just 0, it is a measure bar and resets the key for the next measure. If it is a list with the first entry 1, it is a dynamic and sets the dynamic to the second value. All three of these do not add to the music as a Sound Object. If a note is on top of the note after it, it plays that note at a set time (as opposed to sequentially) on top of the rest of the notes. Otherwise, it just plays the corresponding note from PlayNote for the point. All of the sound objects are joined with Sound to get the final line played
In[]:=
PlaySoundsTogether[image_,points_,key_:<|0->0,1->0,2->0, 3->0, 4->0,5->0,6->0 |>,clef_:"Treble",instrument_:"Piano",secperquarter_:1,echos_:True,whites_:False]:=Module[{sharpflat = key,dynamic = .6,ordered = Keys[KeySort[points,#1[[1]]<#2[[1]]&]],time = 0},Sound[DeleteMissing[Table[Which[ListQ[points[ordered[[j]]]]&&points[ordered[[j]]][[1]]==0,sharpflat[Mod[Round[LedgerLines[image,ordered[[j]],echos,whites][[1]]],7]]=points[ordered[[j]]][[2]]; Missing[],ListQ[points[ordered[[j]]]]&&points[ordered[[j]]][[1]]==1,dynamic = points[ordered[[j]]][[2]]; Missing[],points[ordered[[j]]] ==0,sharpflat = key; Missing[],ListQ[points[ordered[[j]]]]&&points[ordered[[j]]][[1]]==1,dynamic = points[ordered[[j]]][[2]]; Missing[],j<Length[ordered] && !ListQ[points[ordered[[j+1]]]] && (ordered[[j+1]][[1]]-ordered[[j]][[1]]) <= 10,Sound[PlayNote[Round[LedgerLines[image,ordered[[j]],echos,whites][[1]]],clef,points[ordered[[j]]]*secperquarter,sharpflat,instrument,dynamic],{time,time+points[ordered[[j]]]*secperquarter}],True,time = time+points[ordered[[j]]]*secperquarter;Sound@PlayNote[Round[LedgerLines[image,ordered[[j]],echos,whites][[1]]],clef,points[ordered[[j]]]*secperquarter,sharpflat,instrument,dynamic]],{j,Length[ordered]}]]]]
User Input and Dynamics
User Input and Dynamics
In order to ensure that the final music is accurate to the sheet music, we have a LocatorPane of the line to represent where the computer sees the symbols. This allows symbols to be moved, created, or deleted by the user to make the final product accurate to the music. Additionally, we have a menu to designate what symbol each specific point is (if it's a note or rest, which and how long; if it' s a dynamic, what dynamic; if it's an accidental or measure bar, etc). Each symbol is linked to an image and a length. These are represented on the line once the menu is updated. There is also another LocatorPane to refine how the lines (or more specifically, the spaces) are viewed.
There are also many different options for the line to be chosen through different dynamic menus: the tempo, time signature, clef, instrument, and key. The choices in each of these are sent in, with some slight refinement, to the PlaySoundsTogether function to be turned into the music for that line, which is either output to the notebook or a file, depending on the user's choice.
There is also the option to copy, in list form, what would need to be input into the function for future use.
The Code
The Code
The Module
The Module
Future Work
Future Work
Future work on this project would include improving the automatic note detection algorithm. As stated, the image recognition only works on filled-in notes. Half notes and whole notes must be entered by hand, which prove very tedious in some pieces. It would also be very useful to train the image not only to recognize the location of notes, but their length. With enough training data, a neural net could find the location and length of every note, eliminating any need for humans to manually enter information.
Other work would be centered around expanding the scale and knowledge of the program. Currently, it can only read one staff at a time, and it only knows the basic symbols. With future work (and perhaps better image recognition), our program could understand volume and tempo dynamics, read an entire page of music, and recognize the names of instruments to automatically play that line with that part. Also, the ability to play multi-line staffs would be useful.