Exploring Audio Tracks using ShortTimeFourier, AudioData, and SpatialMedian
Exploring Audio Tracks using ShortTimeFourier, AudioData, and SpatialMedian
Roman Parker
Intern, Wolfram Research
By using audio functionalities in WL alongside SpatialMedian and ShortTimeFourier, it is possible to combine audio tracks, adjust their volume, and mute a track when another track is loud at a low-level, which provides insight into how Audio computations work. Additionally, using the higher-level Audio libraries, including AudioIntervals and AudioGenerator, these same things can be accomplished, which provides another method with simpler and faster execution for each of these cases. These functions have several applications, including playing music in the background of a video, which is disabled when the people in the video talk, and synthesizing together multiple feeds, which can have their audio levels at different sections manually modified.
Importing the Data
Importing the Data
For this notebook, the Audio Cats and Dogs dataset from the Wolfram Data Repository will be used. This dataset contains 277 labeled audio tracks of cats and dogs, of varying lengths. This is a perfect dataset for this purpose due to it containing many distinguishable tracks which aren’t all the same length, making it as close to real-world applications of these programs as possible.
Import the data from the Wolfram Data Repository:
In[]:=
testaudio=Normal[ResourceData["Audio Cats and Dogs"]〚All,1〛];
Combining Audio Tracks via Different Techniques
Combining Audio Tracks via Different Techniques
Audio files can be combined by several functions, including Mean, RootMeanSquare, HarmonicMean, and by finding the SpatialMedian of the ShortTimeFouriers (averageAudio). In this section, these techniques are compared and visualized to show which functions provide the highest quality and/or the fastest combination of the Audio tracks.
Define a function which combines Audio files using ShortTimeFourier and SpatialMedian (exploration of the function at end of the essay):
averageAudio[l_List]:=Module[{samplerate,fourierlist,inlist,indata,dataofaudio,endpoints,cutdata,fourier,reshapen},samplerate=ShortTimeFourier[l〚1〛,1024]["SampleRate"];fourierlist=(((ShortTimeFourier[#1,1024]&)/@l)〚#1〛["Data"]&)/@Range[Length[(ShortTimeFourier[#1,1024]&)/@l]];inlist=Flatten/@ReIm[fourierlist];indata=Flatten/@inlist;dataofaudio=SpatialMedian[SpatialPointData[PadRight[indata]]];endpoints=SequencePosition[dataofaudio,Table[0.`,1000]];cutdata=dataofaudio〚1;;If[Length[endpoints]0,-1,endpoints〚1,1〛-1]〛;fourier=(#1+#2&)@@Transpose[Partition[cutdata,2]];reshapen=Partition[fourier,1024];Audio[InverseShortTimeFourier[reshapen],SampleRatesamplerate]];
Combine five of the Audio tracks using six different techniques:
In[]:=
differentAverages=Module[{t15=testaudio[[1;;5]]},{Timing[Mean[t15]],Timing[RootMeanSquare[t15]],Timing[averageAudio[t15]],Timing[GeometricMean[t15]],Timing[HarmonicMean[t15]],Timing[ContraharmonicMean[t15]]}]
Out[]=
0.390625,,0.015625,,6.25,,0.0625,,0.5,,0.1875,
Apply a lowpass filter to make the tracks easier to hear:
In[]:=
LowpassFilter[#,]&/@differentAverages[[All,2]]
Out[]=
,,,,,
From a sound fidelity perspective, both Mean and AverageAudio are about equal, while the other functions are not ideal, but Mean is significantly faster (around 400x on this data and around 600x on the first 20 elements of the list), giving it the advantage in most cases. Adding the lowpass filter makes all the “bad” tracks significantly better from a fidelity perspective, but it can cut out important information and they still aren’t as good as Mean and/or AverageAudio. Next, we will create various visualizations to compare the Audio tracks created by each of the functions to each other.
Plot a spectrogram of each file (adding LowpassFilter makes nearly no difference):
In[]:=
Column[Spectrogram/@differentAverages[[All,2]]]
Out[]=
Make a MatrixPlot of the ShortTimeFourier (time is on the Y axis, frequency on the X axis, color corresponds to value):
In[]:=
MatrixPlot[ShortTimeFourier[#]["Data"]]&/@differentAverages[[All,2]]
Out[]=
,
,
,
,
,
Plot the amplitude over time (nearly identical results with lowpass filter):
In[]:=
AudioPlot[#,PlotRangeAll]&/@differentAverages[[All,2]]
Out[]=
,
,
,
,
,
Plot a periodogram with and without the lowpass filter:
In[]:=
Periodogram/@differentAverages[[All,2]]
Out[]=
,