"Exercise: Understanding Audio Features through Sonification"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is an *exercise* notebook. It's a playground for your Python code. Feel free to write and execute your code without fear.\n",
"\n",
"When you see a cell that looks like this:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"plot?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 49
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"that is a cue to use a particular command, in this case, `plot`. Run the cell to see documentation for that command. (To quickly close the Help window, press `q`.) \n",
"\n",
"For more documentation, visit the links in the Help menu above. Also see the other notebooks; all the exercises here are covered somewhere else in separate notebooks."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This exercise is loosely based upon \"Lab 1\" from previous MIR workshops ([2010](https://ccrma.stanford.edu/workshops/mir2010/Lab1_2010.pdf))."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Goals"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this lab, you will segment, feature extract, and analyze audio files."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Detect onsets in an audio signal.\n",
"2. Segment the audio signal at each onset.\n",
"3. Compute features for each segment.\n",
"4. Gain intuition into the features by listening to each segment separately."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Step 1: Retrieve Audio"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the file `simpleLoop.wav` onto your local machine."
"In this lab, you will segment, feature extract, and analyze audio files."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Goals"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Detect onsets in an audio signal.\n",
"2. Segment the audio signal at each onset.\n",
"3. Compute features for each segment.\n",
"4. Gain intuition into the features by listening to each segment separately."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Segmentation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" For debugging, we have function which generates mixes of the original audio file and onset times. This is demonstrated with `test_onsets.m`.\n",
"\n",
" One of Matlab's greatest features is its rich and easy visualization functions. Visualizing your data at every possible step in the algorithm development process not only builds a practical understanding of the variables, parameters and results, but it greatly aids debugging. \n",
" \n",
"8. Plot the audio file in a figure window. \n",
"\n",
" plot(x)\n",
"\n",
"9. Now, add a marker showing the position of each onset on top of the waveforms. \n",
"\n",
" plot(x); hold on; plot(onsets,0.2,'rx')\n",
"\n",
"10. Adding text markers to your plots can further aid in debugging or visualizing problems. Label each onset with it's respective onset number with the following simple loop:\n",
"\n",
" for i=1:numonsets\n",
" text(onsets(i),0.2,num2str(i)); % num2st converts an number to a string for display purposes\n",
" end\n",
"\n",
" Labeling the data is crucial. Add a title and axis to the figures. (ylabel, xlabel, title.)\n",
"\n",
" xlabel('seconds')\n",
" ylabel('magnitude')\n",
" title('my onset plot')\n",
"\n",
"11. Now that we can view the various onsets, try out the onset detector and visualization on a variety of other audio examples located in `/usr/ccrma/courses/mir2013/audio`. Continue to load the various audio files and run the onset detector - does it seem like it works well? If not, yell at Leigh.\n",
"\n",
" Segmenting audio in Frames\n",
" As we learned in lecture, it's common to chop up the audio into fixed-frames. These frames are then further analyzed, processed, or feature extracted. We're going to analyze the audio in 100 ms frames starting at each onset. \n",
"\n",
"12. Create a loop which carves up the audio in fixed-size frames (100ms), starting at the onsets.\n",
"\n",
"13. Inside of your loop, plot each frame, and play the audio for each frame. \n",
"\n",
" % Loop to carve up audio into onset-based frames\n",
"14. Create a loop which extracts the Zero Crossing Rate for each frame, and stores it in an array. Your loop will select 100ms (in samples, this value is = fs * 0.1) , starting at the onsets, and obtain the number of zero crossings in that frame. \n",
"\n",
" The command `[z] = zcr(x)` returns the number of zero crossings for a vector x.\n",
" Don't forget to store the value of z in a feature array for each frame.\n",
"\n",
" clear features\n",
" % Extract Zero Crossing Rate from all frames and store it in \"features(i,1)\"\n",
" for i=1:numonsets\n",
" features(i,1) = zcr(frames{i})\n",
" end\n",
" \n",
" For simpleLoop.wav, you should now have a feature array of 5 x 1 - which is the 5 frames (one at each detected onset) and 1 feature (zcr) for each frame. \n",
" \n",
" Sort the audio file by its feature array. \n",
" Let's test out how well our features characterize the underlying audio signal. \n",
" To build intuition, we're going to sort the feature vector by it's zero crossing rate, from low value to highest value. \n",
"\n",
"15. If we sort and re-play the audio that corresponds with these sorted frames, what do you think it will sound like? (e.g., same order as the loop, reverse order of the loop, snares followed by kicks, quiet notes followed by loud notes, or ??? ) Pause and think about this. \n",
"\n",
"16. Now, we're going to play these sorted audio frames, from lowest to highest. (The pause command will be quite useful here, too.) How does it sound? Does it sort them how you expect them to be sorted? \n",
"\n",
" [y,index] = sort(features);\n",
"\n",
" for i=1:numonsets\n",
" sound(frames{index(i)},fs)\n",
" figure(1); plot(frames{index(i)});title(i);\n",
" pause(0.5)\n",
" end\n",
"\n",
" You'll notice how trivial this drum loop is - always use familiar and predictable audio files when you're developing your algorithms. \n",
"\n",
"17. Now that you have this file loading, playing , and sorting working, try this with out files, such as:\n",
