Oscilloscope Music

I recently came across an interesting art project called "Oscilloscope Music" by Jerobeam Fenderson:

Here's another excellent example by Chris Allen:

The video shown above is an XY trace taken on an oscilloscope which is being fed a carefully crafted audio file. The oscilloscope is drawing a single bright point of light, with the left audio channel controlling the X axis position of that point and the right audio channel controlling the Y axis position. Varying the amplitude of those two channels together allows that point of light to be used as a pen, drawing shapes on the 2D screen. You can find a much more detailed explanation of what's going on here from Smarter Every Day.

As soon as I saw this, I immediately wanted to see if I could recreate the same kind of visualization from that same audio. I do have an oscilloscope, but it's all the way in the other room, so I decided to see if I could recreate the result in code.

To do that, I started up Julia 1.4 and created the Jupyter notebook you see here.

Goals

My goal here is to take the audio track from a source like this video as input and recreate the video, showing the result you would see by playing that audio file into the X and Y channels of an oscilloscope.

Approach

To do this, I'm going to emulate the behavior of the oscilloscope trace. I'll start off with a black canvas. At each audio sample, I'll use the left and right channel amplitudes to pick an X and Y coordinate in that canvas, and I'll make that pixel white. This is basically what the oscilloscope does in XY mode: it is always drawing a bright dot, and the X and Y channels determine where that dot is.

You'll notice from all the oscilloscope music videos, however, that you can see more than just a single dot at a time. In fact, you can see smooth lines that don't seem to be made up of individual points. This happens because the oscilloscope display itself has some persistence. An electron beam is used to illuminate a single point on the screen, but when that beam moves elsewhere, it takes some time for the previous point to fade to black. Furthermore, if the beam is quickly moved across an area, the whole path of the beam will be illuminated and will only slowly fade back to black. We'll have to be careful to replicate that persistence when we digitally recreate the effect.

Code

To start off, we'll need to import a few Julia packages to handle our various inputs and outputs:

In:
# WAV.jl lets us read .wav format audio files
using WAV: wavread

# Images and ImageMagick handle loading and working with images
using Images
using ImageMagick

# ProgressMeter lets us use `@showprogress` to turn a loop 
# into a progress var
using ProgressMeter: @showprogress

# We'll use FFMPEG to strip the audio out of the Youtube video
using FFMPEG

# For...you know...plots
using Plots

Reading Audio Data

The source I'll be working with is "Blocks" by Jerobeam Fenderson. I've already downloaded the full video file using youtube-dl`:

In:
video_file = "Jerobeam Fenderson - Blocks-0KDekS4YUy4.webm";

We only need the audio from that file, so I'll use FFMPEG to strip out the audio and save it as blocks.wav:

In:
@ffmpeg_env run(
    `$(FFMPEG.ffmpeg) -i $video_file -vn -ab 128k -ar 44100 -y blocks_audio.wav`)

We can read the audio input with wavread, which returns a matrix of samples. Each column of samples is one left/right audio channel.

In:
samples, sample_freq = wavread("blocks_audio.wav");

Let's plot the first 1s of audio, just to see what it looks like:

In:
# Samples from 0.0 to 1.0 seconds
sample_range = round.(Int, 1 .+ (0 * sample_freq : 1 * sample_freq))

plt = plot(sample_range, samples[sample_range, 1], 
    label="Channel 1 (X)",
    xlabel="Sample index",
    ylabel="Amplitude"
    )
plot!(plt, sample_range, samples[sample_range, 2], 
    label="Channel 2 (Y)")
Out: