Creating interactive audio visualisations

A colourful illustration of a man walking on soundwaves as his laptop plays musical notesFor years, one of the great strengths of writing code for the browser was the short feedback loop. Write some HTML, Javascript and CSS, hit refresh. See it sparkle. Rinse and repeat. In time, we've lost some of that power; web apps are a lot more complex these days, code is often compiled and our applications have a lot more state, which is lost on refresh and so our quick feedback loop is gone. Figwheel is a library for Clojurescript that aims to give us some of that power back, provided you write "re-loadable" code. I wasn't really sure what that would entail, so I set out to make a project - a tiny library for creating audio visualisations, that would let you change the graphics while the music is playing.

The key enabler, though in the end a small part of the library, is the Web Audio API. It's somewhat peculiar if you've never worked with audio; it allows you to create nodes and connections between them to create an audio routing graph. For example, we may create a node which will be the audio source (like the microphone or an audio file), route that to some effect node (like a filter or reverb) and then route the output to the destination node, like our system speakers. These graphs can get quite complex as many nodes can have multiple inputs and outputs. It's how a lot of audio hardware works too, with cables, instruments, amplifiers etc, so is intuitive if you worked with that, but maybe less so without that background. Though for my use case, the whole interaction is contained within three functions:

(defn create-contexts [nodes]
  (let [audio-ctx (new js/AudioContext)]
    {:audio-ctx audio-ctx
     :render-ctx (.getContext (:canvas nodes) "2d")
     :analyser (.createAnalyser audio-ctx)}))

(defn connect-audio [{:keys [analyser audio-ctx track-src]} source fft-size]
  (set! (.-fftSize analyser) fft-size)
  (.connect source analyser)
  (.connect source (.-destination audio-ctx))

(defn get-bytes! [analyser freq-data]
   (.getByteFrequencyData analyser freq-data) 

In the first function, we create our AudioContext (which is the container where the audio graph will exist) and an Analyser node, which can turn a sound source into data. connect-audio builds our tiny graph - it connects the source to the analyser and to the destination. The last function gets the current data of the Analyser node and puts it into an Uint8Array, a special, typed array of 8-bit unsigned integers, something modern Javascript provides for performance sensitive things, like processing audio.

A moving picture with a red demonstration of sound

When starting the project, I quickly noticed a limitation - browsers allow only a limited amount of AudioContext's to be created. In Chrome the limit is 6. So we need to be careful when we run our constructors and close them if they are no longer needed. But if we close our AudioContext, the audio graph will be destroyed and the track will stop playing. This is not what we want. On changes, Figwheel recompiles and reloads the file we changed (which by Clojure's convention contains a single namespace), and all the code within it will be re-run: functions will be redefined, bindings will be re-bound. If we bound events or  appended some nodes to the DOM, that will be done again. This might sound familiar if you ever built a SPA application with Backbone (or similar js framework) - when changing a view we need to “clean up” after ourselves and make sure only the “state” that we want is preserved between different views.
Figwheel and Clojure provide us with tools to handle this situation. If we want to clean up at reload time, we can pass an on-js-reload function to Figwheel and if we do not want to reload some var we can define it with defonce. But the more important bit that needs to change is the structure of our code, which to be reloadable needs to fall into one of four categories:

  1. One-off, initialisation code
  2. Runtime data, that configures the app (need the initialisation code to be run to take effect)
  3. App state, the data that flows through the app (does not exist when we start the app)
  4. Business logic

In trying to make code reloadable, this separation became more clear.

Here's a very basic visualiser:

(ns cljs-analyzer.example
  (:require [cljs-analyzer.core :as c]))

(defn render [{:keys [analyser render-ctx bin-count]}
              {:keys [freq-data width height] :as config}]
  (c/clear-canvas render-ctx config)
  (let [bytes (c/get-bytes! analyser freq-data)]
    (set! (.-font render-ctx) "16px serif")
    (loop [i 0]
      (let [val (aget bytes i)]
        (set! (.-fillStyle render-ctx) "rgb(255,255,255)")
        (.fillText render-ctx val (+ 10 (* i 30)) 50)
        (when (< i bin-count)
          (recur (inc i)))))))

(defonce state (atom {}))

(def config
  {:freq-data (js/Uint8Array. 64)
   :render render
   :width (.-innerWidth js/window)
   :height (.-innerHeight js/window)
   :track "/audio/heavy-soul-slinger.mp3"
   :background "rgb(0,0,0)"})

(defn on-js-reload []
  (c/reload config state))

The contents of the render function draw the contents of the Uint8Array into a 2d canvas. state is defined with defonce and contains an atom, an asynchronous data structure that will let us interact with the state of the app while it's running. config holds the app configuration. In the reload function, render will get replaced on each reload as the function passed to requestAnimationFrame, so we can smoothly change our graphics while the music is playing. If we want to reload our AudioContext the core namespace provides also setup and teardown to make a full reload. These can be called from the REPL.

This design isn't perfect and I'm still improving on it, but I found that when I understood the division in my code better, the implementation got simpler and I achieved it with less code. Much of the complexity and problems on the way were due to mutative API's, Web Audio, the DOM (which I've omitted here for sake of brevity) etc, but also the “magic” when working with the REPL as a beginner. Figwheel provides one that connects to your browser and reloads your re-compiled code. This sounds great, but I often found it hard to track what is loaded in the REPL and what is not. The problem lied mostly in between how I structured my application and how Figwheel expected it to work (I have multiple “entry” namespaces in the app, while Figwheel expects one). Not surprisingly, better understanding of how the Figwheel REPL works led to more predictable usage.

My biggest takeaway is that a methodical approach with Clojure/Clojurescript is a must, more than in languages like Javascript or Ruby: as hosted languages, they expose their guts quite quickly and you are expected to get into the rabbit hole from time to time and understand the structures you are working with better (not dissimilar to working on a Linux OS), both on host and hosted level. This usually ends up in better design and more effective use of its abstractions, but can be daunting in the beginning, especially when you are keen “shiny” features like the REPL, code reload etc.

When working with Clojure I often thought the design of the language takes priority in building good abstractions, in contrast for example to Ruby, where a lot of the API's take priority in being convenient to the programmer, intuitive, but grander designs often get bloated and convoluted, because more complex abstractions are left to libraries and these don’t necessarily work well together between each other. I felt Clojure is often less forgiving than other languages because the fundamentals of the language are based less on programmer familiarity and more being general, but gives more flexibility later down the road, allowing for more adequate designs. Even if you don’t see yourself doing Clojure professionally, I think it’s a great language to pick up and learn because it’s such a departure from other mainstream languages and can inform your programming in other languages.

Here at Red Badger, we're always seeking to learn more by adopting the most cutting-edge technologies. If this sounds like something you would enjoy check out our software engineer role and get in touch

Sign up to Badger News