The Cacophony Index

Can we estimate the health of an ecosystem from a digital audio recording?

(Part 2 in a series about Artificial Intelligence and New Zealand native birds.)

Inside a computer, 20 seconds of audio are represented by a sequence of 320,000 numbers.

20 seconds of audio, plotted as a waveform

Our challenge is to take that series of 320,000 numbers and extract one single number, a "Cacophony Index", that has some special properties:

• Birds nearby and birds far away increase the Index about the same.
• Background noises don't affect the Index very much.
• The Cacophony Index for two sparrows chirping should be higher than if there's only one.
• The Cacophony Index for a sparrow chirping and an owl hooting should be higher than for two sparrows chirping.

Wow, that’s a really hard thing to do! As happens often in this blog, we'll make the problem easier by adding in some assumptions:

 "Perfect is the enemy of good" - Voltaire

Are we justified in making all these assumptions?
 ...Well, no...

...but...         

... let's do it anyway.

Lets build something useful instead of freaking out that a perfect solution can't exist.

That means we're going to just ignore a whole bunch of nasty complications like “clipping”, “nyquist rate”, “attenuation”, “noise floor”, etc

Because PROGRESS!

• Most of the loud noises in the recordings are birds, not people or cars or machines.
• The recording is “clean”
• The birds and the recorder stay in the same place.
• No running water or ocean waves (!)
• The recording was taken in New Zealand (!!)

Great stuff! Lets look at the spectrogram:

The spectrogram is a visual representation of the spectrum of frequencies of a signal as it varies with time. 

We don’t care so much about the intensity of any given bird call, that mostly tells us how near or far the bird is.

We don’t care so much if the bird has a short call or a long call.

Background noise? That’s where the spectrogram is well.. noisy..

Count the number of times a yellow box is next to a blue box!
That's the heart of the Cacophony Index calculation.

What we’re really looking for is how the spectrogram changes over time.

Lets zoom in on that starting second and add a grid to isolate the signal in both time and frequency:

A little bit more math and we find the cacophony index for this particular audio recording is: 77

OK, you got me, I'm oversimplifying again!  ¯\_(ツ)\_/¯ If you want all the gory details, the code is on github.com

Lets talk Birds!

The Cacophony Index for 20 seconds of audio is just a number between zero and one hundred. By itself, not super useful.

If we make many recordings in the same location, we can plot how the Cacophony Index changes over time.  Here's one possible presentation of what that might look like over the course of a day:

You can clearly see the birds are more active during the day and less active during the night. The birds getting really noisy around sunrise and sunset, the "Dawn Chorus".

Even though the plot is a mock-up, the data is real. It's data from a real bird monitor, recorded near Christchurch, New Zealand over a three week period in November of 2019. We now have the technology to see how the Cacophony Index changes over a day, or a week, or even seasons, years or decades.

And that's exactly what the Cacophony Project are doing, using real audio recorded right here in New Zealand, uploaded continuously and automatically by people just like you! (Edit: Live! Check it out!)

I think that's awesome. Can we go deeper?

Now we have an automated way to track an ecosystem's health, what else can we do with the audio data?

Watch this space for an update using real AI using Tensorflow and some real world ethical problems.

Engineering in the Native Forest

Part 1 in a 2 part series about Artificial Intelligence and New Zealand native birds.

We’ve all heard that sound, and it is glorious. Native birds singing in pristine native forest.


Then tragedy happens. It could be an introduced pest like possums or rats. Maybe the climate changes and the native birds cannot adapt. Maybe it’s just a really really really bad year for bird flu.

The once vibrant healthy forest falls quiet. The native bird population is in crisis.

Meanwhile, over on social media:

I think it’s awesome when people are passionate about their local environment. I think it’s amazing when folks break out of their comfort zone and try to bring about positive change.

We all know that blindly doing the first thing that pops into your head is rarely the best course of action. Even with the best of intentions, when it comes to the environment, there’s just far too many ways to make the situation worse.

Fortunately, we can use Engineering!

• First, we measure the health of an ecosystem.
  
  
• Next, we apply an intervention:
  
  • pest trapping
  • a breeding program
  • fences
  • [Your idea here]
• Then, we measure the health of the ecosystem a second time.

Mix in a little bit of math, and now we can figure out which interventions are the most effective.

Those interventions which are more (cost) effective? We'll do more of those.

The interventions which have no effect, or worse, are damaging? Well, let's not do that again!

Simple right?

Well how do we measure ecosystem health?

Right now, in New Zealand, the gold standard is a manual process. Listeners walk out into the forest, and for five minutes, makes a record of all the birds they can hear on a piece of paper. Those pieces of paper are all brought together and another person manually enters all that data into a computer.

What if there was a way to estimate ecosystem health directly from an audio stream instead? Then we could leave recorders out in the forest, and monitor them remotely. More data, more timely, more consistency.

I'm good with computers and signal processing and things, maybe I can help...

Find out more over on the 2040 blog, or read Part 2

Time Pressure

A buddy of mine was sharing some cool ideas for a new game when we got to talking about Time Pressure. "Blog post!” I thought!

Time Pressure in Games

If you think of classic games, like Chess, or Tennis, you'll find that most classic games have some sort of time pressure mechanic. A skilled player can apply this pressure to increase the number of “unforced errors” of their opponent.



A great example of time pressure in Ice Hockey is the “Power Play”. When a player commits a penalty, they're sent to the penalty box, giving a 5-4 player advantage on the ice to the other team. As the penalty timer ticks down to zero, the attacking team is under increasing pressure to score a goal and take advantage of the situation.

For a hockey fan, watching at home on TV, the “Power Play” also increases pressure. Either there will be a goal, or the attacking team will make a mistake and squander the opportunity. In either case, the pressure is on the spectator to keenly observe and interpret the action before the penalty clock runs down to zero.

In the casual games, “Dumb Ways To Die” and “WarioWare”, the player must complete amusing tasks, but under tighter and tighter time constraints. At least in the early stages of play, the player actions would be easy to do, if not for the added pressure that comes from the timer.

Curiously, in these types of games, as the player skill increases, the gameplay shifts and becomes more reaction and twitch based. When played at this level, the time pressure is almost completely removed. It's similar to the way the rich, multi layered time pressures in a game like Tennis are largely absent from Ping Pong, which is essentially the twitch-based version of the same game.


My mobile game, ScooterBoy, is a mashup between two popular genres, the endless runner and slicing games. The twist is that when the player slices a spore to get points and combos, the same action also causes ScooterBoy to jump/duck/change lanes. In essence, you're playing two different games simultaneously. For skilled ScooterBoy players, the endless runner determines the length of time for each game, while it's your ability to make combos in the slicing game which most affect your score. In this way, in ScooterBoy, the endless runner acts to apply time pressure on the slicing game.

Time Pressure == A Complication

The common theme across all of these games is that time pressure adds a complication to an already fun activity.

As a game designer, we can turn this observation inside out. If we know that adding time pressure is equivalent to adding a complication, then we are obliged to verify that our game design is still fun, even when the time pressure is removed.

Indeed, if we're following the (so called) “Rational Game Design” principles, we should be able to strip our game down to it's core, removing all the complications... Then we add the complications back in, one at a time and in combinations, in order to maximise player enjoyment.

Under this framework, Time Pressure is just one of many different types of complications we could add, and this means (in general) we can assume a priori, that we could add and remove Time Pressure at any time to our game design, without affecting the strength of the design itself.

Phrased another way, Time Pressure acts as a multiplier to increases the intensity of your underlying experience.

Time Pressure and Difficulty

Lets drill down to the first few moments of your game. First impressions. In free-to-play, these first few minutes of gameplay is where you make or lose your players.

For players familiar with your genre, the time pressure does nothing. They zip through the first few levels with perfect scores, and your effort implementing and balancing the time pressure mechanic has contributed nothing.

For less skilled players, those unfamiliar with your genre, the time pressure adds confusion and failure, additional UI, and worst of all, the pressure increases unforced errors and perceived difficulty. All your hard work to implement time pressure serves to drive off more casual players.

A time-pressure mechanic (generally) makes things *harder* for weaker players, and leaves stronger players unchanged.

This is the exact opposite of how difficulty scaling is supposed to work.

You can compound this disaster by awarding the player a power-up for completing a level in a short amount of time. Here you are explicitly making the game easier for your most skilled players.

Sandbox games with broad market appeal, I'm thinking games like “Disney Infinity” and “Grand Theft Auto” here, commonly avoid these traps by making time pressure optional. There are timed challenges on the map which the player can initiate, but the player isn't required to complete them to advance the player's personal narrative.

Time Pressure as Exotic gameplay

So how do us Indies make time pressure fresh and original? We can take inspiration from some recent games which use the passage of time to completely subvert conventional notions of gameplay.


At first glance, “Braid” appears to be a platformer, but it's actually a puzzle game that explores all manner of time based mechanics.

I won't spoil it for you, but “Five Nights At Freddies” has some amazing time-pressure mechanics where the player has extremely minimal interaction with the game.

I'm sure there's lots more examples you can share with us in the comments below.

Time Pressure, it's been done! Time to do something different.