The Cacophony Index

Can we estimate the health of an ecosystem from a digital audio recording?

(Part 2 in a series about Artificial Intelligence and New Zealand native birds.)

Inside a computer, 20 seconds of audio are represented by a sequence of 320,000 numbers.

20 seconds of audio, plotted as a waveform

Our challenge is to take that series of 320,000 numbers and extract one single number, a "Cacophony Index", that has some special properties:

• Birds nearby and birds far away increase the Index about the same.
• Background noises don't affect the Index very much.
• The Cacophony Index for two sparrows chirping should be higher than if there's only one.
• The Cacophony Index for a sparrow chirping and an owl hooting should be higher than for two sparrows chirping.

Wow, that’s a really hard thing to do! As happens often in this blog, we'll make the problem easier by adding in some assumptions:

 "Perfect is the enemy of good" - Voltaire

Are we justified in making all these assumptions?
 ...Well, no...

...but...         

... let's do it anyway.

Lets build something useful instead of freaking out that a perfect solution can't exist.

That means we're going to just ignore a whole bunch of nasty complications like “clipping”, “nyquist rate”, “attenuation”, “noise floor”, etc

Because PROGRESS!

• Most of the loud noises in the recordings are birds, not people or cars or machines.
• The recording is “clean”
• The birds and the recorder stay in the same place.
• No running water or ocean waves (!)
• The recording was taken in New Zealand (!!)

Great stuff! Lets look at the spectrogram:

The spectrogram is a visual representation of the spectrum of frequencies of a signal as it varies with time. 

We don’t care so much about the intensity of any given bird call, that mostly tells us how near or far the bird is.

We don’t care so much if the bird has a short call or a long call.

Background noise? That’s where the spectrogram is well.. noisy..

Count the number of times a yellow box is next to a blue box!
That's the heart of the Cacophony Index calculation.

What we’re really looking for is how the spectrogram changes over time.

Lets zoom in on that starting second and add a grid to isolate the signal in both time and frequency:

A little bit more math and we find the cacophony index for this particular audio recording is: 77

OK, you got me, I'm oversimplifying again!  ¯\_(ツ)\_/¯ If you want all the gory details, the code is on github.com

Lets talk Birds!

The Cacophony Index for 20 seconds of audio is just a number between zero and one hundred. By itself, not super useful.

If we make many recordings in the same location, we can plot how the Cacophony Index changes over time.  Here's one possible presentation of what that might look like over the course of a day:

You can clearly see the birds are more active during the day and less active during the night. The birds getting really noisy around sunrise and sunset, the "Dawn Chorus".

Even though the plot is a mock-up, the data is real. It's data from a real bird monitor, recorded near Christchurch, New Zealand over a three week period in November of 2019. We now have the technology to see how the Cacophony Index changes over a day, or a week, or even seasons, years or decades.

And that's exactly what the Cacophony Project are doing, using real audio recorded right here in New Zealand, uploaded continuously and automatically by people just like you! (Edit: Live! Check it out!)

I think that's awesome. Can we go deeper?

Now we have an automated way to track an ecosystem's health, what else can we do with the audio data?

Watch this space for an update using real AI using Tensorflow and some real world ethical problems.

The Cap Theorem and Quantum Gravity

Apologies in advance, this post is both extremely technical in multiple fields, and woefully incomplete, and not nearly as humorous as it ought to be. Dragons be here. I'm incredibly sorry.

CAP Theorem for distributed systems

Brewer's CAP Theorem tells us that every distributed computer system must sacrifice at least one of these properties:

• C: Consistency
• A: Availability
• P: Partition Tolerance.

Astonishingly, if we view the universe as a distributed system, then Quantum Field Theory appears to have (analogues of) each of the three properties from the CAP theorem. But at what cost? Paradoxes. So many paradoxes. The double slit experiment, the twin paradox, the EPR “spooky action at a distance” paradox. Many many more. What if QFT adds to the CAP theorem a fourth property we could sacrifice:

• C: Consistency
• A: Availability
• P: Partition Tolerance
• T: Time Never Flows Backwards (!!!)

One Million Boulders

Lets look at that last one, Time Never Flows Backwards. Suppose, inside a computer, we're trying to simulate one million boulders rolling down a mountain side. At every time step, we need to generate all the potential collisions between those million boulders, and then process them in the order in which the collision occurs. You're familiar with Newton's cradle? Every one of those collisions can change the magnitude, and order, of any subsequent collision. And worse, round-off error when dicing the time steps means that a collision over *here* can affect a collision over *there*.

(All the gory details can be found here.)

Starting to sound a little bit like quantum dynamics right?

So how do we solve it efficiently? By briefly reversing the arrow of time. We find all the collisions between those boulders in a given timestep, then, optimistically, we solve each boulder independently (“in parallel”) based on it's known potential collisions, as if the order of collisions didn't matter. Then we do a “Fix-Up” phase where we wind the time step backwards and correct any of the boulders where (A) the collision order was incorrect, and (B) the energy of the correction is above a certain tolerance. (In practice the tolerance is very small, this tolerance only serves to prevent certain pathological worst-cases)

Starting to sound a *lot* like quantum dynamics...

Spinfoam

So imagine the spinfoam. In my mind, I visualize it something like this:

Spinfoam sketch, incomplete

In Quantum Chromo Dynamics terms, every face you can see is “Colourless” = (Red + Green + Blue == Red + Red + AntiRed + Green + Blue). In this diagram, the past is down. It's the rigid fixed lattice and appears unchangable. The future is a soup of these faces to the top of the diagram, and the “present” is the coalescing region where the mobile soup phase-transitions into a fixed lattice. Naturally, each edge is the Planck length, equivalently Plank time.

You can even see what we'd call a 'particle', maybe an electron or a neutrino, zipping along at close to the speed of light. In the spinfoam, it appears as a disturbance in the otherwise orderly lattice.

• <technical> In this diagram, the colours satisfy the Pauli exclusion principle. To represent bosons, simply write integer values at every vertex, and require every cycle-over-edges to sum to zero.
• This 2D diagram with vertices, edges and faces represents {1xspace+1xtime} dimensions. If we axiomatically accept the Holographic Principle, then it might be possible to represent {3xspace+1xtime} dimensions using only vertices, edges, faces and solids.</technical>

Notice too that, at least in the bottom of the diagram (“past”), the laws of physics are symmetric, and invariant under rotations through both time and space. Despite this local invariance, the time dimension can still be identified by it's global properties. The arrow of time, entropy etc, really does exist and has physical meaning.

Mass

What would happen if we tried to simulate this spinfoam in a computer? Well, most obvious to me, is that 'time' in the simulation does not correlate with the amount of computation required to run the simulation. Indeed, the computation required to run the simulation depends primarily on the search activity to coalesce the soup, and it should be easy to find a computation model where that search activity has a cost that matches Einstein's General Theory of Relativity. i.e. the curvature of a region of space is related to the amount of mass in that region, G = m . r^-2

Speed

Now lets take that simulation, and instead of running it on one single computer, we instead run it on a distributed computer system. Suddenly, the CAP theorem applies, and our simulation must sacrifice C, or A, or P.... or.... or..... or... T? What if we could sometimes run our simulation backwards just for a moment, the same as we did when "Fixing up" the simulate of those million boulders. When something doesn't fit, just for a little bit, we'd dissolve that fixed lattice of the past and turn it back into the mobile soup of the future, then reform the lattice into a consistent whole.
From inside the simulation, we'd never be able to send information back into the past (That would be a C violation!), and yet we'd still get “spooky action at a distance” and all those other paradoxes.
But at what cost? Well, surprisingly, only a performance hit. Again, it should be easy to find a model of distributed computation overhead where this performance hit is in exact agreement with Einstein's *Special* Theory of Relativity. Specifically, it's the Lorentz Transform, γ = 1 / sqrt(1-v².c^-2)

Intermission

Okay, big deep breath. The plot-twist is coming up soon. Brace yourself.

String Theory (Science Fiction)

Almost everything I've written above isn't new or novel. It's just a rehash of various discarded String-Theory ideas from the 90s, but with different names and labels. From an experimentalist physicists point of view, String Theory is just not that interesting. In terms of knowing more about the universe we live in, String Theory is pretty much at a dead end. Why? Because it's not *testable*. We can't devise an experiment in the lab to determine if any one of the thousands of competing String Theories makes predictions which match our unique reality. If your theory isn't testable, if there's no way to determine if you theory approximates our universe better than the alternatives, then that's not "Science" with a capital 'S', it's more like Science Fiction with a whole lot more math.

Plot Twist

So here's the plot-twist: CAP Theorem + QFT is testable.
Here's how: Take that exact same familiar double slit experiment we all faithfully reproduced when we first found out about Quantum Mechanics.

• Setup-1: Use one slit, fire the wave/particle, measure the diffraction. (Gaussian)
• Setup-2: Use two slits, fire the wave/particle, measure the diffraction. (Interference pattern)

Now, if we compare Setup-1 with Setup-2, if CAP + QFT is true, then Setup-2 will suffer a tiny time-dilation associated with resolving the CAP constraints. If CAP+QFT is true, we could toggle between Setup-1 and Setup-2 and measure the tiny difference in time dilation.

How tiny? So tiny no-one has ever noticed it before.

...so tiny, it would be much much smaller than the time-dilation associated with the mass of the photon itself.

......so tiny, but, at least in theory, so measurable.

What happens if we go into the lab and measure the time dilation difference between Setup-1 and Setup-2, and that difference turns out to be non-zero?

Quantum Gravity and Friends.

So yeah, that's a testable theory of quantum gravity. It neatly explains why gravity is so weak compared with the other forces (aka the Hierarchy problem), and dramatically simplifies the particle zoo.

Furthermore, this theorem is fully consistent with the Copenhagen Interpretation, and even builds on it! By contrast, In this formulation, the many-worlds alternative, however appears to have a vanishingly strict interpretation.

Black holes? Yip.. (I'll let you puzzle that one through, it's actually quite cute :) Naked singularities? Nope.

It neatly explains the uncertainty principle. It's truly a quantum theory from the get-go. The randomness is real ("no hidden variables"), it's even required, but it's certainly not arbitrary or capricious.

All those crazy dimensions from String Theory? Oh yeah, the dimensionality is there, but they're no longer spatial in nature, they're more like properties stacked on the spinfoam.

There's even some tantalising hints on the nature of dark matter and dark energy and inflation in the early universe..

Anyways, I've probably said way too much, as always, if you have any questions, queries or opinions, please let me know in the comments section below!

Twenty years of Skidmarks

Lately I've been digging through the old Skidmarks archives, and I came across this wee gem from 1993, no doubt written with lots of help from Simon:

Function.q hite {di.q,dj.q,oset.l}

  UNLK a4

  MOVE.l d2,a0

  MOVE.l d1,d2:SWAP d2:EXT.l d2:ASL.l #7,d2:ADD.l d2,a0

  MOVE.l d0,d2:SWAP d2:EXT.l d2:ASL.l #1,d2:ADD.l d2,a0

  MOVEM (a0)+,d2-d3 ;d0-d1 xy d2-d5 p0-p3

  LEA 124(a0),a0:MOVEM (a0)+,d4-d5

         MOVE d0,d6:MULU d1,d6:SWAP d6:MULU d6,d5  ; x. y.p3

  NOT d0:MOVE d0,d6:MULU d1,d6:SWAP d6:MULU d6,d4  ;-x. y.p2

  NOT d1:MOVE d0,d6:MULU d1,d6:SWAP d6:MULU d6,d2  ;-x.-y.p0

  NOT d0:           MULU d1,d0:SWAP d0:MULU d3,d0   ; x.-y.p1

  ADD.l d2,d0:ADD.l d4,d0:ADD.l d5,d0 ;total

  LSR.l#6,d0:RTS

End Function

For those not quite brave enough to decipher the 68000 assembly, here's what a strictly literal translation might be:

float Height(float x, float y, short *heightField){

  //__asm{...}

  heightField += int(x) * 64;

  heightField += int(y);

  short height2 = *heightfield++, height3 = *heightField++;

  heightField += 62; short height4=*heightfield++, height5 = *heightField++;

  float result5 = frac( x) * frac( y) * height5;

  float result4 = frac(-x) * frac( y) * height4;

  float result2 = frac(-x) * frac(-y) * height2;

  float result0 = frac( x) * frac(-y) * height0;

  float result = (result0 + result2 + result4 + result5);
  return result / 64;

}

The motivation is that I've been working towards a new version for mobile devices, with a working title of "Super Skidmarks 2000" (hashtag #SS2K)

Here's what the modern version of that same function looks like, this time in C++ :

float SKTrack::GetHeight(float axisI,float axisJ)const{

  int fi=(int)floor(axisI);

  int fj=(int)floor(axisJ);

  if(fi<0||fi>=63){

    return 0.0f;

  }

  if(fj<0||fj>=63){

   return 0.0f;

  }

  int index=fi+fj*64;

  float v0=HeightField[index];

  float v1=HeightField[index+1];

  float v2=HeightField[index+64];

  float v3=HeightField[index+65];

  float s=axisI-floor(axisI);

  float t=axisJ-floor(axisJ);

  float v01=v0*(1.0f-s)+v1*s;

  float v23=v2*(1.0f-s)+v3*s;

  float height=v01*(1.0f-t)+v23*t;

  return height;

}

As always, any questions / comments, or suggestion for a better name etc, please leave a comment below!

The Squeeze

I've been adding a lot of content to ScooterBoy over the past week, and it's been causing some problems.

In particular, the size of the archive had grown to over 100MB.

Increasingly, I've come to realize that speed of iteration is one of the biggest factors in determining overall project success.

The large archive size was causing slow sync times with the tablet, and even longer upload times on my rural broadband connection.

Something had to be done.

Enter WebP

WebP is a relatively new graphics format (2010), published by Google, both patent-free and open-source.

It's like a super-JPEG and a super-PNG all rolled in to one. Oh, and it fully supports alpha!

Here's one I prepared earlier: (click through for full size images)

Original PNG, 540KB

WebP version, 23KB

(WebP version as lossless PNG, for comparison only)

In this particular example, that's a whopping 95.7% saving in space!!

Obviously results will vary, but I'm consistently seeing 75% - 90% save spacing over the equivalent PNG/JPG, for a reasonable reduction in quality for tablet devices.

I'm also finding that the compression artifacts are of a type that is less annoying than PNG or JPEG. (Apart from some YUV color noise at very low bitrates - hopefully google will fix this in an update)

And it was super easy to integrate into my codebase.

Want even more info? Go grab the source over at developers.google.com


And my archive size? Now down to a very manageable 22MB.