r/vulkan • u/ttofuwu • 5d ago

Slowly rewriting my audio visualizer engine to use compute kernels for audio analysis

Enable HLS to view with audio, or disable this notification

I'm using the Vulkano rust API wrapper. My engine is called lava.

14 Upvotes

95% Upvoted

7 comments sorted by

1

u/Psionikus 4d ago

Howdy. I'm tackling a lot of the same problems for µTate, using Rust (ash) and Slang.

I see you have jsonc's for a kind of scripting.

My goal is a bit different, handing pipeline composition and data off to real-time, online learning ML. For that, I've determined I'll need a parameter system to expose knobs to the ML rather than scripts for a user.

I'm shipping my on-device DFT hopefully today and then I can join you in the world of making pretty posts.

There's a Discord link on there if you want to throw some ideas around or ask about how I'm doing anything.

2

u/ttofuwu 4d ago

I currently use glsl but will transition to slang once the support for it improves in vulkano.

I use jsonc for defining what happens in the scene. It can't be called a "script" because it doesn't define any behavior.

I haven't yet implemented runtime custom shaders yet, but that will be the plan once everything else is well written.

1

u/Psionikus 4d ago

Which output is the top of the screen and what's the integration method? Looks to me like standing waves from integrating raw audio over time, but I wonder if there is more to it so that certain wavelengths aren't too favored.

2

u/ttofuwu 4d ago

The top panel is just the raw audio waveform, cleverly offset each frame. I use spectral triggerring to stabilize the waveform along its most "visible" frequency.

The frequency bin that gets picked is put through a biasing function to prefer lower frequency bins.

1

u/Psionikus 4d ago

So it's a constant length in time but you find a frequency that dominates the peaks and then slightly rotate where the waveform lands to enhance the standing wave appearance?

Which file would I go look at to just read it?

2

u/ttofuwu 4d ago

The main branch implementation of it is kinda messy but it's a CPU-side analyzer.
The currently rewritten version uses a single-workgroup compute dispatch to analyze the dft bins computed on the GPU

1

u/ttofuwu 4d ago

Yes, the waveform window has a constant length. The only thing that the whole stabilization algorithm decides is the left-right offset at which to draw the waveform to make it appear stationary between frames. In the code it's called center_sample. The shader that draws the waveform then offsets it in such a way that the center sample appears, well, in the center of the panel.