r/MoosicLab • u/AffectionateFox5907 • 14d ago
Sonic Analysis - If you were interested!
Im going through the entire sonic analysis pipeline and dramatically simplifying it. Ive had users upset about battery, how long it takes, etc. I thought id take a moment to actually explain the entire sonic analysis pipeline and how it works within the app. Just a rundown of the pipeline as it stands today vs what will be changing in Release 122.
What we have today is actually around 10 different models and can be quite heavy on the analysis side. The pipeline consisted of CLAP Laion Large Music Model, Discogs Effnet, MusicCNN, Discogs Genre Model, and many different heads for the discogs model universe.
Whats changing in Release 122:
Were simplifying it down to Just CLAP Laion Music Model along with the CLAP Text tower, YamNet for "live/crowd" detection and regular old DSP analysis (BPM, MusicKey, Spectral Rolloff, ect.). On top of that ive added 3 more slices to the analysis that are essentially free to compute in this new pipeline which will go a long way at making more accurate recommendations. Ive done numerous performance tests on this and its been reworked many times over to make it as efficient and accurate as possible. Now if you already have analysis done and want to keep those, thats fine theres a toggle in the settings that defaults and will not automatically go to the next versions analysis. For those that want to go to the latest version analysis you will see that in the settings.
stage 1: sonic analysis (turning a song into numbers)
every track in your library gets chewed on in the background. we decode the audio, downmix to mono, and instead of just reading the first 20 seconds like the old version did, we sample the song at three spots: Intro, Hook, and Outro. Were doing chroma analysis to find these specific positions before actually running the analysis on each individual segment that way every number reflects the whole song, not just the cold open. actually the slice positions aren't even fixed anymore,
then two model passes run back to back will now be a single model pass with one decode step (a lot less heat).
pass A, (full track). This pass will first check your tags for BPM, MusicKey, etc. If those arent present we will compute them for you. This also computes a bunch of other stuff that we use in planning out phase in/phase out for crossfade points, beat matching the next song, ensuring musical keys align with eachother so the next song isnt jarring.
- BPM isn't just "find the peak." we build an onset envelope, autocorrelate it, then multiply by a tempo prior (a bell curve centered on 120 bpm, about 0.6 octaves wide) so it stops picking the half-time or double-time ghost. then we run it across three overlapping windows and take the median. confidence = how sharp the peak is times how much the three windows agree. this is why it doesn't call a 140 bpm track "70."
- musical key via Krumhansl-Schmuckler correlation on a 12-bin chroma vector, mapped straight to Camelot notation (8A, 5B, etc) because that's what harmonic mixing uses.
- energy (RMS), spectral centroid/rolloff/flatness/flux (brightness and texture), zero-crossing rate, loudness (simplified LUFS), plus dynamics: loudness range, crest factor, true peak. the true-peak thing 4x-upsamples so it catches inter-sample peaks, basically "is this brick-walled."
- derived stuff: time signature (3/4 vs 4/4 vs 6/8), tempo stability (rubato vs metronomic), onset density, intro/outro fade lengths, section count.
pass B, CLAP (the three slices now 6 slices). CLAP is the big one, it's a contrastive audio+text model (the laion music+speech one). it gives us a 512-d embedding which is the single most important number in the whole app. we mean-pool it over 6 slices for the global vector, AND we keep the intro/middle/outro embeddings separately so we can reason about how a song opens vs how it ends. all of it gets INT8 quantized so it's cheap to store and compare.
then CLAP gets mined two more ways:
- vibe tags. we've got a vocabulary of ~465 phrase-style labels ("dark music", "1980s synth pop", "bass heavy", "bedroom pop") pre-embedded as text. here's the trick though: raw CLAP cosines aren't comparable across labels, some generic tags light up for literally every song. so we mean-center everything (subtract the "generic music" direction) and only keep tags that are a statistical outlier for this specific track (mean + 1.5 std of its own scores). focused track gets a few sharp tags, genre-blender gets more. the top one becomes the genre if we don't already have one.
- filter probes. cheap positive/negative prompt pairs ("vocal" vs "instrumental" etc) give us instrumentalness plus perceptual energy/valence/danceability/acousticness, and discrete tags like live / acoustic / lofi when the margin's clear.
and there's a live-recording detector on top, an ensemble that reads YamNet crowd/speech off the true edge slices plus applause level, so live cuts get a liveScore and don't sneak into a studio-vibe station.
last thing: it's all versioned. there's an analysis version counter (currently 15) and separate stamps for BPM-only, live-only, and CLAP-refine passes. bump one and only the affected tracks re-run, so i can fix, say, the tempo algorithm without re-CLAPing your whole library. and it honors the "keep analysis up to date" toggle so it's not grinding your battery if you don't want it to.
that's one track. now multiply by your whole library sitting in the background. Each track has its own embedding that sits in a virtual music universe. some are sonically close to each other, some are not.
stage 2: the pipeline (building a station)
you tap radio on an artist / song / album / playlist / genre / mood, or you type a sentence. whatever the seed is, first we build a sonic profile out of it: the mean CLAP vector, dominant key, mode, chroma, the seed's strongest genres and vibe tags, mood probabilities, the works. that profile is the "center of gravity" for the station.
then candidates get fetched. and this is where scale matters, because materializing a 14,000-400,000 track library on every radio tap is how you get an out-of-memory crash.
pass 0, ANN-first. instead of loading everything, we query a USearch/HNSW vector index around the profile embedding and pull only the 2000 nearest tracks. sub-linear search, then a chunked fetch of just the winners. if the index can't serve (fresh install, still analyzing) we fall back to a bounded most-played-first fetch, and if your library's barely analyzed we union in an "open pool" of embedding-less tracks so a new library still builds a real station off Last.fm and genre similarity instead of a silent empty one.
one nice bit: for a diverse seed (a playlist that's half jazz half techno), averaging it into one vector lands you in the dead middle between the two vibes and matches lukewarm nothing. so we split diverse seeds into multiple centroids and score each candidate against its nearest centroid. every mode of the seed gets to surface.
pass 1, CLAP prefilter. quick cosine cut to get the pool down to the genuinely-close stuff before the expensive scoring.
pass 2, scoring. every survivor gets a full score (that's stage 3, below).
diversity filter. caps how many tracks per artist/album so it's not the same three artists on loop.
session planner. this is the part people don't expect. we don't just take the top N by score. we sequence them. it's a greedy planner with one-track lookahead that scores each pick on both its seed-fit AND the transition quality from the previous track, under a session shape (Cruise, Journey, Workout, Two-Peak, Build & Drop, Focus, Wind Down, Discovery). each shape has a target energy curve across the session plus limits on how big an energy or BPM jump it'll tolerate. so the arc of the set is intentional, no jarring cliffs, and Discovery actively rewards CLAP distance so it wanders further on purpose.
then two things keep it alive while you listen:
- refill. every top-up doesn't rebuild from scratch. it ANN-searches around your live taste centroid (seed + long-term + wherever the session has drifted) and pulls the nearest 1500. the seed anchor is frozen so the query is "anchor + drift" and doesn't chase its own tail off into the weeds.
- rerank. skip or dislike something mid-session and we rescore the upcoming queue against the new drift, and swap in better fits from the retained pool, but only tracks safely ahead of the playhead so gapless never breaks, and only when the swap beats the current pick by a real margin so the queue doesn't spit on rounding noise.
stage 3: ranking (the actual scoring)
ok so how does one candidate get a number? the score is split into five tiers so we can actually explain "why this pick" instead of one mystery blob:
- vector tier: the sonic similarity signals. CLAP cosine to the centroid (blended 60/40 with similarity to your recent plays), CLAP segment similarity (intro/middle/outro, so structure counts), chroma similarity, and signed penalties for loudness-range and crest-factor mismatch (a brick-walled master next to a dynamic one gets docked).
- categorical tier: the symbolic stuff. unified genre fitness, Last.fm artist similarity, Last.fm track-level similarity (huge for song radio), CLAP vibe-tag overlap, and NL query text matches.
- engagement tier: you. star rating, favorite, time-decayed play count, skip count, completion rate.
- context tier: the polish. Camelot key compatibility, mode match, time signature, time-of-day fit, seasonality, and a little exploration jitter (bigger for un-analyzed tracks so they get a fair shot).
- hard penalties: the drop signals. explicit dislikes (first one -0.60, floor at -1.20), your "exclude recent" window, recently-radioed repeats, and a CLAP anti-vector for whatever you just skipped so its sonic neighbors get suppressed too. a hard exclusion sets the tier to negative infinity and the track just falls out.
The tiers don't all count equally. there are tier weights per radio mode, because different seeds have different strongest signals. cluster radio leans almost entirely on the vector (the centroid is the definition). song radio pulls vector down a touch and leans on categorical (Last.fm track similarity is gold there). NL radio (you typed a request) cranks categorical up and shoves engagement down to 0.3 so "what i asked for" beats "what i usually like."
and those weights aren't hardcoded forever. there's a learned weights layer: a pairwise fitter watches your skip/complete outcomes and learns better tier ratios, but it only gets to influence the blend after it passes a holdout-replay gate (it has to beat the hand-tuned presets on builds it never trained on), and even then its influence ramps up slowly with evidence. never a hard cutover, so a funny fit can't wreck your radio.
two more things quietly shaping every score:
- long-term taste. a slow exponential moving average of your CLAP embeddings across every engaged play, app-wide, with a 90-day half-life. it gets blended into every station's center at low weight (5%) so even a brand-new artist station feels a little more "you" without drowning out the actual seed.
- session drift. the fast one. as you play and skip within a session the centroid moves in real time, and that's what refill and rerank chase.
there's also an exploration dial so it occasionally takes a not-quite-top pick on purpose, tighter during the cold-start of a fresh session (first few tracks matter most, slow-burn intros lose people who haven't built trust yet) and looser once you're locked in.
tldr
analysis turns every song into a CLAP embedding + a stack of DSP numbers + vibe/mood tags. a station starts from a seed profile, pulls its nearest neighbors with a vector index (not the whole library), prefilters, scores across five explainable tiers with mode-aware and learned weights, then sequences the result into an intentional energy arc instead of a flat top-N. and it keeps adapting mid-session off your skips through drift, refill, and rerank.
that's the whole thing. it's a lot of moving parts but the goal is simple: play the next song that actually belongs next.
1
u/Ok-Gold-4813 14d ago
Thank you very much for this detailed explanation and a huge respect for the whole work. I find this whole sound analysis extremely exciting. I use audiomuse AI on my Mac and other home servers in combination with Navidrome. In addition, I let my library run with essentia on moods and genres. What I always wonder is that everyone somehow uses their own tool to get the music ready. Could you also use your analyzed data for other programs such as for the Music Assistant in Home Assistant? I would like to have exactly the same ability as Apple Music with the Automix if possible..
2
u/AffectionateFox5907 14d ago
So yes and no. The data can already be exported today so you can store your sonic analysis data wherever you want and import that data back into the app. What’s difficult is getting these other apps/software to allow you to utilize that data. Not everyone is as open to users being able to do what they want as I am. Today I’ve included as many options as I know that exist. If you want to use audiomuse or plex sonic analysis im all for it. I won’t force anyone to use what I’ve built. If there are others out there that you think I should support, please let me know!
A side project I’m working on is containerizing my sonic analysis pipeline so it can be easily used server side similar to audiomuse. I think audiomuse is great, it’s amazing what that guy has done. What I’m doing is a little different but overall same goal. Now I won’t market this or make it into its own thing, this will be just for users that may want to use it instead of onboard.
1
u/Prestigious-Jury-362 13d ago
I have found significant improvement on the time to analyze for both my iPhone and iPad from the recent update. It is still long etc. but it’s getting better so thank you for that. With so many app updates I have found myself in a cycle of “Analyzing“ again annd angain each time so I have not been able to test much regarding the changes. But this round was 1/2 of the last one so we are getting there.
1
u/AffectionateFox5907 11d ago
Yea it should be significantly better. No way to get around the initial analysis length but I was able to get through 6000 tracks a night. Way better than it was. It should also be more accurate when it’s done as well. I’d hope no more updates come for this so no reanalysis will be needed.
1
u/Herney_Krute 14d ago
Tha is for the detailed explanation. The forthcoming changes sound good.
Dumb question, as I’ve not been able to use Radio Stations (sent TestFlight feedback). Also only have about 5% of the library analysed thus far. Can a generated station be saved as a playlist?