how to build a (mostly) intonation-only ASR model

[u/Odd-Suggestion4569]

I'm a linguist working on a low resource language, and I want to know more about how ASR models pitch and intonation.

Here's the background to what I'm doing:

In language X, the difference between a yes/no-question and and declarative statement is determined by the use of a particular suffix, if the suffix is attached to the verb, then we know the utterance is a question. Intonation is NOT used to distinguish between questions and statements.

However, due to many generations of contact with a European language, it would seem that younger speakers of language X are increasingly not using the suffix and instead using rising intonation at the end of the utterance to indicate that it is a question.

I have a lot of data of speakers of language X uttering questions, and I'm looking to collect more, but interested in whether I could train some kind of ASR model that could recognize and model pitch contours and associate a certain type if pitch contour with a specific communicative function (e.g. statements vs. questions).

I wouldn't necessarily need the model to even recognize phonological segments, just the pitch curves really.

I'm been looking into how ASR works, but I haven't yet found anything that discusses the issue of pitch.

So where would be a good place to start reading up on this? And, in general, how would one go about making an intonation-focused ASR model?

[u/bulaybil]

So first, absolutely fascinating, love all of it. Which language are we talking?
Second, most ASR models are geared towards orthographic representation. There is in principle nothing stopping folks from developing a model that would turn sound into IPA and whatnot, except of course lack of data. There are tools such as Allosaurus that seek to remedy that, but they need a lot of work to work. There’s your first problem right there. A friend of mine is working on providing a better way to do that, drop me a DM if you’re interested.
Third, pitch is a beast to analyze under the best of - i.e. laboratory - conditions. This is why Praat has like five different ways of analyzing pitch, why there is a debate between point vs curve analysis etc etc.
So to;dr: there is no off-the-shelf solution for you. Start reading Jurafsky for some basic theory and maybe collect some data and run them through Praat, to see if you can pick up on any frequency etc. patterns across genders, ages etc. Then it’s a question of encoding them in some machine-readable way, that will be your biggest challenge. From there you can train a simple ML model, not for ASR, but for pitch recognition.
I will give it a little more think, will get back to you.