Voprosy JazykoznanijaVoprosy JazykoznanijaВопросы языкознания0373-658X3034-5243The Russian Academy of Sciences65062610.31857/0373-658X.2024.1.128-155SurveysОбзорыReview ArticlePhoneme sequence-to-speech conversion in dynamic phonological models: A surveyПреобразование «цепочка фонем» → «речь» в динамических моделях: ОбзорMakarovIlya S.МакаровИлья Сергеевич
Russian Federation
im@biometriclabs.ruBiometricLabs, LLCООО «БиометрикЛабс»14032024112815531012025Copyright ©; 2024, Russian Academy of SciencesCopyright ©; 2024, Российская академия наук2024Russian Academy of SciencesРоссийская академия наукhttps://journals.eco-vector.com/0373-658X/article/view/650626

This survey is devoted to dynamic models that model how a discrete phoneme sequence becomes converted to the corresponding continuous flow of articulations. The key concepts of modern dynamic models are discussed: articulatory model, articulatory parameters, goals and gestures, pronunciation effort economy principle, etc. The results of research conducted by specialists from Haskins Laboratories (articulatory phonology, task dynamic model), as well as by Japanese scientists (mostly from Waseda University) are presented. The survey is illustrated by both model examples and real articulatory X-ray microbeam measurements.

Настоящий обзор посвящен динамическим моделям преобразования дискретной цепочки фонем в непрерывный речевой поток. Обсуждаются ключевые для современных динамических моделей понятия: артикуляционная модель, управляющие параметры, целевые артикуляции, артикуляционные жесты, принцип экономии произносительных усилий и проч. Излагаются результаты исследований специалистов Хаскинских лабораторий (артикуляционная фонология, task-dynamic-модель), а также японских исследователей (преимущественно Waseda University). Обзор иллюстрируется как модельными примерами, так и реальными артикуляционными измерениями на базе микролучевой рентгеноскопической установки.

articulatory phoneticsautomatic speech processingcomputational linguisticsphoneticsавтоматическая обработка речиартикуляторная фонетикакомпьютерная лингвистикафонетикаАрнольд 1999 — Арнольд В. И. Лекции об уравнениях с частными производными. М.: ФАЗИС, 1999. [Arnol’d V. I. Lektsii ob uravneniyakh s chastnymi proizvodnymi [Lectures on partial differential equations]. Moscow: FAZIS, 1999.]Баден и др. 2005 — Баден П., Макаров И. С., Сорокин В. Н. Алгоритм вычисления площадей по-перечных сечений речевого тракта. Акустический журнал, 2005, 51(1): 52–58. [Badin P., Makarov I. S., Sorokin V. N. An algorithm for calculating the cross-section areas of the vocal tract. Akustich-eskii zhurnal, 2005, 51(1): 52–58.]Князев 1999 — Князев С. В. О прогрессивной ассимиляции в современном русском языке. Вестник Московского государственного университета. Сер. 9. Филология, 1999, 4: 18–33. [Knyazev S. V. On progressive assimilation in Modern Russian. Vestnik Moskovskogo gosudarstvennogo universiteta. Ser. 9. Filologiya, 1999, 4: 18–33.]Князев 2001 — Князев С. В. Еще раз о соотношении фонетики и фонологии. Вестник Московского государственного университета. Сер. 9. Филология, 2001, 5: 101–112. [Knyazev S. V. Once again on the relationship between phonetics and phonology. Vestnik Moskovskogo gosudarstvennogo uni-versiteta. Ser. 9. Filologiya, 2001, 5: 101–112.]Князев 2004 — Князев С. В. Об иерархии фонологических правил в русском языке (несколько но-вых соображений по поводу язв А. А. Реформатского). Семиотика, лингвистика, поэтика: К столетию со дня рождения А. А. Реформатского. Виноградов В. А. (ред.). М.: Языки славян-ской культуры, 2004, 151–166. [Knyazev S. V. On the hierarchy of phonological rules in Russian (several new thoughts on yazv of A. A. Reformatsky). Semiotika, lingvistika, poetika: K stoletiyu so dnya rozhdeniya A. A. Reformatskogo. Vinogradov V. A. (ed.). Moscow: Yazyki slavyanskoi kul’tury, 2004, 151–166.]Кодзасов, Кривнова 2001 — Кодзасов С. В., Кривнова О. Ф. Общая фонетика: Учебник. М.: РГГУ, 2001. [Kodzasov S. V., Krivnova O. F. Obshchaya fonetika: Uchebnik [General phonetics: A text-book]. Moscow: Russian State Univ. for the Humanities, 2001.]Леонов и др. 2003 — Леонов А. С., Макаров И. С., Сорокин В. Н., Цыплихин А. И. Артикулятор-ный ресинтез гласных. Информационные процессы, 2003, 3(2): 73–82. [Leonov A. S., Makarov I. S., Sorokin V. N., Tsyplikhin A. I. Articulatory resynthesis of vowels. Informatsionnye protsessy, 2003, 3(2): 73–82.]Леонов и др. 2004 — Леонов А. С., Макаров И. С., Сорокин В. Н., Цыплихин А. И. Артикулятор-ный ресинтез фрикативных. Информационные процессы, 2004, 4(2): 141–159. [Leonov A. S., Makarov I. S., Sorokin V. N., Tsyplikhin A. I. Articulatory resynthesis of fricatives. Informatsionnye protsessy, 2004, 4(2): 141–159.]Леонов и др. 2005 — Леонов А. С., Макаров И. С., Сорокин В. Н., Цыплихин А. И. Кодовая книга для речевых обратных задач. Информационные процессы, 2005, 5(2): 101–119. [Leonov A. S., Makarov I. S., Sorokin V. N., Tsyplikhin A. I A codebook for speech inverse problems. Infor-matsionnye protsessy, 2005, 5(2): 101–119.]Макаров 2005 — Макаров И. С. Построение и исследование артикуляторных кодовых книг для ре-шения речевых обратных задач. Дис. … канд. тех. н. М.: ИППИ РАН, 2005. [Makarov I. S. Postroenie i issledovanie artikulyatornykh kodovykh knig dlya resheniya rechevykh obratnykh zadach [Construction and research of articulatory codebooks for solution of speech inverse problems]. Can-didate diss. Moscow: Kharkevich Institute for Information Transmission Problems, 2005.]Макаров 2009 — Макаров И. С. Аппроксимация речевого тракта коническими рупорами. Акусти-ческий журнал, 2009, 55(2): 256–265. [Makarov I. S. Approximating the vocal tract by conical horns. Akusticheskii zhurnal, 2009, 55(2): 261–269.]Макаров 2011 — Макаров И. С. Алгоритм быстрого вычисления передаточной функции для неод-нородной акустической трубы. Акустический журнал, 2011, 57(5): 695–708. [Makarov I. S. A fast transfer function algorithm for nonuniform acoustic tubes. Akusticheskii zhurnal, 2011, 57(5): 695–708.]Макаров 2023 — Макаров И. С. Двухмерная математическая модель языка. Рук., 2023. [Makarov I. S. Dvukhmernaya matematicheskaya model’ yazyka [A 2D-tongue mathematical model]. Ms., 2023.]Макаров, Сорокин 2004 — Макаров И. С., Сорокин В. Н. Резонансы разветвленного речевого тракта с податливыми стенками. Акустический журнал, 2004, 50 (3): 389–396. [Makarov I. S., Sorokin V. N. Resonances of the branched vocal tract with compliant walls. Akusticheskii zhurnal, 2004, 50(3): 323–330.]Сорокин 1992 — Сорокин В. Н. Синтез речи. М.: Наука, 1992. [Sorokin V. N. Sintez rechi [Speech Synthesis]. Moscow: Nauka, 1992.]Сорокин 2012 — Сорокин В. Н. Речевые процессы. М.: Народное образование, 2012. [Sorokin V. N. Rechevye protsessy [Speech Processes]. Moscow: Narodnoe obrazovanie, 2012.]Щерба 1974 — Щерба Л. В. О разных стилях произношения и об идеальном фонетическом составе слов. Языковая система и речевая деятельность. Зиндер Л. Р., Матусевич М. И. (ред.). Л.: Наука, 1974. [Shcherba L. V. On different pronunciation styles and on the ideal phonetic structure of words. Yazykovaya sistema i rechevaya deyatel’nost’. Zinder L. R., Matusevich M. I. (eds.). Lenin-grad: Nauka, 1974.]Browman 1992 — Browman C. Articulatory Phonology: An overview. Phonetica, 1992, 49: 155–180.Browman, Goldstein 1989 — Browman C., Goldstein L. Articulatory gestures as phonological units. Phonology, 1989, 6: 201–251.Browman, Goldstein 1990 — Browman C., Goldstein L. Tiers in articulatory phonology, with some implications for casual speech. Papers in laboratory phonology I: Between the grammar and the phys-ics of speech. Kingston J., Beckman M. E. (eds.). Cambridge: Cambridge Univ. Press, 1990, 341–376.Byrd, Krivokapic 2021 — Byrd D., Krivokapic E. Cracking prosody in articulatory phonology. Annual Review in Linguistics, 2021, 7: 31–53.Chen et al. 2013 — Chen X., Dang J., Yan H., Fang Q., Kröger B. A neural understanding of speech motor learning. Proc. of the 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (Kaohsiung, Oct. 29–Nov. 1, 2013). New York: Institute of Electrical and Electronics Engineers, 2013, 1–14.Dang, Honda 2004 — Dang J., Honda K. Construction and control of a physiological articulatory model. Journal of the Acoustical Society of America, 2004, 115(2): 853–870.Dusan 2000 — Dusan S. Statistical estimation of articulatory trajectories from the speech Signal using dynamic and phonological constraints. Ph.D. diss. Waterloo: Univ. of Waterloo, 2000.Fant 2001 — Fant G. Swedish vowels and a new three-parameter model. Quarterly Progress and Status Report, 2001, 42(1): 43–49.Gafos et al. 2020 — Gafos A., Roeser J., Sotiropoulou S., Hoole P., Zeroual C. Structure in mind, struc-ture in vocal tract. Natural Language Linguistic Theory, 2020, 38: 43–75.Goldstein, Fowler 2003 — Goldstein L., Fowler C. Articulatory Phonology: A phonology for public lan-guage use. Phonetics and phonology in language Comprehension and production. Schiller N. O., Mey-er A. S. (eds.). Berlin: Mouton de Gruyter, 2003, 159–207.Gomez et al. 2020 — Gomez A., Stone M. L., Woo J., Xing F., Prince J. L. Analysis of fiber strain in the human tongue during speech. Computer Methods in Biomechanics and Biomedical Engineering, 2020: 23(8), 312–322.Hanson, Stevens 2002 — Hanson H., Stevens K. A quasiarticulatory approach to controlling acoustic source parameters in a Klatt-type formant synthesizer using HLsyn. Journal of the Acoustical Society of America, 2002, 112: 1158–1182.Hashi et al. 1998 — Hashi M., Westbury J. R., Honda K. Vowel posture normalization. Journal of the Acoustical Society of America, 1998, 104: 2426–2437.Honorof 2004 — Honorof D. Articulatory events are given in advance. Hard-Science Linguistics. Yngve V. H., Wasik Z. (eds.). London: Continuum, 2004, 67–86.Huang et al. 2001 — Huang X., Acero A., Hon H.-W. Spoken Language Processing. New Jersey: Pren-tice Hall, 2001.Iskarous et al. 2003 — Iskarous K., Goldstein L., Whalen D., Tiede M., Rubin P. CASY: The Haskins Configurable Articulatory Synthesizer. Proc. of the 15th International Congress of Phonetic Sciences (Barcelona, Aug. 3–9, 2003). Solé M. J., Recasens D., Romero J. (eds.). Barcelona: FUTURGRAFIC, 2003, 185–188.Ito et al. 2004 — Ito T., Gomi H., Honda M. Dynamical simulation of speech cooperative articulation by muscle linkages. Biological Cybernetics, 2004, 91: 275–282.Kaburagi, Honda 1994 — Kaburagi T., Honda M. Determination of sagittal tongue shape from the posi-tions of points on the tongue surface. Journal of the Acoustical Society of America, 1994, 96(3): 1356–1366.Kaburagi, Honda 1996 — Kaburagi T., Honda M. A model of articulator trajectory formation based on the motor tasks of vocal-tract shapes. Journal of the Acoustical Society of America, 1996, 99(5): 3154–3170.Kaburagi, Honda 2001 — Kaburagi T., Honda M. Dynamic articulatory model based on multidimen-sional invariant-feature task representation. Journal of the Acoustical Society of America, 2001, 110(1): 441–451.Kaburagi, Kim 2007 — Kaburagi T., Kim J. Generation of the vocal tract spectrum from the underlying articulatory mechanism. Journal of the Acoustical Society of America, 2007, 121(1): 456–468.Lu, Dang 2010 — Lu X., Dang J. Vowel Production Manifold: Intrinsic Factor Analysis of Vowel Artic-ulation. IEEE Transactions on Audio, Speech, and Language Processing, 2010, 18(5): 1053–1062.Mascheretti et al. 2021 — Mascheretti S., Perdue M., Feng B., Andreola Ch., Dionne G., Jasińska K., Pugh K., Grigorenko E., Landi N. From BDNF to reading: Neural activation and phonological pro-cessing as multiple mediators. Behavioral Brain Research, 2021, 396: 112859.McGowan 1994 — McGowan R. Recovering articulatory movement from formant frequency trajectories using task dynamics and a genetic algorithm: Preliminary model tests. Speech Communication, 1994, 14: 19–48.McGowan, Lee 1996 — McGowan R., Lee M. Task dynamic and articulatory recovery of lip and velar approximations under model mismatch conditions. Journal of the Acoustical Society of America, 1996, 99(1): 595–608.McGowan, Saltzman 1995 — McGowan R., Saltzman E. Incorporating aerodynamic and laryngeal components into task dynamics. Journal of Phonetics, 1995, 23: 255–269.Nam et al. 2013 — Nam H., Mooshammer Ch., Iskarous K., Whalen D. Hearing tongue loops: Perceptu-al sensitivity to acoustic signatures of articulatory dynamics. Journal of the Acoustical Society of America, 2013, 134(5): 3808–3817.Narayanan, Alwan 2000 — Narayanan S., Alwan A. Noise source models for fricative consonants. IEEE Transactions on Speech and Audio Processing, 2000, 8(3): 328–344.Ohashi, Ostry 2021 — Ohashi H., Ostry D. Neural Development of Speech Sensorimotor Learning. The Journal of Neuroscience, 2021, 41(18): 4023–4035.Okadome, Honda 2001 — Okadome T., Honda M. Generation of articulatory movements by using a kinematic triphone model. Journal of the Acoustical Society of America, 2001, 110(1): 453–462.Rubertus, Noiray 2018 — Rubertus E., Noiray A. On the development of gestural organization: A cross-sectional study of vowel-to-vowel anticipatory coarticulation. PLOS ONE, 2018, 13(9): 1–21.Rubin et al. 1981 — Rubin P., Baer T., Mermelstein P. An articulatory synthesizer for perceptual re-search. Journal of the Acoustical Society of America, 1981, 70, 321–328.Rubin et al. 1996 — Rubin P., Saltzman E., Goldstein L., McGowan R., Tiede M., Browman C. CASY and extensions to the task-dynamic model. 1st ESCA Tutorial and Research Workshop on Speech Prodution Modeling — 4th Speech Production Seminar (Autrans, May 20–24, 1996). 1996, 125–128.Saltzman, Byrd 2000 — Saltzman E., Byrd D. Task-dynamics of gestural timing: Phase windows and multifrequency rhythms. Human Movement Science, 2000, 19: 499–526.Saltzman, Munhall 1989 — Saltzman E., Munhall K. A dynamical approach to gestural patterning in speech production. Ecological Psychology, 1989, 1(4): 333–382.Schroeter, Sondhi 1991 — Schroeter J., Sondhi M. M. Speech coding based on physiological models of speech production. Advances in speech signal processing. Furui S., Sondhi M. M. (eds.). New York: Marcel Dekker, 1991, 231–266.Serrurier et al. 2017 — Serrurier A., Badin P., Boe L.-J., Lamalle L., Neuschaefer-Rube C. Inter-speaker variability: Speaker normalisation and quantitative estimation of articulatory invariants in speech production for French. Proc. of 18th Annual Conf. of the International Speech Communication Associa-tion (Stockholm, Aug. 20–24, 2017). Red Hook (NY): Curran Associates, 2017: 2272–2276.Serrurier et al. 2023 — Serrurier A., Neuschaefer-Rube Ch. Morphological and acoustic modeling of the vocal tract. Journal of the Acoustical Society of America, 2023, 153: 1867–1886.Smith et al. 1993 — Smith C., Browman C., McGowan R., Kay B. Extracting dynamic parameters from speech movement data. Journal of the Acoustical Society of America, 1993, 93(3): 1580–1586.Sorokin et al. 2005 — Sorokin V. N., Leonov A. S., Makarov I. S., Tsyplikhin A. I. Speech inversion and resynthesis. Proc. of the 6th Interspeech 2005 and 9th European Conf. on Speech Communication and Technology (Lisboa, Sept. 4–8, 2005). Red Hook (NY): Curran Associates, 2005, 3209–3212.Story, Bunton 2011 — Story B., Bunton K. Decomposition of vowel and consonant contributions to the time‐varying vocal tract shape. Journal of the Acoustical Society of America, 2011, 129, 2456.Story, Bunton 2019 — Story B., Bunton K. A model of speech production based on the acoustic relativ-ity of the vocal tract. Journal of the Acoustical Society of America, 2019, 146: 2522–2528.Studdart-Kennedy, Goldstein 2003 — Studdart-Kennedy M., Goldstein L. Launching Language: The Gestural Origin of Discrete Infinity. Language Evolution. Christiansen M., Kirby S. (eds.). Oxford: Oxford Univ. Press, 2003, 235–254.Wang et al. 2014 — Wang W., Arora R., Livescu K. Reconstruction of articulatory measurements with smoothed low-rank matrix completion. Proc. of the 2014 IEEE Spoken Language Technology Work-shop (South Lake Tahoe, Dec. 7–8, 2014). 54–59.Wei, Dang 2008 — Wei J., Dang J. Vocal tract normalization in articulatory space using thin‐plate spline method. Journal of the Acoustical Society of America, 2008, 123, 3885.Westbury 1994 — Westbury J. R. X-ray microbeam speech production database. User’s handbook. Ver-sion 1. Madison: Univ. of Wisconsin, 1994.Yan et al. 2014 — Yan H., Dang J., Cao M., Kröger B. A new framework of neurocomputational model for speech production. Proc. of the 9th International Symposium on Chinese Spoken Language Pro-cessing (Singapore, Sept. 12–14, 2014). 2014, 294–298.Zhou et al. 2008 — Zhou X., Espy-Wilson C., Boyce S., Tiede M., Holland Ch., Choe A. A magnetic resonance imaging-based articulatory and acoustic study of “retroflex” and “bunched” American English ∕r∕. Journal of the Acoustical Society of America, 2008, 123(6): 4466–4481.