How it works

• IntonTrainer is a software system designed to train learners in producing a variety of intonation patterns of speech.
• The system is based on comparing the melodic (tonal) portraits of a reference phrase and a phrase spoken by the learner and involves active learner-system interaction.
• Since parametric representation of intonation features of the speech signal faces fundamental difficulties, the paper intends to show how these difficulties can be overcome.
• The main algorithms used in the training system proposed for analyzing and comparing intonation features are considered.
• A set of reference sentences is given which represents the basic intonation patterns of English speech and their main varieties.
• The system’s interface is presented and the results of the system operation are illustrated.

Importance of intonation learning

• A current linguistic idea is that a foreign accent is more evident and stable in intonation than in segmental sounds.
• A foreign accent in intonation emerges mainly as a result of prosodic interference, an inevitable ‘by-product’ of bilingualism and, particularly, under the influence of the prosodic patterns of the learner’s native language on those of the target language.
• Considering the variety of functions of intonation in speech and its potential socio-cultural effects, deviations in this area can lead to serious semantic losses in communication.
• It is a well-known fact that it is incorrect intonation that is often the cause of the wrong impression a non-native language speaker might produce.
• This study is concerned with the progress achieved in providing an additional visual feedback as well as a quantitative assessment accuracy helping non-native learners eliminate intonation errors.

Five tasks we should solve

• Task 1. An adequate comparison of the pattern signal and a spoken one which is usually characterized by a non-linear time deformation and its beginning and end are not known beforehand.
  The solution of this problem has become possible thanks to the application of the modified method of a continuous dynamic time warping (CDTW) of two signals.
• Task 2. Automatic segmentation of the signal being analyzed into areas for which the notion of F₀ is relevant as far as the formation of the tonal contour of the phrase is concerned (the segments of vowels and most of the sonorants).
  This problem is being solved by means of a non-linear transfer of segment markers from the preliminarily marked pattern-phrase onto the phrase being uttered
• Task 3. Precise calculation of F₀ of the pattern speech signal and of that produced by the learner within a very wide voice range {30 – 1000 Hz}, for male and female voices pooled.
  The task is solved by using known modern methods of singling F₀ out of a speech signal.
• Task 4. Automatic interpolation of current values F₀ on the segments for which measuring F₀ is invalid, i.e. on most of the consonants.
  This task is solved by using well-known interpolation mathematical formulas determining the way of finding intermediate values.
• Task 5. An adequate calculation of a similarity measure between the pattern signal and the uttered one under the condition of their differences in duration and F₀ voice-ranges.
  This task is solved by using a representation of intonation curves in the form of unified melodic portraits (UMPs) described below. Calculation of the similarity measure of two UMPs is carried out through determining the vector distance between the curves.

UMP stylization model

• According to UMP model, a phrase is represented by one or more AUs (often referred to as Accent Group). Each AU can be composed of one or more words and consists of pre-nucleus (all phonemes preceding the main stressed vowel), nucleus (the main stressed vowel) and post-nucleus (all phonemes following the main stressed vowel).
• The model allows representing the intonation constructions of a given language as a set of melodic patterns in normalized space {Time – Frequency}.
• Time normalization is performed by bringing pre-nucleus, nucleus and post-nucleus elements of AU to standard time lengths.
• For fundamental frequency normalization F_0 min and F_0 max are determined within the ensemble of melodic contours produced by a certain speaker.

Graphical representation of UMP

Normalized space for UMP is presented as a rectangle with axes (T_N, F₀^N).
The interval [0 - ¹/₃] on the abscissa T_N is a pre-nucleus,
[¹/₃ - ²/₃] is a nucleus, and
[²/₃ - 1] is a post-nucleus.

The intervals on the ordinate F₀^N: [0 ‑¹/₃] - low level,
[¹/₃ - ²/₃] - mid level,
[²/₃ - 1] - high level.

Acoustic Database

• The developed prototype of the system is realized in 2 variants for implementation in multimedia course-books for advanced learners of English and Russian intonation.
• Application of this system makes it possible for the students not only to listen to phrases pronounced with standard intonation but also observe the model F₀(t) и A₀(t) curves on display.
• It is possible for the students to reproduce these phrases, compare their F₀(t) и A₀(t) curves with the original ones and obtain a numerical evaluation of their similarity.
• Used as models are male-and-female-spoken sample phrases from the Internet-available multimedia course-books.
• In the database used, there are several samples of common types of utterances for different intonation pattern as well as several samples of conversational speech and a piece of narrative prose.

The basic types of English tonal patterns used

Block diagram of the training system

Illustration of speech signals marking:

the phrase “It’s Sa+turday” pronounced by the teacher (above) and by a learner (below)

PreNucleus - NUCLEUS - PostNucleus

Illustration of F0 trajectory processing

for the teacher’s phrase “It’s Sa+turday”:
original (light curve line) and interpolated, smoothed and normalized (dark curve line)

PreNucleus - NUCLEUS - PostNucleus

Illustration of F0 curve comparison (correct pronunciation)

between the teacher’s (light curve line) and a student’s (dark curve line) phrase “It’s Saturday” in real time space (above) and in UMP-space (belowe)

Illustration of F0 curve comparison (incorrect pronunciation)

between the teacher’s (light curve line) and a student’s (dark curve line) phrase “It’s Saturday” in real time space (above) and in UMP-space (belowe)

User Interface - Starting page

Selection box - “TONE TYPES”

“TONE TYPES” – FallRise

(example)

Selection box - “TONES COMPARISON”

“TONES COMPARISON” – Rise & Fall

(example)

Selection box – “TONE USAGE”

“TONE USAGE” – Imperative

(example)

Selection box – “DIALOGUES”

“DIALOGUES” - Fragment

(example)

Selection box – “LITERATURE”

“LITERATURE” – Fragment

(example)

CONCLUSIONS

• Software realization of the system named "IntonTrainer" is written in C++ using the Qt framework.
• It can be compiled for Windows (XP through 10) and Linux (Debian).
• At present, using the IntonTrainer system, experiments are conducted to learn by students the intonation of Russian and English.
• Preliminary results indicate a significant effectiveness of its use.