Configuring for ToBI annotations
Jonathan Harrington
WiSe 2021
- 1 Preliminaries
- 2 Overall aim
- 3 Forced alignment
- 4 Configuring a database for TOBI (tones and break indices) annotation
1 Preliminaries
If you have not already done so as a consequence of completing earlier modules, follow the setup instructions here
library(tidyverse)## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ──## ✓ ggplot2 3.3.5 ✓ purrr 0.3.4
## ✓ tibble 3.1.4 ✓ dplyr 1.0.7
## ✓ tidyr 1.1.3 ✓ stringr 1.4.0
## ✓ readr 2.0.1 ✓ forcats 0.5.1## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()library(emuR)##
## Attaching package: 'emuR'## The following object is masked from 'package:base':
##
## normlibrary(wrassp)
sourceDir = "./testsample"
targetDir = "./emu_databases"2 Overall aim
Starting out with just a text collection the aim is to create Emu database that is useful for tones-and-break-indices annotation. The more specific aim is to configure the database to obtain a configuration as in Fig. 2.1.
Figure 2.1: A TOBI annotation
The starting point will be the text collection of Albanian data analysed in the previous module.
3 Forced alignment
The first task is to get apply forced alignment to these Albanian data. The commands will be presented in this section without detailed comment given that these were discussed in the previous module
The resulting Emu database will be stored as alb2_DB (to distinguish it from the one created in the last module)
# the path of the text collection of Albanian utterances
path.albanian = file.path(sourceDir, "albanian")
# convert it to an Emu database called alb2_DB
convert_txtCollection(dbName = "alb2",
sourceDir = path.albanian,targetDir = targetDir)## INFO: Parsing plain text collection containing 4 file pair(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Copying 4 media files to EMU database...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%# load the emu database
alb2_DB = load_emuDB(file.path(targetDir, "alb2_emuDB"))## INFO: Loading EMU database from ./emu_databases/alb2_emuDB... (4 bundles found)
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%# run forced alignment
runBASwebservice_all(alb2_DB,
transcriptionAttributeDefinitionName = "transcription",
language = "sqi-AL", runMINNI = F)## INFO: Preparing temporary database. This may take a while...
## INFO: Checking if cache needs update for 1 sessions and 4 bundles ...
## INFO: Performing precheck and calculating checksums (== MD5 sums) for _annot.json files ...
## INFO: Nothing to update!
## INFO: Sending ping to webservices provider.
## INFO: Running G2P tokenizer on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running G2P on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running MAUS on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running Pho2Syl (canonical) on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Sending ping to webservices provider.
## INFO: Running Pho2Syl (segmental) on emuDB containing 4 bundle(s)...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%
## INFO: Autobuilding syllable -> segment links from time information
## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%4 Configuring a database for TOBI (tones and break indices) annotation
The database currently has an ITEM tier ORT than dominates an ITEM tier MAS containing syllabifications than dominates the SEGMENT tier MAU containing the phonetic segmenation as shown by either serve(alb2_DB, useViewer=F) or by summary(alb2_DB).
The task is to add a tier Tone for imarking ntonational events and to add a new SEGMENT tier, ORT2 showing word segmentations and their times. The tonal events of the Tone tier and annotations of the ORT2 tier are to be linked automatically, just as in Fig. 2.1.
Here are the steps needed:
- Create the
SEGMENTtierORT2containing word annotations linked to times. - Create a
Tone tierand link it toORT2 - Calculate and display f0.
- Annotate the
Tonetier and link the annotations automatically to word annotations.
4.1 Creating and displaying the SEGMENT tier ORT2 with word annotations.
This requires four steps:
- Make a
SEGMENTtier calledORT2 - Make a segment list of all words in the database using
query(). - Add the word annotations to
ORT2. - Display the annotations of
ORT2time-aligned to the signals
4.1.1 Make a SEGMENT tier called ORT2
As discussed in an earlier module, this can be done with the add_levelDefinition() function, thus:
add_levelDefinition(alb2_DB, "ORT2", "SEGMENT")## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%# list the tiers (annotation levels):
list_levelDefinitions(alb2_DB)## name type nrOfAttrDefs attrDefNames
## 1 bundle ITEM 2 bundle; transcription;
## 2 ORT ITEM 3 ORT; KAN; KAS;
## 3 MAU SEGMENT 1 MAU;
## 4 MAS ITEM 1 MAS;
## 5 ORT2 SEGMENT 1 ORT2;4.1.2 Make a segment list of all words in the database
This can be done with the query() function.
# find all words in the database and
# get their start and end times.
text.s = query(alb2_DB, "[ORT =~ .*]")
text.s## # A tibble: 17 × 16
## labels start end db_uuid session bundle start_item_id end_item_id level
## <chr> <dbl> <dbl> <chr> <chr> <chr> <int> <int> <chr>
## 1 lena 1050. 1670. 72827a40… 0000 0001B… 2 2 ORT
## 2 leu 1670. 2100. 72827a40… 0000 0001B… 3 3 ORT
## 3 nje 2100. 2290. 72827a40… 0000 0001B… 4 4 ORT
## 4 mur 2290. 2770. 72827a40… 0000 0001B… 5 5 ORT
## 5 malena 1300. 2190. 72827a40… 0000 0001B… 2 2 ORT
## 6 leu 2190. 2670. 72827a40… 0000 0001B… 3 3 ORT
## 7 murin 2670. 3420. 72827a40… 0000 0001B… 4 4 ORT
## 8 marilena 1090. 2110. 72827a40… 0000 0001B… 2 2 ORT
## 9 leu 2110. 2500. 72827a40… 0000 0001B… 3 3 ORT
## 10 njomezen 2500. 3420. 72827a40… 0000 0001B… 4 4 ORT
## 11 lena 900. 1370. 72827a40… 0000 0001C… 2 2 ORT
## 12 leu 1370. 1620. 72827a40… 0000 0001C… 3 3 ORT
## 13 njomezen 1620. 2400. 72827a40… 0000 0001C… 4 4 ORT
## 14 jo 2550. 2840. 72827a40… 0000 0001C… 5 5 ORT
## 15 lena 2840. 3280. 72827a40… 0000 0001C… 6 6 ORT
## 16 leu 3280. 3550. 72827a40… 0000 0001C… 7 7 ORT
## 17 murin 3550. 4290. 72827a40… 0000 0001C… 8 8 ORT
## # … with 7 more variables: attribute <chr>, start_item_seq_idx <int>,
## # end_item_seq_idx <int>, type <chr>, sample_start <int>, sample_end <int>,
## # sample_rate <int>The attribute and name of the tier from which this segment list was derived is ORT as shown by this:
text.s$attribute## [1] "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT"
## [13] "ORT" "ORT" "ORT" "ORT" "ORT"text.s$level## [1] "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT" "ORT"
## [13] "ORT" "ORT" "ORT" "ORT" "ORT"These all need to be changed to the name of the new SEGMENT tier ORT2. This can be done as follows:
text.s$attribute = paste(text.s$attribute, "2", sep="")
text.s$level = text.s$attribute
text.s$attribute## [1] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"
## [11] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"text.s$level## [1] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"
## [11] "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2" "ORT2"4.1.3 add the annotations to ORT2
The next step is to add the information from text.s to the ORT2 tier. This is done with the function create_itemsInLevel() thus:
create_itemsInLevel(alb2_DB,
itemsToCreate = text.s)## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%So it should now be possible to query the word annotations directly from ORT2:
query(alb2_DB, "[ORT2 =~ .*]")## # A tibble: 17 × 16
## labels start end db_uuid session bundle start_item_id end_item_id level
## <chr> <dbl> <dbl> <chr> <chr> <chr> <int> <int> <chr>
## 1 lena 1050. 1670. 72827a40… 0000 0001B… 26 26 ORT2
## 2 leu 1670. 2100. 72827a40… 0000 0001B… 27 27 ORT2
## 3 nje 2100. 2290. 72827a40… 0000 0001B… 28 28 ORT2
## 4 mur 2290. 3665. 72827a40… 0000 0001B… 29 29 ORT2
## 5 malena 1300. 2190. 72827a40… 0000 0001B… 28 28 ORT2
## 6 leu 2190. 2670. 72827a40… 0000 0001B… 29 29 ORT2
## 7 murin 2670. 3665. 72827a40… 0000 0001B… 30 30 ORT2
## 8 marilena 1090. 2110. 72827a40… 0000 0001B… 34 34 ORT2
## 9 leu 2110. 2500. 72827a40… 0000 0001B… 35 35 ORT2
## 10 njomezen 2500. 3665. 72827a40… 0000 0001B… 36 36 ORT2
## 11 lena 900. 1370. 72827a40… 0000 0001C… 54 54 ORT2
## 12 leu 1370. 1620. 72827a40… 0000 0001C… 55 55 ORT2
## 13 njomezen 1620. 2550. 72827a40… 0000 0001C… 56 56 ORT2
## 14 jo 2550. 2840. 72827a40… 0000 0001C… 57 57 ORT2
## 15 lena 2840. 3280. 72827a40… 0000 0001C… 58 58 ORT2
## 16 leu 3280. 3550. 72827a40… 0000 0001C… 59 59 ORT2
## 17 murin 3550. 3665. 72827a40… 0000 0001C… 60 60 ORT2
## # … with 7 more variables: attribute <chr>, start_item_seq_idx <int>,
## # end_item_seq_idx <int>, type <chr>, sample_start <int>, sample_end <int>,
## # sample_rate <int>4.1.4 Display the annotations of ORT2 time-aligned to the signals
Entering serve(alb2_DB, useViewer = F) shows that the SEGMENT tier MAU is displayed underneath the signals. As also discussed in the same earlier module, the reason for this is because of the setting in the get/set_levelCanvasesOrder() functions. Thus this confirms that the current display is to MAU
get_levelCanvasesOrder(alb2_DB, perspectiveName = "default")## [1] "MAU"But the aim is to display only the newly created ORT2 tier. This can be done as follows:
set_levelCanvasesOrder(alb2_DB,
perspectiveName = "default",
order = "ORT2")If you view the database now, it will show only the segment tier ORT2 underneath the signals:
serve(alb2_DB, useViewer=F)4.2 Create a Tone tier and link it to ORT2
4.2.1 Creating the new Tone tier
This can be done using the add_levelDefinition() function used earlier in creating ORT2. In this case, Tone should be an EVENT tier for marking intonational targets as single points in time. The command is:
add_levelDefinition(alb2_DB,
name = "Tone",
type = "EVENT")## INFO: Rewriting 4 _annot.json files to file system...
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%4.2.2 Linking Tone and ORT2
The tiers Tone and ORT2 are currently unlinked as the following verifies:
list_linkDefinitions(alb2_DB)## type superlevelName sublevelName
## 1 ONE_TO_MANY bundle ORT
## 2 ONE_TO_MANY ORT MAS
## 3 ONE_TO_MANY MAS MAUBut they need to be linked in order that their annotations can be queried relatively to each other. On the assumption that a word can be associated with one or more tones, but that a given tone can only be associated with one word, then the association between the tiers is ONE-TO-MANY. The tiers should therefore be linked with the add_linkDefinitions() function in the following way:
add_linkDefinition(alb2_DB, "ONE_TO_MANY", "ORT2", "Tone")The same command as earlier now verifies that these tiers are linked (see the last line of the following output):
list_linkDefinitions(alb2_DB)## type superlevelName sublevelName
## 1 ONE_TO_MANY bundle ORT
## 2 ONE_TO_MANY ORT MAS
## 3 ONE_TO_MANY MAS MAU
## 4 ONE_TO_MANY ORT2 Tone4.2.3 Displaying the Tone tier.
As before, this is done with the function set_levelCanvasesOrder()
set_levelCanvasesOrder(alb2_DB,
perspectiveName="default",
order = c("Tone", "ORT2"))You can verify the relationship that the Tone and ORT2 tiers are being displayed underneath the signals with:
serve(alb2_DB, useViewer=F)4.3 Calculating and displaying f0
There are three steps here:
- Calculate f0
- Add the f0 information to the database
- Configure the database to display the f0 track.
Most of these steps were covered in an earlier module. Therefore, only a brief summary is given as follows.
4.3.1 Calculating f0
There are four parts to this:
- Identifying the
.wavfiles. - Calculating pitch data.
- Adding the pitch data to the database.
- Displaying the pitch data.
4.3.1.1 Identifying the .wav files
Identify the .wav files for which fundamental frequency data is to be calculated. They are here:
alb_wav_paths = list.files(path.albanian,
pattern = ".*wav$", recursive = T, full.names = T)
alb_wav_paths## [1] "./testsample/albanian/0001BF_1syll_1.wav"
## [2] "./testsample/albanian/0001BF_2syll_1.wav"
## [3] "./testsample/albanian/0001BF_3syll_1.wav"
## [4] "./testsample/albanian/0001CF_ord1_4.wav"4.3.1.2 Calculating pitch data
Calculate pitch for each .wav file and store the output:
mhsF0(alb_wav_paths, outputDirectory = path.albanian)##
## INFO: applying mhspitch to 4 files
##
|
| | 0%
|
|================== | 25%
|
|=================================== | 50%
|
|==================================================== | 75%
|
|======================================================================| 100%4.3.1.3 Adding the pitch data to the database
Add these f0 files to the Emu database with the add_files() function
add_files(alb2_DB, dir = path.albanian,
fileExtension = "pit", targetSessionName = "0000")4.3.1.4 Diplaying the pitch data
This requires two steps:
- defining the pitch track.
- displaying the pitch data.
4.3.1.4.1 Defining the pitch track
Use the add_ssffTrackDefinition() function for this purpose. See also
this earlier module for further details.
add_ssffTrackDefinition(alb2_DB,
name = "pitch",
columnName = "pitch",
fileExtension = "pit")4.3.1.4.2 Displaying the pitch data
As
explained in an earlier module, this is done with the function set_signalCanvasesOrder(). Currently, the waveform and spectrogram are being displayed, as shown by:
get_signalCanvasesOrder(alb2_DB, perspectiveName="default")## [1] "OSCI" "SPEC"To set things up so that the pitch data is displayed underneath the spectrogram and without displaying the spectrogram:
set_signalCanvasesOrder(alb2_DB, perspectiveName = "default",
order = c("SPEC", "pitch"))Overlaying the pitch on the spectrogram is a bit more intricate. For this, you will have to use a text editor (outside of R) to edit alb2_DBconfig.json that is located here.
file.path(targetDir, "alb2_emuDB", "alb_DBconfig.json")## [1] "./emu_databases/alb2_emuDB/alb_DBconfig.json"Then proceed as follows:
- Optionally make a backup copy of
alb2_DBconfig.jsonin case anything goes wrong. - Open
alb2_DBconfig.jsonwith a plain text editor. - Search for
"assign" - Carefully replace
"assign": [],
with
"assign": [{ "signalCanvasName": "SPEC", "ssffTrackName": "pitch" }],
- Save
alb2_DBconfig.json - Look at the database again
serve(alb2_DB, useViewer = F)- Finally, get rid of the pitch track.
set_signalCanvasesOrder(alb2_DB,
perspectiveName = "default",
order = "SPEC")And look again at the database. It should now be as in Fig. 2.1.
serve(alb2_DB, useViewer = F)4.4 Automatically linking annotations at the Tone tier
The first task is to provide a couple of annotations. More specifically add two pitch targets (e.g., L* to Lena, L* to leu) for the utterance 0001BF_1syll_1 as in Fig. 2.1. See
the sub-section in this module (scroll down to the grey box) to see details of how to annotate in Emu.
serve(alb2_DB, useViewer = F)The next task is to link automatically the annotations at the tonal levelTonetier to the corresponding words inORT2using the functionautobuild_linkFromTimes(). This function links an annotation at time *t* at theTonetier to an annotation at theORT2tier ifw_onset < t < w_offsetwherew_onsetandw_offset` are the word’s start and end times respectively (i.e. automatically link any tone to a word if the tone’s time falls within the boundary times of the word).
autobuild_linkFromTimes(alb2_DB,
superlevelName = "ORT2",
sublevelName = "Tone")Look at the database again: Check in the hierarchy view that the links between ORT2 and Tone were created in the path ORT2 -> Tone).
serve(alb2_DB, useViewer = F)It should now be possible to query tones with respect to words and vice-versa. For example:
# Find all pitch-accented words
# (i.e. find any word linked to any tone):
query(alb2_DB, "[Tone =~.* ^ #ORT2 =~.*]")
# A tibble: 2 × 16
# labels start end db_uuid session bundle start_item_id end_item_id level
# <chr> <dbl> <dbl> <chr> <chr> <chr> <int> <int> <chr>
#1 lena 1050. 1670. 589b69bd-e69a-… 0000 0001BF… 40 40 ORT2
#2 leu 1670. 2100. 589b69bd-e69a-… 0000 0001BF… 41 41 ORT2