Compare Annotator reliability — compare.rating • ChildRecordsR

Provide indication of quality for each annotator. Following classical test theory, each annotator is removed from the data and a reliability indicator is provided for the annotation as a whole. Removing a good annotator should decrease the indicator. Additionally, a mean of precision, recall and F-score are provided by rater.

compare.rating(raterData, verbose = FALSE, threads = 1)

Arguments

raterData	: a raterData class
verbose	: if TRUE information will be printed out in the console
threads	the number of threads to run in parallel

Value

an object of class 'raterComp'

Examples


if (FALSE) {

library(ChildRecordsR)
path = "/mnt/94707AA4707A8CAC/CNRS/corpus/vandam-daylong-demo"
CR = ChildRecordings(path)

# Add a dummy anotation for the example

New.annotations(row.meta = CR$all.meta[2,], ChildRecordings = CR, time.sd = 1500,
                change.cat.prob = 0.10, new.name = "vtc_BAD")

# if no time windows is specified, this function will only return at table for all the know raters
# All the rater need to ratter any segment find

search = find.rating.segment(CR, "BN32_010007.mp3")
rez = aggregate.rating(search, CR, cut=100, verbose=T, use_data_table = TRUE, threads = 2)

comparaison = compare.rating(rez, verbose = TRUE, threads = 5)
plot(comparaison)

}