kotelnikovevetal (Аннотации), страница 2

Two identical word lists, which contain alland second weight RFneg towards1 − words from full dictionary, are generated. Lists are sorted, the first—in the order of weightsRFpos, and the second—in the order of weights RFneg. First P words from each list are chosenso that 2P = N, where N—a number of words for manual annotation (at the top of both liststhe same words may occur). Thus the dictionary for manual labeling containing N hypo‑thetical sentiment words is made for the second stage.Table 2 shows the characteristicsof full dictionaries and dictionaries forlabeling.7https://tech.yandex.ru/mystemKotelnikov E.

V. et al.Table 2. Size of dictionariesDomainRestaurantsCarsMoviesBooksCamerasSize of full dictionary21,45417,81028,95515,32813,974Size of labelled dictionary10,00010,00010,00010,00010,000At the second stage M annotators independently label dictionary. In our studyM = 4 annotators take part in the annotation process. N = 10,000 is the compromisebetween the laboriousness and the completeness. The annotators labelled 50,000words (5 domains) altogether.

The dictionary was shuffled before annotation.Each word can be assigned one of four labels: positive, negative, neutral andunclear. Further neutral words are not used. The unclear word lists are of interestto further studies.The desktop application that shows the current word, its context and possiblelabels is used for the labeling process (Fig. 1).Fig.

1. Annotation toolThe annotators labelled the word as positive or negative in case they couldimagine it in any sentiment context of current domain. If the annotator had somedoubt the word was labelled as unclear, otherwise as neutral. The average time of la‑beling of a thousand of words was 90 minutes, overall labeling time was about 300man-hours.The anotators had the following main problems:Manually Created Sentiment Lexicons: Research and Development1)the ambiguity, e. g. «нашли кусок пластика» — «прекрасная пластикатанца» (“we found a piece of plastic”—“a great plastic of dance”);2)the reviews often have two parts—descriptive and evaluative.

The wordsthat are sentiment-bearing for descriptive part are not those for evaluativeand vice versa. In (Taboada et al., 2009) the solution of the problem of thedescriptive noise is proposed;3)the author of review’s was afraid of something but his or her fear was notconfirmed;4)for many words a number of reviews containing such words exceeds severaltens (and even hundreds)—for the annotators it was hard to see all reviewsin such cases;5)the morphological errors, e.g. word «отстой» (“bullshit”) is recognizedas «отстоять» (“to stand”);6)typos, e.g. «комплимент — комплемент» (“compliment—complement”).At the third stage positive and negative labelled word lists are joined, domaindependent and universal sentiment lexicons are formed8.5.

Analysis of lexicons5.1.DescriptionAs a result of the proposed procedure each annotator created four lexicons foreach of five domains (80 lexicons altogether). The characteristics of lexicon for restau‑rant domain are shown in Fig. 2.877590008152800077547442A number of words70006000Positive5000Negative4000NeutralUnclear30002000100016413936851475518274701354820666226970Annotator1Annotator2Annotator3Annotator4Fig.

2. The distribution of sentiment words for restaurant reviews8Created sentiment lexicons are available at: https://goo.gl/KRWo5X.Kotelnikov E. V. et al.The analysis of created lexicons allows us to draw following conclusions. Firstly,negative lexicon is more diverse: on average the size of negative lexicons is 1.63 timesmore than of positive ones, despite the fact that the positive words prevail in texts(Boucher, Osgood, 1969).

Secondly, the annotators differ in the degrees of confi‑dence in their labels: the average rate of unclear words varies from 0.5% to 3.6%.At the same time the intersections of all or the most part of manual lexicons givegood results of sentiment analysis comparable with automatic dictionaries (see Sec‑tion 6).Thirdly, the rate of sentiment lexicon ranges from 8.4% to 17.3% on average forvarious domains (Table 3). It should be noted that this rate is in specially collecteddictionary of candidate words.

For the full dictionary this rate is likely to be evenlower.Table 3. Average sizes of lexiconsDomainRestaurantsCarsMoviesBooksCamerasPositive Negative Neutral Unclear6084293894915351,1279754516239658,0318,4449,0268,7548,382235152134132119Total(Pos+Neg)/Total10,00010,00010,00010,00010,00017.3%14.0%8.4%11.1%15.0%5.2.Intersections and unionsWe built the intersection of two types of lexicons: for which all 4 annotatorsagree and for which at least 3 of 4 annotators agree. The characteristics of these lexi‑cons are shown in Tables 4 and 5.Table 4. Lexicons with 4 agreed annotatorsDomainRestaurantsCarsMoviesBooksCamerasAveragePositive Negative Neutral Unclear200878710979112410159109155891846,6737,1838,1237,7866,9697,347000100Total7,2837,4298,3198,0517,1377,644Part of labelleddictionary72.8%74.3%83.2%80.5%71.4%76.4%Manually Created Sentiment Lexicons: Research and DevelopmentTable 5.

Lexicons with the minimum 3 agreed annotatorsDomainRestaurantsCarsMoviesBooksCamerasAveragePositive Negative Neutral Unclear4833422513593963668577803174777396347,7408,0918,8738,5077,9748,2371422134Part of labelleddictionaryTotal9,0949,2159,4439,3449,1129,24290.9%92.2%94.4%93.4%91.1%92.4%The study of Tables 4 and 5 shows the decrease in scattering of labelled lexiconsparts in the transition from the agreement of all annotators to an agreement of at leastthree of them: from [71.4%...83.2%] to [90.9%...94.4%]. Thus, the degree of agree‑ment of the majority is higher than 90%.Also the universal dictionaries were created—the unions of dictionaries for alldomains with different minimum number of agreed annotators (Table 6).Table 6. The characteristics of universal lexiconsA minimum numberof agreed annotators Positive12342,7311,6141,047388Positive ∪Negative Negative4,9783,3382,2107247,5264,9273,2471,111Neutral25,68824,26023,02621,145Unclear2,324260221It may be noticed that the size of positive and negative lexicons union is less thanthe sum of positive and negative lexicons sizes separately.

The reason is that somewords occur in positive and negative lexicons simultaneously. For example in Table 7there are 10 such words for the minimum three agreed annotators.Table 7. Words belonging to both universal lexiconsPositive lexiconNegative lexiconWordDomain ExamplesDomain Examplesзасасыватьпредсказу‑емостьнепредска‑зуемостьпредсказу‑емыйнепредска‑зуемыйbookscarscamerasmovies,bookscars,camerasbooksзасасывает пыльпредсказуемостьинтригинепредсказуемостьрезультата съемкиконец предсказуемcars,camerasнепредсказуемыеотказыbookscars,camerasmovies,booksсюжет засасываетпредсказуемостьв поворотахсюжет нравитсянепредсказуемостьюпредсказуемо ведетсебянепредсказуемыереакции героевKotelnikov E. V. et al.WordPositive lexiconNegative lexiconDomain ExamplesDomain Examplesпростенько camerasцеплятьbooksзатрепы‑ватьреветьразжевы‑ватьbooksbookscamerasвсе простенькои со вкусомкнига цепляетза живоекнига ужезатрепанаревела в три ручьяразжевано для«тормозов»booksслишком простенькоcarsцепляет днищемземлюинвентарь затрепанrestau‑rantscarsbooksмотор реветразжеванный автором до неприличия�max (1, ) = log 2 �2 +�5.3.Inter-annotatormaxagreement(1, ) = log 2 �2 + = log 2 �2 +�max(1, )kappa statistical mea‑We compute inter-annotator agreement by means of Fleiss’sure (Fleiss,It is calculated as the ratio of degree of annotators agreement actu‑ − 1971).=ally attainedabove what would be predicted by chance and the degree of agreement1 − −above chance:attainable =1 − = − 1 − ,where —the mean of the proportions of agreeing annotator-annotator pairs for eachword; —the degree of agreement expected by chance.If the annotators are in complete agreement then  = 1.

If there is chance agree‑ment then  = 0.Also we compute inter-annotator agreementfor each category—positive, nega‑tive, neutral and unclear. The results are shown in Table 8.Table 8. Inter-annotator agreementDomainRestaurantsCarsMoviesBooksCamerasAveragePositive0.3530.3170.2480.2970.2620.295Negative0.3640.3060.2840.3220.2740.310Neutral0.7900.7960.8770.8490.7750.817Unclear0.0270.0170.0110.0190.0170.018Fleiss’ kappa0.5350.4710.4620.5040.4320.481The obtained values of Fleiss’ kappa (from 0.432 for cameras to 0.535 for res‑taurants) on the scale from paper (Landis, Koch, 1977) refer to “the moderate agree‑ment” (0.4...0.6).

Although (Artstein, Poesio, 2008) indicate, that only values above0.8 ensured an annotation of reasonable quality, our experiments show that the cre‑ated lexicons are of sufficient quality for sentiment analysis (see Section 6). = − Manually Created Sentiment Lexicons: Research and Development1 − The relatively low value of Fleiss’ kappa = 0.432 for the cameras, is possibly dueto a lesser awareness of annotators in this domain than in others.Note that Fleiss’ kappa was lower for movies regarding restaurants (despite thehigh degree of agreement in the Table 4), due to the high values of the degree of agree‑ment expected by chance.5.4.Parts of speechWe analyzed parts of speech distribution in the unions of positive and negative lexi‑cons for different domains (see Table 5), formed by at least 3 agreed annotators (Table 9).Table 9.

The distribution of parts of speechNounsVerbsAdjectives AdverbsOthersTotalDomain#%#%###%RestaurantsCarsMoviesBooksCamerasUniversalAverage33628114618925586524125.125.025.722.622.526.624.62763387214129483422420.6 512 38.230.1377 33.612.7 226 39.816.9 334 40.025.9437 38.525.7 1,118 34.422.0377 37.41,3401,1225688361,1353,247100100100100100100%#%21512512117114842815616.011.121.320.513.013.215.91131121.5%0.10.10.50.10.10.10.2As a result of the analysis it was found that adjectives occupy the largest partin the sentiment dictionaries (on average 37.4%). Adverbs have the smallest part(15.9%), except for Others. Nouns and verbs have approximately the same proportion(24.6% and 22%, respectively).Verbs have the highest variation of proportions in the domains: from 12.7% formovies to 30.1% for cars. This is probably due to the predominance of actions descrip‑tion in the reviews of the goods (cameras, cars), than in the reviews of the works of art(movies, books).5.5.Interconnection between manual and automatic lexiconsWe compared the sentiment lexicons created by annotators (minimum threeagreed) and automatically generated based on the weight RF.