kotelnikovevetal (Аннотации), страница 3

If the size of manuallexicon is equal to N, we take N first words with maximal RF-weights (Table 10).You may notice that in general, the coincidence is low—on average 17.1% in alllexicons and domains. At the same time the scattering is very large: for the positive—from 6.0% to 33.3%, for the negative—from 11.0% to 31.3%.Therefore, you should not rely only on automatic methods for sentiment lexi‑con creating. For example, top-100 RF-weighted positive words for the books domainKotelnikov E.

V. et al.contains such neutral words as подход (an approach), окружающий (surrounding),сестра (a sister), вставать (to stand up), держать (to hold), etc. In our opinion, theused hybrid approach where human annotators mark up a subset of the words se‑lected by automatic methods is more promising.Table 10. A comparison of manual and automatic lexiconsPositiveDomainRestaurantsCarsMoviesBooksCamerasAverageNegativeSizeCoincidence Size48334225135939636633.3%15.5%6.0%19.2%15.9%18.0%Coincidence85778031747773963431,3%20.1%11.0%19.1%14.2%16.1%6.

Comparison of lexicons in automatic sentiment analysisWe researched the performance of the sentiment analysis for different domainsusing prepared sentiment lexicons and compared with the dictionaries automaticallyformed on the basis of train collections, as well as with the existing lexicons (see Sec‑tion 2).A vector space model of text representation was used.

Automatically created dic‑tionaries based on the training collection were weighted using an RF scheme (Lanet al, 2009). Also a feature selection was applied for the dictionaries—the first p%of words with the highest weights were selected. The ratio p ranged from 10% to 100%with 10% step. For the other dictionaries the binary weights were used.The method SVM from scikit-learn package (Pedregosa et al., 2011) was used forclassification. The kernel (linear, polynomial, RBF), SVM parameters and parameterp in the feature selection through grid search and 3-fold cross-validation were se‑lected. The best results were achieved with a linear kernel.We included in testing the formed by annotators domain-dependent sentimentlexicons, which contained only the words about which agree all annotators (denoted“Domain, n = 4”) and most annotators (“Domain, n = 3”).

In addition, we used uni‑versal sentiment lexicons (n = 3 and n = 4).We also compared the quality of the analysis with the results of publicly availableRussian sentiment lexicons: ProductSentiRus (Chetviorkin, Loukachevitch, 2012),NRC (Mohammad, Turney, 2013) and Chen-Skiena (Chen, Skiena, 2014). The sizesof all the lexicons are listed in Table 11.As a baseline we used dummy classifier, which categorized all the objects as positive.For evaluation we used F1-measure, for which macro-averaging was carried outdue to the strong imbalance of test corpora.

The test results are shown in Table 12.Manually Created Sentiment Lexicons: Research and DevelopmentTable 11. Size of lexicons (for the lexicons of train collection in the bracketsit shows the part p of the full lexicon—the result of feature selection)LexiconDictionaries of traincorpus (RF)Domain (n = 4)Domain (n = 3)Universal (n = 4)Universal (n = 3)ProductSentiRusNRCChen-SkienaRestaurantsCarsMoviesBooksCameras21,454(1.0)6101,34012,467(0.7)2461,12223,164(0.8)1965681,1113,2475,0004,5902,91415,328(1.0)2648369,781(0.7)1681,135Table 12.

The results of experiments—F1-measure, %LexiconBaselineDictionaries of traincorpus (RF)Domain (n = 4)Domain (n = 3)Universal (n = 4)Universal (n = 3)ProductSentiRusNRCChen-SkienaRestaurants Cars Movies Books CamerasAverageF147.674.449.263.645.264.447.461.247.980.247.568.874.975.074.375.376.271.871.262.365.263.365.863.662.259.665.262.061.465.761.758.658.764.060.563.160.259.253.656.676.073.976.878.982.682.980.268.567.367.869.268.765.865.2From Table 12 it can be seen that the created sentiment lexicons allow to per‑form the sentiment analysis with high quality, comparable or superior the auto-gen‑erated dictionaries. At the same time the size of manual lexicons is much smallerthan of automatic lexicons: for example, lexicon books (Domain, n = 4) comprisesa total of 264 words and shows the quality that surpasses all other lexicons (64%).Also the universal lexicons demonstrate the high quality, for example, the uni‑versal lexicon (n = 3) shows the best results in two areas of the five (cars and movies),as well as on the average.Due to the high degree of imbalance of corpora (see Table 1) and the use of macroaveraging scheme, the quality of the analysis highly depends on the F1-measure fornegative texts.

Almost all relatively low results in Table 12 (e.g., Chen-Skiena for cars,dictionary of train corpus for books, NRC for movies) are closely related with poorrecognition of negative texts. Low results of manual lexicons for cameras also dependon it. The reason is the insufficient size of the negative lexicon for cameras (89 words,n = 4). Perhaps it was the result of poor awareness of annotators in a given domain.Kotelnikov E. V. et al.We note also that ProductSentiRus performed well in the analysis of product re‑views (cars and cameras), as well as the restaurants. Lexicons, received by automatictranslation into Russian (NRC and Chen-Skiena) tend to show relatively low quality(except cameras).7.

ConclusionThus, the proposed in the article procedure allows creating a compact and do‑main-dependent sentiment lexicon, which is very effective in sentiment analysis. Thelaboriousness of lexicon creation is reduced through the use of automated methodsof terms weighting to generate a set of words to labeling process. It is also importantfor annotators to be familiar with the domain.The universal lexicons created by union of manual lexicons also show good re‑sults comparable or superior to automatic dictionaries.We see the following directions for future research: to expand the set of domains(news, social networks, policy) to increase the reliability of research; to investigate theinfluence of collocations and parts of speech on the effectiveness of lexicons; to testthe lexicons with lexical-based method of sentiment analysis (Taboada et al., 2011).AcknowledgementsThe reported study was funded by RFBR according to the research projectNo. 16-07-00342 a.References1.2.3.4.5.Amiri F., Scerri S., Khodashahi M.

(2015), Lexicon-based Sentiment Analysis forPersian Text, Proceedings of Recent Advances in Natural Language Processing,Hissar, pp. 9–16.Andreevskaia A., Bergler S. (2006), Mining WordNet for Fuzzy Sentiment: Sen‑timent Tag Extraction from WordNet Glosses, Proceedings EACL-06, the 11rdConference of the European Chapter of the Association for Computational Lin‑guistics, Trento, pp. 209–216.Artstein R., Poesio M.

(2008), Inter-Coder Agreement for Computational Linguis‑tics, Computational Linguistics, Vol. 34, No. 4, pp. 555–596.Blinov P., Kotelnikov E. (2014), Using Distributed Representations for AspectBased Sentiment Analysis, Computational Linguistics and Intellectual Tech‑nologies: Proceedings of the International Conference “Dialogue 2014”, Beka‑sovo, No. 13 (20), Vol. 2., pp.

68–79.Boucher J. D., Osgood Ch. E. (1969), The Pollyanna Hypothesis, Journal of VerbalLearning and Verbal Behavior, No. 8, pp. 1–8.Manually Created Sentiment Lexicons: Research and Development6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.Chen Y., Skiena S. (2014), Building Sentiment Lexicons for All Major Languages,Proceedings of the 52nd Annual Meeting of the Association for ComputationalLinguistics, Baltimore, pp. 383–389.Chetviorkin I., Braslavskiy P., Loukachevitch N.

(2012), Sentiment Analysis Trackat ROMIP 2011, Computational Linguistics and Intellectual Technologies: Pro‑ceedings of the International Conference “Dialog2012”, No. 11 (18), Vol. 2.,pp. 1–14.Chetviorkin I., Loukachevitch N. (2012), Extraction of Russian Sentiment Lexiconfor Product Meta-Domain, Proceedings of COLING 2012, Mumbai, pp. 593–610.Chetviorkin I., Loukachevitch N. (2013), Sentiment Analysis Track at ROMIP2012, Computational Linguistics and Intellectual Technologies: Proceedingsof the International Conference “Dialog2013”, No.

12 (19), Vol. 2, pp. 40–50.Feldman R. (2013), Techniques and Applications for Sentiment Analysis, Com‑munications of ACM, Vol. 56, No. 4, pp. 82–89.Fellbaum C. (1998), WordNet: An Electronic Lexical Database, Cambridge, MA:MIT Press.Fleiss J. L. (1971), Measuring nominal scale agreement among many raters, Psy‑chological Bulletin, Vol. 76, No. 5, pp. 378–382.Freitas L., Vieira R. (2013), Comparing Portuguese Opinion Lexicons in FeatureBased Sentiment Analysis, International Journal of Computational Linguisticsand Applications, Vol.

4 (1), pp. 147–158.Ivanov V., Tutubalina E., Mingazov N., Alimova I. (2015), Extracting Aspects, Sen‑timent and Categories of Aspects in User Reviews about Restaurants and Cars,Computational Linguistics and Intellectual Technologies: Proceedings of the In‑ternational Conference “Dialogue 2015”, Moscow, pp. 22–33.Kiselev Y., Braslavski P., Menshikov I., Mukhin M., Krizhanovskaya N. (2015),Russian Lexicographic Landscape: a Tale of 12 Dictionaries, Computational Lin‑guistics and Intellectual Technologies: Proceedings of the International Confer‑ence “Dialog2015”, No. 14 (21), Vol.

1, pp. 254–50271Lan M., Tan C. L., Su J., Lu Y. (2009), Supervised and Traditional Term WeightingMethods for Automatic Text Categorization, IEEE Trans. on Pattern Analysis andMachine Intelligence, Vol. 31 (4), pp. 721–735.Landis J. R., Koch G. G. (1977), The Measurement of Observer Agreement forCategorical Data, Biometrics, Vol.

33, pp. 159–174.Liu B. (2012), Sentiment Analysis and Opinion Mining, Synthesis Lectureson Human Language Technologies, Vol. 5 (1).Mohammad S., Turney P. (2013), Crowdsourcing a Word-Emotion AssociationLexicon, Computational Intelligence, Vol.

29 (3), pp. 436–465.Musto C., Semeraro G., Polignano M. (2014), A Comparison of Lexicon-basedApproaches for Sentiment Analysis of Microblog Posts, DART 2014 8th Interna‑tional Workshop on Information Filtering and Retrieval, Pisa.Ozdemir C., Bergler S. (2015), A Comparative Study of Different Sentiment Lex‑ica for Sentiment Analysis of Tweets, Proceedings of Recent Advances in NaturalLanguage Processing, Hissar, pp. 488–496.Kotelnikov E. V. et al.22. Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., Blondel M.,Prettenhofer P., Weiss R., Dubourg V., Vanderplas J., Passos A., Cournapeau D.,Brucher M., Perrot M., Duchesnay É. (2011), Scikit-learn: Machine Learning in Py‑thon, Journal of Machine Learning Research, Vol.

12, pp. 2825–2830.23. Taboada M., Brooke J., Stede M. (2009), Genre-based Paragraph Classification forSentiment Analysis, Proceedings of the 10th Annual SIGDIAL Meeting on Dis‑course and Dialogue, London, pp. 62–70.24. Taboada M., Brooke J., Tofiloski M., Voll K., Stede M. (2011), Lexicon-Based Meth‑ods for Sentiment Analysis, Computational Linguistics, Vol. 37 (2), pp.

267–307.25. Ulanov A., Sapozhnikov G. (2013), Context-Dependent Opinion Lexicon Transla‑tion with the Use of a Parallel Corpus, Computational Linguistics and Intellec‑tual Technologies: Proceedings of the International Conference “Dialogue 2013”,Bekasovo, pp. 165–174..