Building machine learning systems with Python (779436), страница 25
Текст из файла (страница 25)
Finally, itprovides the best estimator in the form of the member variable, best_estimator_.As we want to compare the returned best classifier with our current best one, weneed to evaluate it in the same way. Therefore, we can pass the ShuffleSplitinstance using the cv parameter (therefore, CV in GridSearchCV).The last missing piece is to define how GridSearchCV should determine the bestestimator. This can be done by providing the desired score function to (surprise!)the score_func parameter. We can either write one ourselves, or pick one from thesklearn.metrics package. We should certainly not take metric.accuracy becauseof our class imbalance (we have a lot less tweets containing sentiment than neutralones).
Instead, we want to have good precision and recall on both classes, tweetswith sentiment and tweets without positive or negative opinions. One metric thatcombines both precision and recall is the so-called F-measure, which is implementedas metrics.f1_score:After putting everything together, we get the following code:from sklearn.grid_search import GridSearchCVfrom sklearn.metrics import f1_scoredef grid_search_model(clf_factory, X, Y):cv = ShuffleSplit([ 142 ]Chapter 6n=len(X), n_iter=10, test_size=0.3,random_state=0)param_grid = dict(vect__ngram_range=[(1, 1), (1, 2), (1, 3)],vect__min_df=[1, 2],vect__stop_words=[None, "english"],vect__smooth_idf=[False, True],vect__use_idf=[False, True],vect__sublinear_tf=[False, True],vect__binary=[False, True],clf__alpha=[0, 0.01, 0.05, 0.1, 0.5, 1],)grid_search = GridSearchCV(clf_factory(),param_grid=param_grid,cv=cv,score_func=f1_score,verbose=10)grid_search.fit(X, Y)return grid_search.best_estimator_We have to be patient while executing this:clf = grid_search_model(create_ngram_model, X, Y)print(clf)Since we have just requested a parameter, sweep overparameter combinations, each being trained on 10 folds:...
waiting some hours...Pipeline(clf=MultinomialNB(alpha=0.01, class_weight=None, fit_prior=True),clf__alpha=0.01,clf__class_weight=None,clf__fit_prior=True,vect=TfidfVectorizer(analyzer=word, binary=False,charset=utf-8, charset_error=strict,dtype=<type 'long'>,input=content,[ 143 ]Classification II – Sentiment Analysislowercase=True, max_df=1.0,max_features=None, max_n=None,min_df=1, min_n=None, ngram_range=(1, 2),norm=l2, preprocessor=None, smooth_idf=False,stop_words=None,strip_accents=None,sublinear_tf=True,token_pattern=(?u)\b\w\w+\b,token_processor=None, tokenizer=None,use_idf=False, vocabulary=None),vect__analyzer=word, vect__binary=False,vect__charset=utf-8,vect__charset_error=strict,vect__dtype=<type 'long'>,vect__input=content, vect__lowercase=True,vect__max_df=1.0,vect__max_features=None,vect__max_n=None, vect__min_df=1,vect__min_n=None, vect__ngram_range=(1, 2),vect__norm=l2, vect__preprocessor=None,vect__smooth_idf=False, vect__stop_words=None,vect__strip_accents=None, vect__sublinear_tf=True,vect__token_pattern=(?u)\b\w\w+\b,vect__token_processor=None, vect__tokenizer=None,vect__use_idf=False, vect__vocabulary=None)0.7950.0070.7020.028The best estimator indeed improves the P/R AUC by nearly 3.3 percent to now 70.2,with the settings shown in the previous code.Also, the devastating results for positive tweets against the rest and negative tweetsagainst the rest improve if we configure the vectorizer and classifier with thoseparameters we have just found out:== Pos vs.
rest ==0.8890.0100.5090.041== Neg vs. rest ==0.8860.0070.6150.035[ 144 ]Chapter 6Have a look at the following plots:Indeed, the P/R curves look much better (note that the plots are from the medium ofthe fold classifiers, thus, slightly diverging AUC values). Nevertheless, we probablystill wouldn't use those classifiers. Time for something completely different…[ 145 ]Classification II – Sentiment AnalysisCleaning tweetsNew constraints lead to new forms. Twitter is no exception in this regard.
Becausethe text has to fit into 140 characters, people naturally develop new languageshortcuts to say the same in less characters. So far, we have ignored all the diverseemoticons and abbreviations. Let's see how much we can improve by taking that intoaccount. For this endeavor, we will have to provide our own preprocessor() toTfidfVectorizer.First, we define a range of frequent emoticons and their replacements in a dictionary.Although we can find more distinct replacements, we go with obvious positive ornegative words to help the classifier:emo_repl = {# positive emoticons"<3": " good ",":d": " good ", # :D in lower case":dd": " good ", # :DD in lower case"8)": " good ",":-)": " good ",":)": " good ",";)": " good ","(-:": " good ","(:": " good ",# negative emoticons:":/": " bad ",":>": " sad ",":')": " sad ",":-(": " bad ",":(": " bad ",":S": " bad ",":-S": " bad ",}# make sure that e.g.
:dd is replaced before :demo_repl_order = [k for (k_len,k) in reversed(sorted([(len(k),k) for k inemo_repl.keys()]))][ 146 ]Chapter 6Then, we define abbreviations as regular expressions together with their expansions(\b marks the word boundary):re_repl = {r"\br\b": "are",r"\bu\b": "you",r"\bhaha\b": "ha",r"\bhahaha\b": "ha",r"\bdon't\b": "do not",r"\bdoesn't\b": "does not",r"\bdidn't\b": "did not",r"\bhasn't\b": "has not",r"\bhaven't\b": "have not",r"\bhadn't\b": "had not",r"\bwon't\b": "will not",r"\bwouldn't\b": "would not",r"\bcan't\b": "can not",r"\bcannot\b": "can not",}def create_ngram_model(params=None):def preprocessor(tweet):tweet = tweet.lower()for k in emo_repl_order:tweet = tweet.replace(k, emo_repl[k])for r, repl in re_repl.items():tweet = re.sub(r, repl, tweet)return tweettfidf_ngrams = TfidfVectorizer(preprocessor=preprocessor,analyzer="word")# ...[ 147 ]Classification II – Sentiment AnalysisCertainly, there are many more abbreviations that can be used here.
But already withthis limited set, we get an improvement for sentiment versus not sentiment of half apoint, being now 70.7 percent:== Pos vs. neg ==0.8080.0240.8850.029== Pos/neg vs. irrelevant/neutral ==0.7930.0100.6850.024== Pos vs. rest ==0.8900.0110.5170.041== Neg vs. rest ==0.8860.0060.6240.033Taking the word types into accountSo far, our hope was that simply using the words independent of each other withthe bag-of-words approach would suffice. Just from our intuition, however, neutraltweets probably contain a higher fraction of nouns, while positive or negative tweetsare more colorful, requiring more adjectives and verbs.
What if we use this linguisticinformation of the tweets as well? If we could find out how many words in a tweetwere nouns, verbs, adjectives, and so on, the classifier could probably take that intoaccount as well.Determining the word typesThis is what part-of-speech tagging, or POS tagging, is all about. A POS tagger parsesa full sentence with the goal to arrange it into a dependence tree, where each nodecorresponds to a word and the parent-child relationship determines which word itdepends on. With this tree, it can then make more informed decisions, for example,whether the word "book" is a noun ("This is a good book.") or a verb ("Could youplease book the flight?").You might have already guessed that NLTK will play its role in this area as well.And indeed, it comes readily packaged with all sorts of parsers and taggers.
ThePOS tagger we will use, nltk.pos_tag(), is actually a full blown classifier trainedusing manually annotated sentences from the Penn Treebank Project (http://www.cis.upenn.edu/~treebank). It takes as input a list of word tokens and outputs alist of tuples, where each element contains the part of the original sentence and itspart-of-speech tag.>>> import nltk>>> nltk.pos_tag(nltk.word_tokenize("This is a good book."))[ 148 ]Chapter 6[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('good', 'JJ'), ('book','NN'), ('.', '.')]>>> nltk.pos_tag(nltk.word_tokenize("Could you please book theflight?"))[('Could', 'MD'), ('you', 'PRP'), ('please', 'VB'), ('book', 'NN'),('the', 'DT'), ('flight', 'NN'), ('?', '.')]The POS tag abbreviations are taken from the Penn Treebank (adapted fromhttp://www.anc.org/OANC/penn.html):POS tagDescriptionExampleCCcoordinating conjunctionorCDcardinal number2, secondDTdeterminertheEXexistential therethere areFWforeign wordkindergartenINpreposition/subordinating conjunctionon, of, likeJJadjectivecoolJJRadjective, comparativecoolerJJSadjective, superlativecoolestLSlist marker1)MDmodalcould, willNNnoun, singular or massbookNNSnoun pluralbooksNNPproper noun, singularSeanNNPSproper noun, pluralVikingsPDTpredeterminerboth the boysPOSpossessive endingfriend'sPRPpersonal pronounI, he, itPRP$possessive pronounmy, hisRBadverbhowever, usually, naturally, here,goodRBRadverb, comparativebetterRBSadverb, superlativebestRPparticlegive upTOtoto go, to himUHinterjectionuhhuhhuhhVBverb, base formtake[ 149 ]Classification II – Sentiment AnalysisPOS tagDescriptionExampleVBDverb, past tensetookVBGverb, gerund/present participletakingVBNverb, past participletakenVBPverb, sing.
present, non-3dtakeVBZverb, 3rd person sing. presenttakesWDTwh-determinerwhichWPwh-pronounwho, whatWP$possessive wh-pronounwhoseWRBwh-abverbwhere, whenWith these tags, it is pretty easy to filter the desired tags from the output of pos_tag().We simply have to count all words whose tags start with NN for nouns, VB for verbs,JJ for adjectives, and RB for adverbs.Successfully cheating using SentiWordNetWhile linguistic information, as mentioned in the preceding section, will mostlikely help us, there is something better we can do to harvest it: SentiWordNet(http://sentiwordnet.isti.cnr.it). Simply put, it is a 13 MB file that assignsmost of the English words a positive and negative value.
More complicated put,for every synonym set, it records both the positive and negative sentiment values.Some examples are as follows:POSIDPosScoreNegScoreSynsetTermsDescriptiona003113540.250.125studious#1Marked by care andeffort; "made a studiousattempt to fix thetelevision set"a0031166300.5careless#1Marked by lack ofattention or considerationor forethought orthoroughness; notcareful…n0356371000implant#1A prosthesis placedpermanently in tissuev0036212800kink#2curve#5curl#1Form a curl, curve, orkink; "the cigar smokecurled up at the ceiling"[ 150 ]Chapter 6With the information in the POS column, we will be able to distinguish between thenoun "book" and the verb "book".
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
