An introduction to information retrieval. Manning_ Raghavan (2009) (811397), страница 38
Текст из файла (страница 38)
If a search engine allowsa user to specify a wildcard operator as part of a free text query (for instance,the query rom* restaurant), we may interpret the wildcard component of thequery as spawning multiple terms in the vector space (in this example, romeOnline edition (c) 2009 Cambridge UP7.4 References and further reading149and roman would be two such terms) all of which are added to the queryvector.
The vector space query is then executed as usual, with matchingdocuments being scored and ranked; thus a document containing both romeand roma is likely to be scored higher than another containing only one ofthem. The exact score ordering will of course depend on the relative weightsof each term in matching documents.Phrase queriesThe representation of documents as vectors is fundamentally lossy: the relative order of terms in a document is lost in the encoding of a document asa vector. Even if we were to try and somehow treat every biword as a term(and thus an axis in the vector space), the weights on different axes not independent: for instance the phrase German shepherd gets encoded in the axisgerman shepherd, but immediately has a non-zero weight on the axes germanand shepherd. Further, notions such as idf would have to be extended to suchbiwords.
Thus an index built for vector space retrieval cannot, in general, beused for phrase queries. Moreover, there is no way of demanding a vectorspace score for a phrase query — we only know the relative weights of eachterm in a document.On the query german shepherd, we could use vector space retrieval to identify documents heavy in these two terms, with no way of prescribing thatthey occur consecutively. Phrase retrieval, on the other hand, tells us of theexistence of the phrase german shepherd in a document, without any indication of the relative frequency or weight of this phrase.
While these tworetrieval paradigms (phrase and vector space) consequently have differentimplementations in terms of indexes and retrieval algorithms, they can insome cases be combined usefully, as in the three-step example of query parsing in Section 7.2.3.7.4TOP DOCSReferences and further readingHeuristics for fast query processing with early termination are described byAnh et al. (2001), Garcia et al.
(2004), Anh and Moffat (2006b), Persin et al.(1996). Cluster pruning is investigated by Singitham et al. (2004) and byChierichetti et al. (2007); see also Section 16.6 (page 372). Champion lists aredescribed in Persin (1994) and (under the name top docs) in Brown (1995),and further developed in Brin and Page (1998), Long and Suel (2003). Whilethese heuristics are well-suited to free text queries that can be viewed as vectors, they complicate phrase queries; see Anh and Moffat (2006c) for an indexstructure that supports both weighted and Boolean/phrase searches.
Carmelet al. (2001) Clarke et al. (2000) and Song et al. (2005) treat the use of queryOnline edition (c) 2009 Cambridge UP1507 Computing scores in a complete search systemterm proximity in assessing relevance. Pioneering work on learning of ranking functions was done by Fuhr (1989), Fuhr and Pfeifer (1994), Cooper et al.(1994), Bartell (1994), Bartell et al. (1998) and by Cohen et al. (1998).Online edition (c) 2009 Cambridge UPDRAFT! © April 1, 2009 Cambridge University Press.
Feedback welcome.8151Evaluation in informationretrievalWe have seen in the preceding chapters many alternatives in designing an IRsystem. How do we know which of these techniques are effective in whichapplications? Should we use stop lists? Should we stem? Should we use inverse document frequency weighting? Information retrieval has developedas a highly empirical discipline, requiring careful and thorough evaluation todemonstrate the superior performance of novel techniques on representativedocument collections.In this chapter we begin with a discussion of measuring the effectivenessof IR systems (Section 8.1) and the test collections that are most often usedfor this purpose (Section 8.2).
We then present the straightforward notion ofrelevant and nonrelevant documents and the formal evaluation methodology that has been developed for evaluating unranked retrieval results (Section 8.3). This includes explaining the kinds of evaluation measures thatare standardly used for document retrieval and related tasks like text classification and why they are appropriate. We then extend these notions anddevelop further measures for evaluating ranked retrieval results (Section 8.4)and discuss developing reliable and informative test collections (Section 8.5).We then step back to introduce the notion of user utility, and how it is approximated by the use of document relevance (Section 8.6). The key utilitymeasure is user happiness. Speed of response and the size of the index arefactors in user happiness.
It seems reasonable to assume that relevance ofresults is the most important factor: blindingly fast, useless answers do notmake a user happy. However, user perceptions do not always coincide withsystem designers’ notions of quality. For example, user happiness commonlydepends very strongly on user interface design issues, including the layout,clarity, and responsiveness of the user interface, which are independent ofthe quality of the results returned.
We touch on other measures of the quality of a system, in particular the generation of high-quality result summarysnippets, which strongly influence user utility, but are not measured in thebasic relevance ranking paradigm (Section 8.7).Online edition (c) 2009 Cambridge UP1528 Evaluation in information retrieval8.1Information retrieval system evaluationTo measure ad hoc information retrieval effectiveness in the standard way,we need a test collection consisting of three things:1. A document collection2. A test suite of information needs, expressible as queries3. A set of relevance judgments, standardly a binary assessment of eitherrelevant or nonrelevant for each query-document pair.RELEVANCEGOLD STANDARDGROUND TRUTHINFORMATION NEEDThe standard approach to information retrieval system evaluation revolvesaround the notion of relevant and nonrelevant documents. With respect to auser information need, a document in the test collection is given a binaryclassification as either relevant or nonrelevant.
This decision is referred to asthe gold standard or ground truth judgment of relevance. The test documentcollection and suite of information needs have to be of a reasonable size:you need to average performance over fairly large test sets, as results arehighly variable over different documents and information needs. As a ruleof thumb, 50 information needs has usually been found to be a sufficientminimum.Relevance is assessed relative to an information need, not a query. Forexample, an information need might be:Information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine.This might be translated into a query such as:wine AND red AND white AND heart AND attack AND effectiveA document is relevant if it addresses the stated information need, not because it just happens to contain all the words in the query. This distinction isoften misunderstood in practice, because the information need is not overt.But, nevertheless, an information need is present.
If a user types python into aweb search engine, they might be wanting to know where they can purchasea pet python. Or they might be wanting information on the programminglanguage Python. From a one word query, it is very difficult for a system toknow what the information need is. But, nevertheless, the user has one, andcan judge the returned results on the basis of their relevance to it. To evaluate a system, we require an overt expression of an information need, whichcan be used for judging returned documents as relevant or nonrelevant.
Atthis point, we make a simplification: relevance can reasonably be thoughtof as a scale, with some documents highly relevant and others marginallyso. But for the moment, we will use just a binary decision of relevance. WeOnline edition (c) 2009 Cambridge UP8.2 Standard test collectionsDEVELOPMENT TESTCOLLECTION8.2153discuss the reasons for using binary relevance judgments and alternatives inSection 8.5.1.Many systems contain various weights (often known as parameters) thatcan be adjusted to tune system performance. It is wrong to report results ona test collection which were obtained by tuning these parameters to maximize performance on that collection. That is because such tuning overstatesthe expected performance of the system, because the weights will be set tomaximize performance on one particular set of queries rather than for a random sample of queries.
In such cases, the correct procedure is to have oneor more development test collections, and to tune the parameters on the development test collection. The tester then runs the system with those weightson the test collection and reports the results on that collection as an unbiasedestimate of performance.Standard test collectionsHere is a list of the most standard test collections and evaluation series. Wefocus particularly on test collections for ad hoc information retrieval systemevaluation, but also mention a couple of similar test collections for text classification.C RANFIELDThe Cranfield collection.
This was the pioneering test collection in allowingprecise quantitative measures of information retrieval effectiveness, butis nowadays too small for anything but the most elementary pilot experiments. Collected in the United Kingdom starting in the late 1950s, it contains 1398 abstracts of aerodynamics journal articles, a set of 225 queries,and exhaustive relevance judgments of all (query, document) pairs.TRECText Retrieval Conference (TREC).
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
