An introduction to information retrieval. Manning_ Raghavan (2009) (811397), страница 43
Текст из файла (страница 43)
The models typically combine positional factors, favoring the first and last paragraphs of documents and the first and lastsentences of paragraphs, with content factors, emphasizing sentences withkey terms, which have low document frequency in the collection as a whole,but high frequency and good distribution across the particular documentbeing returned. In sophisticated NLP approaches, the system synthesizessentences for a summary, either by doing full text generation or by editingand perhaps combining sentences used in the document. For example, itmight delete a relative clause or replace a pronoun with the noun phrasethat it refers to.
This last class of methods remains in the realm of researchand is seldom used for search results: it is easier, safer, and often even betterto just use sentences from the original document.Dynamic summaries display one or more “windows” on the document,aiming to present the pieces that have the most utility to the user in evaluating the document with respect to their information need. Usually thesewindows contain one or several of the query terms, and so are often referred to as keyword-in-context (KWIC) snippets, though sometimes they maystill be pieces of the text such as the title that are selected for their queryindependent information value just as in the case of static summarization.Dynamic summaries are generated in conjunction with scoring.
If the queryis found as a phrase, occurrences of the phrase in the document will beOnline edition (c) 2009 Cambridge UP1728 Evaluation in information retrieval. . . In recent years, Papua New Guinea has faced severe economicdifficulties and economic growth has slowed, partly as a result of weakgovernance and civil war, and partly as a result of external factors such asthe Bougainville civil war which led to the closure in 1989 of the Pangunamine (at that time the most important foreign exchange earner andcontributor to Government finances), the Asian financial crisis, a decline inthe prices of gold and copper, and a fall in the production of oil.
PNG’seconomic development record over the past few years is evidence thatgovernance issues underly many of the country’s problems. Goodgovernance, which may be defined as the transparent and accountablemanagement of human, natural, economic and financial resources for thepurposes of equitable and sustainable development, flows from properpublic sector management, efficient fiscal and accounting mechanisms, anda willingness to make service delivery a priority in practice.
. . .◮ Figure 8.5 An example of selecting text for a dynamic snippet. This snippet wasgenerated for a document in response to the query new guinea economic development.The figure shows in bold italic where the selected snippet text occurred in the originaldocument.shown as the summary. If not, windows within the document that containmultiple query terms will be selected.
Commonly these windows may juststretch some number of words to the left and right of the query terms. This isa place where NLP techniques can usefully be employed: users prefer snippets that read well because they contain complete phrases.Dynamic summaries are generally regarded as greatly improving the usability of IR systems, but they present a complication for IR system design. Adynamic summary cannot be precomputed, but, on the other hand, if a system has only a positional index, then it cannot easily reconstruct the contextsurrounding search engine hits in order to generate such a dynamic summary. This is one reason for using static summaries.
The standard solutionto this in a world of large and cheap disk drives is to locally cache all thedocuments at index time (notwithstanding that this approach raises variouslegal, information security and control issues that are far from resolved) asshown in Figure 7.5 (page 147). Then, a system can simply scan a documentwhich is about to appear in a displayed results list to find snippets containingthe query words.
Beyond simply access to the text, producing a good KWICsnippet requires some care. Given a variety of keyword occurrences in adocument, the goal is to choose fragments which are: (i) maximally informative about the discussion of those terms in the document, (ii) self-containedenough to be easy to read, and (iii) short enough to fit within the normallystrict constraints on the space available for summaries.Online edition (c) 2009 Cambridge UP8.8 References and further reading173Generating snippets must be fast since the system is typically generatingmany snippets for each query that it handles.
Rather than caching an entiredocument, it is common to cache only a generous but fixed size prefix ofthe document, such as perhaps 10,000 characters. For most common, shortdocuments, the entire document is thus cached, but huge amounts of localstorage will not be wasted on potentially vast documents. Summaries ofdocuments whose length exceeds the prefix size will be based on materialin the prefix only, which is in general a useful zone in which to look for adocument summary anyway.If a document has been updated since it was last processed by a crawlerand indexer, these changes will be neither in the cache nor in the index.
Inthese circumstances, neither the index nor the summary will accurately reflect the current contents of the document, but it is the differences betweenthe summary and the actual document content that will be more glaringlyobvious to the end user.8.8FMEASUREReferences and further readingDefinition and implementation of the notion of relevance to a query got offto a rocky start in 1953. Swanson (1988) reports that in an evaluation in thatyear between two teams, they agreed that 1390 documents were variouslyrelevant to a set of 98 questions, but disagreed on a further 1577 documents,and the disagreements were never resolved.Rigorous formal testing of IR systems was first completed in the Cranfieldexperiments, beginning in the late 1950s. A retrospective discussion of theCranfield test collection and experimentation with it can be found in (Cleverdon 1991). The other seminal series of early IR experiments were those on theSMART system by Gerard Salton and colleagues (Salton 1971b; 1991).
TheTREC evaluations are described in detail by Voorhees and Harman (2005).Online information is available at http://trec.nist.gov/. Initially, few researcherscomputed the statistical significance of their experimental results, but the IRcommunity increasingly demands this (Hull 1993). User studies of IR systemeffectiveness began more recently (Saracevic and Kantor 1988; 1996).The notions of recall and precision were first used by Kent et al. (1955),although the term precision did not appear until later. The F measure (or,rather its complement E = 1 − F) was introduced by van Rijsbergen (1979).He provides an extensive theoretical discussion, which shows how adoptinga principle of decreasing marginal relevance (at some point a user will beunwilling to sacrifice a unit of precision for an added unit of recall) leads tothe harmonic mean being the appropriate method for combining precisionand recall (and hence to its adoption rather than the minimum or geometricmean).Online edition (c) 2009 Cambridge UP1748 Evaluation in information retrievalR- PRECISIONKAPPA STATISTICBuckley and Voorhees (2000) compare several evaluation measures, including precision at k, MAP, and R-precision, and evaluate the error rate ofeach measure.
R-precision was adopted as the official evaluation metric inthe TREC HARD track (Allan 2005). Aslam and Yilmaz (2005) examine itssurprisingly close correlation to MAP, which had been noted in earlier studies (Tague-Sutcliffe and Blustein 1995, Buckley and Voorhees 2000). A standard program for evaluating IR systems which computes many measures ofranked retrieval effectiveness is Chris Buckley’s trec_eval program usedin the TREC evaluations.
It can be downloaded from: http://trec.nist.gov/trec_eval/.Kekäläinen and Järvelin (2002) argue for the superiority of graded relevance judgments when dealing with very large document collections, andJärvelin and Kekäläinen (2002) introduce cumulated gain-based methods forIR system evaluation in this context. Sakai (2007) does a study of the stability and sensitivity of evaluation measures based on graded relevance judgments from NTCIR tasks, and concludes that NDCG is best for evaluatingdocument ranking.Schamber et al.
(1990) examine the concept of relevance, stressing its multidimensional and context-specific nature, but also arguing that it can be measured effectively. (Voorhees 2000) is the standard article for examining variation in relevance judgments and their effects on retrieval system scores andranking for the TREC Ad Hoc task. Voorhees concludes that although thenumbers change, the rankings are quite stable. Hersh et al. (1994) presentsimilar analysis for a medical IR collection. In contrast, Kekäläinen (2005)analyze some of the later TRECs, exploring a 4-way relevance judgment andthe notion of cumulative gain, arguing that the relevance measure used doessubstantially affect system rankings. See also Harter (1998). Zobel (1998)studies whether the pooling method used by TREC to collect a subset of documents that will be evaluated for relevance is reliable and fair, and concludesthat it is.The kappa statistic and its use for language-related purposes is discussedby Carletta (1996).
Many standard sources (e.g., Siegel and Castellan 1988)present pooled calculation of the expected agreement, but Di Eugenio andGlass (2004) argue for preferring the unpooled agreement (though perhapspresenting multiple measures). For further discussion of alternative measures of agreement, which may in fact be better, see Lombard et al. (2002)and Krippendorff (2003).Text summarization has been actively explored for many years.
Modernwork on sentence selection was initiated by Kupiec et al. (1995). More recentwork includes (Barzilay and Elhadad 1997) and (Jing 2000), together witha broad selection of work appearing at the yearly DUC conferences and atother NLP venues. Tombros and Sanderson (1998) demonstrate the advantages of dynamic summaries in the IR context. Turpin et al. (2007) addresshow to generate snippets efficiently.Online edition (c) 2009 Cambridge UP8.8 References and further reading175Clickthrough log analysis is studied in (Joachims 2002b, Joachims et al.2005).In a series of papers, Hersh, Turpin and colleagues show how improvements in formal retrieval effectiveness, as evaluated in batch experiments, donot always translate into an improved system for users (Hersh et al.
2000a;b;2001, Turpin and Hersh 2001; 2002).User interfaces for IR and human factors such as models of human information seeking and usability testing are outside the scope of what we coverin this book. More information on these topics can be found in other textbooks, including (Baeza-Yates and Ribeiro-Neto 1999, ch.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
