An introduction to information retrieval. Manning_ Raghavan (2009) (811397), страница 51
Текст из файла (страница 51)
For example, the contexts /play/act/scene/title and/play/title in the document will be merged when matching againstthe query term /play/title#"Macbeth".• The context resemblance function is further relaxed: Contexts have a nonzero resemblance in many cases where the definition of C R in Equation (10.1)returns 0.See the references in Section 10.6 for details.These three changes alleviate the problem of sparse term statistics discussed in Section 10.2 and increase the robustness of the matching functionagainst poorly posed structural queries. The evaluation of S IM N O M ERGEand S IM M ERGE in the next section shows that the relaxed matching conditions of S IM M ERGE increase the effectiveness of XML retrieval.?Exercise 10.1Consider computing df for a structural term as the number of times that the structuralterm occurs under a particular parent node. Assume the following: the structuralterm hc, ti = author#"Herbert" occurs once as the child of the node squib; there are10 squib nodes in the collection; hc, ti occurs 1000 times as the child of article; there are1,000,000 article nodes in the collection.
The idf weight of hc, ti then is log2 10/1 ≈ 3.3when occurring as the child of squib and log2 1,000,000/1000 ≈ 10.0 when occurringas the child of article. (i) Explain why this is not an appropriate weighting for hc, ti.Why should hc, ti not receive a weight that is three times higher in articles than insquibs? (ii) Suggest a better way of computing idf.Exercise 10.2Write down all the structural terms occurring in the XML document in Figure 10.8.Exercise 10.3How many structural terms does the document in Figure 10.1 yield?10.4Evaluation of XML retrievalINEXThe premier venue for research on XML retrieval is the INEX (INitiative forthe Evaluation of XML retrieval) program, a collaborative effort that has produced reference collections, sets of queries, and relevance judgments.
Ayearly INEX meeting is held to present and discuss research results. TheOnline edition (c) 2009 Cambridge UP21110.4 Evaluation of XML retrieval12,107494 MB1995–20021,5326.93030number of documentssizetime of publication of articlesaverage number of XML nodes per documentaverage depth of a nodenumber of CAS topicsnumber of CO topics◮ Table 10.2 INEX 2002 collection statistics.articlebodyfront matterjournal titlearticle titleIEEE Transaction on Pattern AnalysisActivityrecognitionsectiontitleparagraphIntroductionThis work focuses on . . .◮ Figure 10.11 Simplified schema of the documents in the INEX collection.CO TOPICSCAS TOPICSINEX 2002 collection consisted of about 12,000 articles from IEEE journals.We give collection statistics in Table 10.2 and show part of the schema ofthe collection in Figure 10.11.
The IEEE journal collection was expanded in2005. Since 2006 INEX uses the much larger English Wikipedia as a test collection. The relevance of documents is judged by human assessors using themethodology introduced in Section 8.1 (page 152), appropriately modifiedfor structured documents as we will discuss shortly.Two types of information needs or topics in INEX are content-only or COtopics and content-and-structure (CAS) topics. CO topics are regular keyword queries as in unstructured information retrieval. CAS topics have structural constraints in addition to keywords. We already encountered an exam-Online edition (c) 2009 Cambridge UP21210 XML retrievalCOMPONENTCOVERAGEple of a CAS topic in Figure 10.3. The keywords in this case are summer andholidays and the structural constraints specify that the keywords occur in asection that in turn is part of an article and that this article has an embeddedyear attribute with value 2001 or 2002.Since CAS queries have both structural and content criteria, relevance assessments are more complicated than in unstructured retrieval.
INEX 2002defined component coverage and topical relevance as orthogonal dimensions of relevance. The component coverage dimension evaluates whether theelement retrieved is “structurally” correct, i.e., neither too low nor too highin the tree. We distinguish four cases:• Exact coverage (E). The information sought is the main topic of the component and the component is a meaningful unit of information.• Too small (S). The information sought is the main topic of the component,but the component is not a meaningful (self-contained) unit of information.• Too large (L). The information sought is present in the component, but isnot the main topic.• No coverage (N). The information sought is not a topic of the component.TOPICAL RELEVANCEThe topical relevance dimension also has four levels: highly relevant (3),fairly relevant (2), marginally relevant (1) and nonrelevant (0).
Componentsare judged on both dimensions and the judgments are then combined intoa digit-letter code. 2S is a fairly relevant component that is too small and3E is a highly relevant component that has exact coverage. In theory, thereare 16 combinations of coverage and relevance, but many cannot occur.
Forexample, a nonrelevant component cannot have exact coverage, so the combination 3N is not possible.The relevance-coverage combinations are quantized as follows:1.00 if (rel, cov) = 3E 0.75 if (rel, cov) ∈ {2E, 3L}0.50 if (rel, cov) ∈ {1E, 2L, 2S}Q(rel, cov) =0.25 if (rel, cov) ∈ {1S, 1L}0.00 if (rel, cov) = 0NThis evaluation scheme takes account of the fact that binary relevance judgments, which are standard in unstructured information retrieval (Section 8.5.1,page 166), are not appropriate for XML retrieval.
A 2S component providesincomplete information and may be difficult to interpret without more context, but it does answer the query partially. The quantization function Qdoes not impose a binary choice relevant/nonrelevant and instead allows usto grade the component as partially relevant.Online edition (c) 2009 Cambridge UP21310.4 Evaluation of XML retrievalalgorithmS IM N O M ERGES IM M ERGEaverage precision0.2420.271◮ Table 10.3 INEX 2002 results of the vector space model in Section 10.3 for contentand-structure (CAS) queries and the quantization function Q.The number of relevant components in a retrieved set A of componentscan then be computed as:#(relevant items retrieved) =∑ Q(rel (c), cov(c))c∈ AAs an approximation, the standard definitions of precision, recall and F fromChapter 8 can be applied to this modified definition of relevant items retrieved, with some subtleties because we sum graded as opposed to binaryrelevance assessments.
See the references on focused retrieval in Section 10.6for further discussion.One flaw of measuring relevance this way is that overlap is not accountedfor. We discussed the concept of marginal relevance in the context of unstructured retrieval in Section 8.5.1 (page 166). This problem is worse inXML retrieval because of the problem of multiple nested elements occurring in a search result as we discussed on page 203. Much of the recent focusat INEX has been on developing algorithms and evaluation measures thatreturn non-redundant results lists and evaluate them properly. See the references in Section 10.6.Table 10.3 shows two INEX 2002 runs of the vector space system we described in Section 10.3. The better run is the S IM M ERGE run, which incorporates few structural constraints and mostly relies on keyword matching.S IM M ERGE’s median average precision (where the median is with respect toaverage precision numbers over topics) is only 0.147.
Effectiveness in XMLretrieval is often lower than in unstructured retrieval since XML retrieval isharder. Instead of just finding a document, we have to find the subpart of adocument that is most relevant to the query. Also, XML retrieval effectiveness – when evaluated as described here – can be lower than unstructuredretrieval effectiveness on a standard evaluation because graded judgmentslower measured performance. Consider a system that returns a documentwith graded relevance 0.6 and binary relevance 1 at the top of the retrievedlist.
Then, interpolated precision at 0.00 recall (cf. page 158) is 1.0 on a binaryevaluation, but can be as low as 0.6 on a graded evaluation.Table 10.3 gives us a sense of the typical performance of XML retrieval,but it does not compare structured with unstructured retrieval. Table 10.4directly shows the effect of using structure in retrieval. The results are for aOnline edition (c) 2009 Cambridge UP21410 XML retrievalprecision at 5precision at 10precision at 20precision at 30content only0.20000.18200.17000.1527full structure0.32650.25310.17960.1531improvement63.3%39.1%5.6%0.3%◮ Table 10.4 A comparison of content-only and full-structure search in INEX2003/2004.language-model-based system (cf.
Chapter 12) that is evaluated on a subsetof CAS topics from INEX 2003 and 2004. The evaluation metric is precisionat k as defined in Chapter 8 (page 161). The discretization function used forthe evaluation maps highly relevant elements (roughly corresponding to the3E elements defined for Q) to 1 and all other elements to 0. The contentonly system treats queries and documents as unstructured bags of words.The full-structure model ranks elements that satisfy structural constraintshigher than elements that do not.
For instance, for the query in Figure 10.3an element that contains the phrase summer holidays in a section will be ratedhigher than one that contains it in an abstract.The table shows that structure helps increase precision at the top of theresults list. There is a large increase of precision at k = 5 and at k = 10. Thereis almost no improvement at k = 30.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
