LBSC 796/INFM 718R: Homework 1

Assignment adapted from James Allan's CMPSCI 646 course (Fall, 2004) at U. Mass.

Solutions

Part 1: Gathering Relevance Judgments

The purpose of this exercise is to gain some "hands-on" experience in the process of evaluating information retrieval systems. You will be assessing documents that are retrieved in response to two "topics" (statements of information needs). The two search engines we'll be comparing are Google and Teoma.

  1. gardening wet soil conditions
    What special considerations must be made when planting a garden in very wet soil conditions? Are there plants that will not work and/or that will work particularly well? Are there any special techniques that will help reduce the amount of moisture? Only pages that deal with gardening are relevant.
    [Google results] [Teoma results]

  2. oil vs. propane furnace
    Looking for pages that list the tradeoffs for using a propane furnace rather than a fuel oil furnace. The pages should provide comparison and are only relevant if they talk about both. They can talk about other types of heating (e.g., electric), provided they talk about oil and propane. Propane is also called LP or LPG (liquid propane [gas]).
    [Google results] [Teoma results]

To ensure that everyone evaluates the same hits, results from each search engine have been cached for you; follow the above links. You will evaluate the relevance of a subset of these hits. If your social security number ends in an even digit, evaluate the first 15 hits of each query. If you social security number ends in an odd digit, evaluate the last 15 hits of each query.

Use this Excel spreadsheet to keep track of your relevance judgments. In the column marked "Relevance", enter "R" if you think the document is relevant. Enter "N" if you think the document is not relevant. Email the results back to Lynne, who will gather the judgments from everyone. Change the name of the spreadsheet to your last name so that Lynne does not receive a dozen files with the same filename.

Part 2: Evaluating Systems

This Excel spreadsheet contains three separate sets of judgments for the hits examined in part one of the homework.

In the second part of this assignment, for one of the topics, you will

  1. analyze agreement on the relevance judgments
  2. adjudicate the judgments
  3. use the adjudicated set to evaluate both Teoma and Google

1. Agreement on Relevance Judgments

The above spreadsheet should contain three sets of judgments for every document (Web page). The first question you'll answer is: How often do judges agree on relevance? There are four possibilities:

For your chosen topic, figure out how often each case happens (both in terms of counts and in terms of percentage). Turn this information in. Pick three cases where judgments about a particular hit are not uniform, and briefly speculate why this may be so. Try to employ the concepts of relevance discussed in lecture. Turn this in.

2. Adjudication

Adjudication is simply the process of reconciling inconsistent judgments. Do this by simple majority voting. You do not need to turn anything in for this, but you will need the results for the third question.

3. Evaluation of Teoma and Google

Now, evaluate Teoma and Google using the adjudicated relevance judgments you just created (for the topic you chose). Make sure you are pooling judgments from both systems! Turn in the following information for both search engines:

In addition, answer the following questions: