Call for Participation

			SIGIR 2003 Workshop on
   Defining Evaluation Methodologies for Terabyte-Scale Collections

			    August 1, 2003
			   Toronto, Canada

				-----
			  DEADLINE EXTENDED!
	  Position papers from those wishing to participate
		     are due Wednesday, June 4!!
				-----

Early retrieval test collections were small, allowing relevance
judgments to be based on an exhaustive examination of the documents,
but limiting the general applicability of the findings.  Karen Sparck
Jones and Keith van Rijsbergen proposed a way of building
significantly larger test collections by using pooling, a procedure
adopted and subsequently validated by TREC.  Now TREC-sized
collections (several gigabytes of text and a few million documents)
are small for some realistic tasks, but current pooling practices do
not scale to substantially larger document sets.  The goal of this
workshop is to develop an evaluation methodology for terabyte-scale
document collections.  

In particular, we will assume the existence of a collection of several
hundred million web pages, and investigate methods for reliably
evaluating retrieval tasks on such a collection.  Search tasks
currently evaluated using large web collections, such as known-item
and high-precision searching, focus on the needs of the common web
searcher but also arise from our inability to measure recall on very
large collections.  Good estimates of the total set of relevant
documents are critical to the reliability and reusability of test
collections as we now use them, but it would take hundreds of
different systems, hundreds of relevance assessors, and years of
effort to produce a terabyte-sized collection with completeness of
judgments comparable to a typical TREC collection.  Hence, new
evaluation methodologies and ways of building test collections are
needed to scale retrieval experiments to the next level.

The workshop will focus on discussions among workshop participants
rather than paper presentations.  The morning session will begin with
an introduction and three or four brief focus talks.  These
presentations will necessarily depend on who attends, but we expect
topics to include building realistic web collections, the recall
problem in large collections, and novel evaluation methodologies.  The
workshop will then break out into small groups to discuss each of
these facets and how they might be made to work in practice.  After
lunch, each of the break-out groups will report back to the workshop.
Finally, we will brainstorm a proposal for a new Very Large Corpus
track to be held at TREC 2004.

Those wishing to participate should submit a short (1-2 page) position
paper describing their thoughts on very large evaluations and
terabyte-sized test collections.  The workshop is limited to 25
attendees who will be selected by the organizing committee based on
their position papers.  

Organizers: Ian Soboroff, NIST, USA          (ian.soboroff@nist.gov)
	    Ellen Voorhees, NIST, USA        (ellen.voorhees@nist.gov)
	    Nick Craswell, CSIRO, Australia  (nick.craswell@csiro.au)

Submissions may be sent via e-mail in PDF or PostScript format to 
Ian Soboroff <ian.soboroff@nist.gov>.

DATES -- Submissions due:           June 4, 2003
	 Acceptance notification:   June 16, 2003
	 Workshop:                  August 1, 2003