\documentclass[12pt]{article}
\usepackage{fancyhdr}
%\usepackage{url}

\topmargin -0.8cm
\oddsidemargin -0.7cm

\textwidth 17.5cm
\textheight 22.6 cm

\usepackage[
bookmarks=false
,bookmarksnumbered=true
,hypertexnames=false
,breaklinks=true
]{hyperref}
\hypersetup{
  pdfauthor={Kamps et al.},
  pdftitle={Report on the SIGIR 2009 Workshop on the Future of IR Evaluation},
  pdfsubject={SIGIR Forum 2009},
  pdfkeywords={H.3 [Information Storage and Retrieval]},
  pdfcreator={LaTeX with hyperref package},
  pdfproducer={pdflatex}
}
\usepackage{graphics}
\usepackage{color}
\definecolor{gray}{gray}{.7}%
\usepackage{tabularx}

% Natbib?
\usepackage[square,comma,numbers,sort&compress,sectionbib]{natbib}
%\usepackage{natbib}
%\newcommand{\newblock}{}
%\newcommand{\citep}[1]{\cite{#1}}
%\newcommand{\citet}[1]{\citeNP{#1}~\cite{#1}}


\pagestyle{fancy} {\fancyhead{}
\fancyfoot[c]{\small{\rule{17.5cm}{1pt}}}}

\begin{document}
\title{\vspace{-2.5cm}
\begin{center}
\textbf{\small{WORKSHOP REPORT}}\\\vspace{-0.5cm} \rule{17.5cm}{1pt}
\end{center}
\vspace{1cm}\textbf{Report on the SIGIR 2009 Workshop on\\ the Future of
IR Evaluation}}

\author{
  \begin{tabular}{ccc}
    \bf Jaap Kamps$^{1}$
    & \bf Shlomo Geva$^{2}$
    % & \bf Noriko Kando$^{3}$
    & \bf Carol Peters$^{3}$
    \\
    \bf Tetsuya Sakai$^{4}$ % tetsuyasakai@acm.org
    & \bf Andrew Trotman$^{5}$
    & \bf Ellen Voorhees$^{6}$
    \\
  \end{tabular}
}

\date{
  $^{1}$~University of Amsterdam, The Netherlands \& INEX
  \\
  $^{2}$~QUT, Australia \& INEX 
  \\
  $^{3}$~ISTI-CNR, Italy \& CLEF
  \\
  $^{4}$~Microsoft Research Asia, China \& NTCIR
  \\
  $^{5}$~University of Otago, New Zealand \& INEX
  \\
  $^{6}$~NIST, USA \& TREC/TAC
  \\
}

\maketitle 
\thispagestyle{fancy} 

\abstract{% abstract here 
  \noindent
  On July 23, 2009 the SIGIR Workshop on the Future of IR Evaluation
  was held as part of SIGIR in Boston.  The program consisted of four
  keynotes, a boaster and poster session with 20 accepted papers, four
  breakout groups, and a final panel discussion of the breakout
  group reports.  This report outlines the events of the workshop and
  summarizes the major outcomes.
}


\section{Introduction}
Evaluation is at the core of the field of Information Retrieval (IR).
One of the greatest achievements of the field is the development of a
rigorous methodology to evaluate retrieval effectiveness.  This
so-called Cranfield approach, as continued by the current evaluation
fora, has served us very well: virtually all progress in IR owes
directly or indirectly to test collections built within the Cranfield
paradigm.
%
However, in recent years, IR researchers are routinely pursuing tasks
outside the traditional paradigm.
%
For example, by taking a broader view on tasks, users, and
context~\citep{spar:what06}.  There is a fast moving evolution in
content from traditional static text to diverse forms of dynamic,
collaborative, and multilingual information sources.
%\todo[Are they?]
Also industry is embracing
``operational'' evaluation based on the analysis of sheer endless
streams of queries and clicks.
%
The recent MINDS research agenda calls for changes in data and
context, in information analysis and organization, and in novel
evaluation paradigms~\citep{call:meet07}.

It has been 50 years since the start of the Aslib Cranfield research
project that laid the foundations of experimental research in
Information Retrieval \citep{cleverdon60}.  The joint chairs of the
evaluation fora (CLEF, INEX, NTCIR, and TREC) organized a workshop on
\emph{the future of IR evaluation}, aiming to perform a sanity-check
on the current IR evaluation fora against the novel evaluation needs
of IR researchers (what are we doing right? and what do we fail to
address?) as well as to work out concrete new IR tracks and tasks to
take IR evaluation forward.
% (why? what scenario/task?  what collection? what information needs?
% how to evaluate?).

The questions we expected to address can be succinctly summarized as
to make IR evaluation more ``realistic.''  There is however no
consensus on what then ``real'' IR is, or should be, and various
directions have been proposed:
\begin{itemize} 
\item System: from \emph{ranking component} to \ldots ?
\item Scale: from \emph{megabytes/terabytes} to \ldots ?
\item Tasks: from \emph{library search/document triage}, to \ldots ?
\item Results: from \emph{documents} to \ldots ?
\item Genre: from \emph{English news} to \ldots ?
\item Users: from \emph{abstracted users} to \ldots ?
\item Information needs: from \emph{crisp fact finding} to \ldots ?
\item Usefulness: from \emph{topically relevant} to \ldots ?
\item Judgments: from \emph{explicit judgments} to \ldots ?
\item Interactive: from \emph{one-step batch processing} to \ldots ?
\item Adaptive: from \emph{one-size-fits-all} to \ldots ?
\item And many, many more... %\todo[Needs work...]
\end{itemize}

% The workshop brings together, on the one hand, those with a novel IR
% evaluation needs, and, on the other hand, senior IR evaluation experts
% and the chairs of evaluation fora.
% The workshop brings together all stake-holders, ranging from those
% with novel evaluation needs, such as a PhD candidate pursuing a new
% IR-related problem, to senior IR evaluation experts. Desired outcomes
% are insight into how to make IR evaluation more "realistic," and at
% least one concrete idea for a retrieval track or task (at CLEF, INEX,
% NTCIR, TREC) that would not have happened otherwise. 
% \begin{itemize}
% \item those with a novel IR evaluation need, such as a phd candidate
%   pursuing a new IR-related problem, and
% \item senior IR evaluation experts and the chairs of evaluation
%   initiatives (CLEF, INEX, NTCIR, TAC, TREC).
% \end{itemize}
We envisioned a true \emph{work}shop where all stake-holders, ranging
from those with novel evaluation needs to senior IR evaluation
experts, are brought together, and develop concrete ideas for IR
evaluation in the coming years.
% By bringing the ``evaluation problem owners'' and ``evaluation
% problem solvers'' together,
The first part of the workshop consisted of four keynotes to set the
stages and frame the problem (discussed in Section~\ref{sec:key}); and
a boasters and posters of twenty contributed papers (discussed in
Section~\ref{sec:pos}). The second part of the workshop consisted of
breakout groups on 4 themes (discussed in Section~\ref{sec:bre}), and
a report of the outcome to a panel of experts (discussed in
Section~\ref{sec:pan}).  The major outcomes of the workshop are
discussed in Section~\ref{sec:sum}.

% The workshop aimed to achieve:
% \begin{itemize} 
% \item some concrete insight into the future of IR evaluation;
% \item finding middle ground between novel IR task characteristics and
%   IR evaluation abstractions and requirements; and
% \item at least one concrete idea for a retrieval track or task that
%   would not have happened otherwise.
% \end{itemize}
% In addition to this, the discussion itself will help participating
% researchers to improve their own research plans in various ways.  For
% research problems unsuitable for large-scale collaborative test
% collection building, contact with fellow researchers working on
% closely related problems may lead to several grassroots initiatives.


\section{Keynotes}
\label{sec:key}
The program started with four keynote speakers.
% STEPHEN
Stephen Robertson's keynote was titled ``Richer theories, richer
experiments'' \citep{4}.  Stephen took a bird's eye view of IR
evaluation in relation to the roles of theory and experiment in the
philosophy of science.  In particular, there is a (partial) standoff
between experimental evaluation in the Cranfield/TREC tradition, which
is powerful but of limited scope, and observational studies with real
users, which are realistic but of limited scale.
%
IR experimentation has focused almost entirely on evaluating systems
with respect to predicting relevance.  Without denying the importance
of this, it is a rather limited view, and scientifically we should aim
at understanding the underlying phenomena.  Instead of systems we may
endeavor testing other hypotheses, even complete models or theories,
and against a broader range of useful predictions than relevance.
Think of redundancy/novelty/diversity, optimal thresholds,
satisfaction, clicks, satisfactory or unsatisfactory search
termination, query modification, etc.  The resulting richer models
should have something to say about both lab experiments and
observational studies.

% COMMENT
We have no grand theory of IR, not even the beginning of it.  Hence
there is a sense of urgency in Stephen's call for research aiming at
broadening our understanding the phenomena in IR.  But this will be a
long road ahead...

\medskip


% SUE
Susan Dumais's keynote was titled ``Evaluating IR \emph{in situ}''
\citep{2}. Sue addressed the limitations of current benchmark test
collections to address the scale, diversity, and interaction that
characterize information systems today.  Fortunately, there are also
many ways of collecting user data today, ranging from small-scale user
studies and user panels to log analysis and experimentation in the
large.  There is a need for sharable resources, such as the
infrastructure and instruments for capturing user activity, or user
interaction data that can be shared by the community.  An example is
the search logs obtained from the \citet{lemur}.
% \footnote{Available at \url{http://lemurstudy.cs.umass.edu/}}
An attractive proposal is to set up an operational system by the
community \citep{kell:eval09}.  Such a ``living laboratory'' would not
only allow for generating logs but also for conducting controlled
experiments with novel search or user interface components.

% Comment
Nowadays, there are unprecedented ways of capturing user interaction
data.  Sue's ``living laboratory'' proposal has a number of advantages
that may prove to be invaluable to the field.  First, it can greatly
facilitate user-centered research in IR, by allowing comparative
evaluation across systems and over time.
%
Second, the \emph{in situ} testing of retrieval components holds the
promise to build powerful connections between the user-centered
research and the system-centered research---Stephen's apparent
standoff discussed above.

\medskip

% CHRIS
Chris Buckley's keynote was titled ``Towards good evaluation of
individual topics'' \citep{1}.  Test collections are developed to
fairly compare systems, and need to average over many topics.  Chris
analyzed to what extent the resulting score informs us about the
per-topic performance of a system.  As it turn out the system-ranking
on a single topic correlates poorly (0.25-0.50) with the overall
system-ranking.  The correlation per measure corresponds to the amount
of information used: precision at 5 does worst, and recall at 1,000
does best.  In fact, measures do not agree more with their own overall
average than they agree with the other overall measures.
% : the per-topic recall at 1,000 also beter predicts all other measures.  
Better per topic evaluation would require richer evaluation
information---such as multi-level relevance judgments, partial
preference orderings, or multi-user judgments---and novel measures.

% Comment
The performance at individual topics is of obvious importance, not
only to improve our IR measures but also to better understand the
effect of topics on performance.  After all the user's experience
is directly tied to the results for his or her topic.

\medskip

% GEORGES
Georges Dupret's keynote was titled ``User models \& metrics''
\citep{3}.  Georges argued that all metrics make (often implicitly)
assumptions on user behavior, which can also be evaluated against
observations in search logs.  If a model better predicts the user
behavior on unseen data, then it is arguably more realistic, and hence
supports the associated metric.  There are two broad families of user
models underlying IR metrics.  On the one hand, there are effort-based
models that fix a certain effort and measure the utility in terms of
relevance retrieved.  Examples of associated metrics are DCG and other
rank-based metrics.  On the other hand, there are utility-based models
that fix a certain utility and measure the effort needed to obtain it.
Examples of associated measures are MAP and other recall-based
metrics.  Predicting user behavior in search sessions requires user
models that neither fix effort and utility, but combine both in a
single model.

% Comment 
Georges's focus on the comparison of user models rather than metrics
is very attractive, since the fidelity of the user models can be
established independently.
%
In fact, such formal user models open up a whole new line of research
that naturally incorporate dynamic, interactive aspects of information
seeking behavior.

\section{Posters}
\label{sec:pos}

The program continued with boasters and posters session with twenty
accepted papers, roughly falling in four themes.  We will discuss the
themes and papers in turn.

\subsection{Human in the loop}

There was a group of papers addressing user centered IR evaluation.
%
\citet{5} discuss their experience with side-by-side comparison tools
for information retrieval evaluation.  They also introduce a file
format for test files specifying salient features of the queries, the
result documents and the ranking.  Experiments can use these test
files to evaluate and tune enterprise search systems to maximize the
actual user satisfaction.

\citet{7} provide a methodological view to the problem of evaluating
interactive IR. The main idea is centered on breaking down the
information seeking episode into a sequence of interactions, each with
a sequence of information seeking strategies, and propose to evaluate
each interaction as well as the overall episode in term of ``usefulness.''

\citet{9} propose a holistic model of evaluation that considers
all participants or stakeholders and both costs and benefits. The four
main participant roles are information seeker, information provider,
information intermediaries, and system providers.  For each of these
the respective costs and benefits are listed.

\citet{11} proposes a community effort (as a TREC track or elsewhere)
to collect large amounts of shared user-interaction data, with the
goal of enabling researchers to develop models that predict human
performance rather than one of the more traditional offline measures
such as precision/recall.  

\citet{13} describe research aimed at understanding user satisfaction
for queries that do not receive any clicks.  In a user study, test
persons were asked to complete a short questionnaire about each of
their searches for one day, and they report on the fraction of queries
without clicks as well as the underlying intentions.

\subsection{Social data and social evaluation}

There was a group of papers addressing social evaluation and
evaluating social data search.
%
\citet{15} compare the judgments of TREC assessors to the judgments
of Mechanical Turk'ers on a set of TREC qrels. The results are
promising and show that employing turkers might be a viable solution
worth further study.

\citet{17} propose to evaluate search and recommendation methods in
social tagging networks using community-based relevance judging.  The
search requests would consist of both a topic and a user, and the 
judging would be relative to the individual user's position in the
network. 

\citet{19} propose a virtual evaluation forum for evaluating
cross-language link discovery.  This task can be evaluated without
human judgments, when pages with known links are withhold from the
collection and later used as topics.  This will also obviate
traditional cycles and allow for continual evaluation where runs can
be submitted and evaluated at any time against the withheld data.
Additional, richer, human assessments can be collected to give fuller
evaluations of submitted runs.

\citet{21} analyze crowdsourcing as a means to obtain the ground truth
needed for IR evaluation, based on the experiences during the INEX
2008 Book Track's game model. The diverse backgrounds of the
assessors, and the incentives of the crowdsourcing models may
influence the trustworthiness and quality of the resulting data.
Several indicators of trustworthiness are discussed, such as
familiarity with the topic and content, dwell time and changes in
dwell time, and agreement amongst judges.

\citet{23} describe the development and properties of a social media
test collection, containing a large number of academic bibliographic
records with accompanying annotations, plus topics and queries
generated and judged by a small set of experts.  Experiments on the
utility of the annotations for personalized search show promising
results.

\subsection{Improving Cranfield}

A group of papers discussed a range of issues within the current
evaluation methodology.
%
\citet{25} tabulated effectiveness claims in papers published over the
last decade.  Whilst many of these papers report significant
improvements over a baseline, there is no evidence of an overall gain
in absolute retrieval effectiveness: rarely systems outperform the
best scores obtained at the original TREC conference.  In order to
demonstrate verifiable improvements, there is a need for reporting
practices that allow for rigorous comparison with prior results.  This
can be facilitated with a common place where all relevant
effectiveness results are brought together, \citet{evaluatir}.
% \footnote{Available at \url{http://evaluatir.org/}.}

\citet{27} looks at evaluation using risk-reward curves, addressing
not only the average performance but also the stability or variance of
the performance.  The risk-reward trade-off is used to study the
impact of query expansion, which is known to be risky technique.
There are several ways in which the risk or variance can be
quantified, and we can apply concepts from economics as reasonable
starting points.

\citet{29} propose ``component-level'' evaluation by splitting
various processing steps (as required for many IR tasks) into separate
components, and to evaluate the impact of each processing step.  This
will allow for evaluating individual components, and for studying
component interactions.  Experiments are ongoing in the Grid@CLEF
track at CLEF 2008.

\citet{31} develop an IR test collection of ambiguous queries based
on Wikipedia disambiguation pages.  Queries correspond to
disambiguation pages, and the query's ambiguity is measured using
average cosine similarity between pairs of disambiguated pages.  A
preliminary test collection is build by submitted queries to
commercial search engines, and then manually judging the results with
respect to the different interpretations. 

\citet{33} explore how often quite different measures will actually
produce different comparative results across a wide range of possible
rankings.  When comparing over millions of pairs of TREC runs, the
metrics are shown to result in very similar comparative results.

\subsection{New domains and tasks}
The final group of papers discussed new tasks or aspects to evaluate.
%
\citet{35} discuss a tree-based view of search results, arguing that
this may be more appropriate in some cases where the results list has
a complex grouped or hierarchical organization. As a result, we cannot
use a ranked-list-of-results-type of evaluation for this scenario.
Evaluation may be based on search and click data in transaction logs.

\citet{37} propose a new evaluation track focused on web archives.
This extends earlier web search test collections with a temporal
dimension.  Web archives have many versions of the same pages that may
on the hand help locate relevance URLs, and on the other hand creates
the problem of which version(s) of the page to present to a user.

\citet{39} propose building an artificial test-collection to conduct
research into desktop search, a domain where real data is difficult to
obtain due to privacy concerns.  For people mentioned in the TREC
Enterprise Track's W3C collection, a thousand documents with a variety
of document types are gathered from the Web.  Artificial known-item
topics were generated from individual document.

\citet{41} propose to take the temporal aspect of recommendation
systems explicitly into account.  The proposal is a test collection
that treats the training data as a stream and use at any given time
point the earlier historical as the only input.  This captures the
effects of new users joining the system, and providing more input
ratings.

\citet{43} contend that efficiency is a too-often ignored component
of QA system evaluation, and describes and demonstrates an approach
for incorporating answer-time using the real-time QA experiment at CLEF
2006.  A straightforward but naive method of a linear combination of
effectiveness and efficiency is used.

\section{Breakout session}
\label{sec:bre}

Next, the workshop participants were divided into four breakout groups
that discussed the four themes:
\begin{itemize}
\item \emph{Human in the Loop} (chaired by {Paul Thomas});
\item \emph{Social Data and Evaluation} (chaired by {Ralf Schenkel});
\item \emph{Improving Cranfield} (chaired by {Justin Zobel}); and 
\item \emph{New Domains and Tasks} (chaired by {Mariano Consens}).
\end{itemize}
This was the most exciting part of the day, but impossible to
summarize.  
%
\begin{figure}[!tbp]
  \centering
  \resizebox{0.6\textwidth}{!}{\includegraphics{Sakai-twitter}}
  \caption{Live report from the ``Beyond Cranfield'' breakout group (tweets by Dr.\ T.\ Sakai).}
  \label{fig:sakai}
\end{figure}
% 
Figure~\ref{fig:sakai} give an impression of one of the four breakout
groups. Fortunately, each of the groups reported on their discussion
in the final closing panel, which we will discuss now.

\section{Closing Panel}
\label{sec:pan}

The program continued with the report out of the four breakout groups,
and the reactions of a panel consisting of: %
Charlie Clarke, %
David Evans, %
Donna Harman, and %
Diane Kelly. %

\subsection{Human in the loop}

The breakout group addressed the problem that that the traditional
``library consultation'' user model is breaking down, and that we want
to evaluate our systems with respect to better ``user models.''
However, we don't really know what such a model would look like, and
even if we did, we wouldn't know whether it is any good. 
% 
The proposal, building on the earlier keynote presentations, is to
evaluate user models, not systems, by their ability to predict
observable user behavior.  The input would be the information needs,
details of documents and rankings, etc.  The output would be a
prediction of user behavior. It is not clear a priori what we should
model exactly, user 'satisfaction' or more directly observable
behavior.
% 
We may be able to use experimental data for this in the form of
(extended) test collections.
% 
A more ambitious approach would be a ``living lab'' collecting both
interaction data and explicit (self-reported) satisfaction, hence
providing some grounded data and the whole interaction path.
% 
% Collaborate with those having data for validation
This should lead to a spiral development where we develop initial
models, and evaluate and refine them, etc.

\medskip

The panel reacted positively to the focus on users and user models,
however there was some concern that this could actually result in
replacing real users with a formal model---in fact getting rid of
users altogether.

% Reactions: \textsf{Dianne}: Happy about user-focus but wouldn't this
% take the user out of the loop? % \textsf{David}: XXX.

\subsection{Social data and social evaluation}

The breakout groups addressed the problem of getting annotations and
assessments in novel ways, in particular using the methodology of
crowdsourcing.  There is a need for richer assessments, without
leading to extravagant expenses, and preferable get them fast.
Crowdsourcing methods, such as offered through the Mechanical Turk,
offers the workforce needed for this.  There are still open questions
on the setup of the evaluation task, the quality and scale of the
resulting assessments, and the quality of the resulting rankings.  The
concrete proposal is to systematically compare crowdsourcing judgments
to those obtained from traditional assessors, and experiment with
various parameters such as the complexity of the assessment task, the
grouping or size of results, the number of judges per result, etc. 

\medskip

The panel liked the suggestion, which seems straightforward to set up,
but also expressed worries about the lack of control and potential
biases: who are these turkers? what population do they represent? and
what is their motivation to participate?

\subsection{Improving Cranfield}

The breakout group address substantial extensions to the current
Cranfield paradigm for system measurement.  The first part of the
proposal is to collect a rich ground truth corresponding to modern
information use.  This includes differences in user psychology,
underspecified queries, and closer user involvement.  This is a
substantial departure from the current ``plain'' relevance judgments
which are context-free, unannotated, etc.
% 
We should be open to new methods for gathering user data, e.g. from
the community, in an ongoing way. % , and evaluate over enough
% queries---more than now---with explicit treatment of ambiguity
% (temporal, spatial, lexical, referential).
% 
The second part of the proposal is to facilitate longitudinal
evaluation, with rich reporting and recording all runs.
%Independent availability of system components
This will help demonstrate that, of if, systems improve significantly
over time.  A fair comparison of results is achieved by comparing
against common baselines, and using unseen withheld relevance
judgments.

\medskip 

The panel stressed that comparing over time/users/tasks is crucial for
progress, but also expressed concerns whether the proposals were
radical enough.

\subsection{New domains and tasks}

The breakout group discussed more ``realistic'' evaluation scenarios.
The key idea is to study many different tasks, genres, and contexts
with direct relation to actual information access problems.  A broad
range of different tasks and scenario's was discussed.  For example,
think of an ``iPhone task'' giving access to a variety of different
sources.  Working on a multitude of tasks will allow us to validate IR
techniques across different scenario's, and corresponding user models,
aiding to our understanding of what works when, and why.

\medskip

The panel liked the interest in novel user models that go beyond the
``library consultation'' scenario, and stressed that information
access is more than search, and it is multi-modal, multi-lingual,
% multi-medial,
multi-cultural, etc. and we should study scenario's that do justice to
that.

\section{Trying to Summarize}
\label{sec:sum}

The setup of the workshop aimed to focus on concrete first steps for
the near future.  This failed miserably!  The majority of participants
wanted to discuss more fundamental aspects of IR evaluation.  This
gave a very exciting workshop with a lot of ``food for thought.''
Many of the suggestions have potentially far reaching consequences.
% Think of the per-topic evaluation, by exploring some of
% the context of the topic of request and its underlying information
% need we are potentially opening a ``can of worms''
This also makes it more difficult to summarize the outcome in concrete
lessons for the future of IR evaluation.  

There seems to be one direction that surfaced throughout the workshop.
% 
There is more to IR than the evaluation of systems and their rankings,
and important exceptions aside there is a general disconnect between
user-centered and system-centered research in IR.  Hence, we need to
find novel connections between both strands by broadening the scope of
system-centered research to provide richer context of the search
requests and the judgments, as well as to other useful predictions
than relevance.  And we have to match these with large-scale
user-centered research.  The time for this seems exactly right: there
are now powerful ways to gather user data.
%
Key element in this is the need of more explicit hypotheses and models
of the phenomena we are investigating.  We need new informal ``user
models'' underlying tasks, and formal models of information seeking
behavior that can be subjected to empirical testing.  That is, we need
to evaluate models of users/interaction directly.

% This suggestion is addressing the points of \citet[p.15]{spar:what06}:
% \begin{quote}%\small
% %
%   \sl TREC needs to engage, more positively and fully, with context and
%   the nature of the whole setup in\-for\-ma\-tion-seeking task T
%   rather than just the experimental task X. \ldots\ 
%   % 
%   I believe that it is important for TREC's system-building
%   participants to be encouraged to work forward from a fuller
%   knowledge of the context, rather than limiting their attention to
%   the attenuated form of the context that the D * Q * R environment
%   normally embodies, and recovering what may be distinctive about this
%   -- and hence somewhat indicative of significant features of the
%   larger context -- by working backwards from system results.
% \end{quote}
At the end of his presentation, Stephen Robertson recalled the
``revolution'' that the Cranfield work caused in IR, and wondered
another ``IR revolution'' may come.  We may have seen the seeds of
this second revolution being planted at the workshop...

\subsection*{Acknowledgments}
We would like to thank ACM and SIGIR for hosting this workshop, in
particular Jay Aslam and James Allan for their outstanding support in
the organization.  We would also like to thank the program committee,
consisting of %
Omar Alonso, %
Chris Buckley, %
Charles Clarke, %
Nick Craswell, %
Susan Dumais, %
Georges Dupret, %
Nicola Ferro, %
Norbert Fuhr, %
Donna Harman, %
David Hawking, %
Gareth Jones, %
Noriko Kando, %
Gabriella Kazai, %
Mounia Lalmas, %
Stefano Mizzaro, %
Iadh Ounis, %
Stephen Robertson, %
Ian Ruthven, %
%John Tait, %
Anastasios Tombros, %
Stephen Tomlinson, %
Justin Zobel, %
and the six program chairs.  Final thanks are due to the paper
authors, the panelists, and the participants for a great and lively
workshop.  The University of Amsterdam is hosting the workshop
proceedings and presentations, which are on-line at
\url{http://staff.science.uva.nl/~kamps/ireval/}.


% %\section*{Acknowledgments}
% Jaap Kamps was supported by the Netherlands Organization for
% Scientific Research (NWO, grants \#
% %612.\-066.\-302, % ITeQA (?)
% 612.\-066.\-513, % EfFoRT
% 639.\-072.\-601, and % README
% 640.\-001.\-501), % MuSeUM
% and by the E.U.'s 6th FP for RTD (project MultiMATCH contract
% IST-033104).
%

% Configure NATBIB.STY:
%\renewcommand{\bibsection}{\section*{REFERENCES}}
%%\providecommand{\bibfont}{\scriptsize}
%\providecommand{\bibfont}{\small}
%\setlength{\bibhang}{1em}
%\setlength{\bibsep}{1ex} %{\parsep} % Originally was \itemsep + \parsep (?)
%\bibliographystyle{abbrv}
%\bibliographystyle{splncs}
\bibliographystyle{abbrvnat}
\bibliography{forum09ireval} 



\end{document}
