Call for Participation - DESIRE 2011 Workshop at CIKM - Data infrastructurEs for Supporting Information Retrieval Evaluation

[Nicola Ferro <fe...@dei.unipd.it>]

** Apologies for cross postings **

--------------------------------------------------------------------------------------------------
Call for Participation

Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011) Workshop

Co-located with CIKM 2011, the 20th ACM Conference on Information and Knowledge Management

Friday 28th October 2011, Glasgow, UK

http://www.promise-noe.eu/events/desire-2011/


Early registration deadline: 31st August 2011
http://www.cikm2011.org/registration
--------------------------------------------------------------------------------------------------

AIM AND SCOPE

The objectives of the workshop are: to gather interested experts from the three areas of interest 
to CIKM - databases, information retrieval, and knowledge management - and discuss how to address 
the problem of envisaging and designing evaluation  infrastructures able to store, manage, and 
make accessible the scientific data and knowledge of interest for advancing the evaluation of 
information retrieval and access tools.

The main outcome will be to produce a roadmap and initial best practices to guide the development 
of the necessary evaluation infrastructures. 


KEYNOTE SPEAKERS

- An Infrastructure fo Supporting the Evaluation of Interactive Information Retrieval
 Norbert Fuhr, University of Duisburg-Essen, Germany.

- Ontology-based data management
 Maurizio Lenzerini, Sapienza University of Rome, Italy.


ACCEPTED PAPERS

- Evaluation with the VIRTUOSO platform
 Gérard Dupont, CASSIDIAN, Elancourt, France
 Gaël de Chalendar, CEA, Fontenay-aux-Roses, France 
 Khaled Kheliff, CASSIDIAN, Val de Reuil, France 
 Dmitri Voitsekhovitchy, CEA, Fontenay-aux-Roses, France 
 Géraud Canety, CEA, Fontenay-aux-Roses, France 

- A Lightweight Framework for Reproducible Parameter Sweeping in Information Retrieval
 Richard Eckart de Castilho, Ubiquitous Knowledge Processing (UKP) Lab, Technische Universität Darmstadt. Darmstadt, Germany <br/>
 Iryna Gurevych, Ubiquitous Knowledge Processing (UKP) Lab, Technische Universität Darmstadt. Darmstadt, Germany <br/><br/>

- Use Cases as a Component of Information Access Evaluation
 Jussi Karlgren, Swedish Institute of Computer Science, Sweden
 Anni Järvelin, Swedish Institute of Computer Science, Sweden
 Preben Hansen, Swedish Institute of Computer Science, Sweden
 Gunnar Eriksson, Swedish Institute of Computer Science, Sweden 

- PatOlympics - An Infrastructure for Interactive Evaluation of Patent Retrieval Tools
 Mihai Lupu, Vienna University of Technology, Austria

- Infrastructure and Workflow for the Formal Evaluation of Semantic Search Technologies
 Stuart N. Wrigley, Dept. Computer Science, University of Sheffield, UK
 Raúl García-Castro, Facultad de Informática Universidad Politécnica de Madrid, Spain
 Cássia Trojahn, INRIA & LIG, Montbonnot Saint Martin, France

- Principles for Robust Evaluation Infrastructure
 Justin Zobel, Department of Computer Science & Software Engineering, The University of Melbourne, Australia 
 William Webber, Department of Computer Science & Software Engineering, The University of Melbourne, Australia 
 Mark Sanderson, School of Computer Science & Information Technology, RMIT University, Australia
 Alistair Moffat, Department of Computer Science & Software Engineering, The University of Melbourne, Australia


IMPORTANT DATES

- Early Registration deadline: 31 August 2011
- Workshop: 28 October 2011


PROGRAM COMMITTEE

- Stefano Ceri, Politecnico di Milano, Italy
- Maarten de Rijke, University of Amsterdam, The Netherlands
- Donna Harman, National Institute of Standards and Technology (NIST), USA
- Noriko Kando, National Institute of Informatics (NII), Japan
- Henning Müller, University of Applied Sciences Western Switzerland
- Shila Ofek-Koifman, IBM Haifa, Israel
- Christos Papatheodorou, Ionian University, Greece
- Fausto Rabitti, ISTI-CNR, Italy
- Mark Sanderson, RMIT University, Australia
- Timos Sellis, National Technical University of Athens, Greece
- Gerhard Weikum, Max-Planck-Institut für Informatik, Germany

ORGANIZERS

- Maristella Agosti, University of Padua, Italy
- Nicola Ferro, University of Padua, Italy
- Costantino Thanos, ISTI-CNR, National Council of Research, Italy

CONTACT 

- Nicola Ferro, ferro@dei.unipd.It


FURTHER INFORMATION ABOUT THE WORKSHOP

The Information Retrieval area has a strong and long tradition dating back to the 1960s in producing 
and processing scientific data resulting from the experimental evaluation of search algorithms and 
search systems. This attitude towards evaluation has led to fast and continuous progress in the 
evolution of information retrieval systems and search engines. 

However, in order to make these data test collections understandable and usable they must be endowed 
with some auxiliary information, i.e., provenance, quality, context, etc. Therefore, there is a need 
for metadata models able to describe the main characteristics of evaluation data. In addition, in 
order to make distributed data collections accessible, sharable, and interoperable, there is a need 
for advanced data infrastructures.

Nevertheless, the information retrieval area has barely explored and exploited the possibilities for 
managing, storing, and effectively accessing the scientific data produced during the evaluation 
studies by making use of the methods typical of the database and knowledge management areas.

It is thus time for these three communities – information retrieval, databases, and 
knowledge management – to join efforts, meet, and cooperate to address the problem of envisaging and 
designing useful infrastructures able to coherently manage pertinent data collections and sources 
of information, and so take concrete steps towards developing them.

Indeed, the information retrieval experts need to recognise this need, while the database and 
knowledge management experts need to understand the problem and work together to solve it by using 
the methods and techniques specific to information management.

Therefore, the workshop will gather together experts from these three areas,  to encourage them to 
recognise the urgency of addressing the problem in an integrated and coherent way, and to coordinate 
efforts towards drawing a roadmap and suggesting best practices for an effective solution of the 
problem.

The goal of the workshop is to understand how to make use of the expertise of the three scientific 
areas in a cooperative way to avoid the duplication of efforts which may occur when addressing the 
problem separately in each specific area and to trigger synergies and joint actions on this issue. 

The workshop is intended to be a single full day workshop, consisting of: invited talks and paper 
presentations during the morning; active discussion groups during the afternoon, engaging the 
participants in the topics which have emerged in the previous sessions in order to define a roadmap 
and best practices.