Infrastructures for Living Labs - Project Phase II (STELLA II)
Abstract
The STELLA
project aims to create an evaluation infrastructure allowing retrieval and
recommender systems to be run within academic search systems with real users.
Thus, STELLA provides an integrated e-research environment, a so-called living
lab. The experimental setup differs significantly from classical offline TREC
studies as it allows academic researchers to use an evaluation method
previously reserved for industrial research or operators of large online
platforms. The STELLA framework is designed to support academic search
solutions, dealing with heterogeneity and mass of documents (and datasets) that
still introduce new challenges to the disciplines of information retrieval,
recommender systems, digital libraries, and related fields.
In STELLA
I, the three project partners, TH Köln, ZB MED, and GESIS, aimed to strengthen
the concept of user-centric living labs for the domain of academic search. By
allowing researchers to evaluate their retrieval approaches in two real-world
search portals, the methodological gap between real-world and lab-based evaluations
was effectively reduced. As a result, STELLA was the cornerstone of the CLEF
2021 evaluation lab LiLAS and proved to be reliable and extensible.
In STELLA
II, the three partners will extend the existing framework to include the
additional external partners DIPF, TIB, and ZBW and their corresponding search
platforms and evaluation use-cases. Furthermore, STELLA's current ad-hoc
retrieval and recommendation setup will be extended as a continuous evaluation
platform.