Kevin Roitero
Kevin Roitero
Ph.D. Candidate
University of Udine

Education & Experience

Current Position

November 2016 - Today

PhD Candidate.

University of Udine, Udine (UD), Italy.


Experience

September 2018 - December 2018

Research Intern at Spotify.

Spotify, London.

Manager: Mounia Lalmas-Roelleke (Director, Tech Research).


Education

November 2016 - Today

Ph.D.

Department of Mathematics, Computer Science, and Physics. University of Udine.

Title of the research project: Economical Evaluation of IR systems (Supervisor Prof. Stefano Mizzaro).
Topics of interest: Information retrieval, crowd-sourcing, machine and transfer learning, data mining and analysis, and statistical modeling.

July 2018 (one week)

Short Visiting period at University of Queensland (UQ).

School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, Queensland, Australia.

Visit Gianluca Demartini.

March 2018 - July 2018

Visiting period at RMIT University.

ISAR Group, RMIT University, Melbourne, Victoria, Australia.

Visit J. Shane Culpepper, Mark Sanderson, and Falk Scholer.

April 2017 - June 2017

Visiting period at University Of Sheffield.

Information School, University Of Sheffield, United Kingdom.

Visit Gianluca Demartini, at Information School, University Of Sheffield, United Kingdom.

2014 - 2016

Master Joint Degree in International Computer Science.

University of Udine, Udine (UD), Italy,
Alpen-Adria-Universität, Klagenfurt, Austria. (6 months).

Title of the thesis: Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering and Dimensionality Reduction (Supervisor Prof. Stefano Mizzaro; Co-Supervisor Prof. Klaus Schoeffmann).

September 2015 - February 2016

Visiting period at Alpen-Adria-Universität.

Alpen-Adria-Universität, Klagenfurt, Austria.

2011 - 2014

Bachelor Degree in Computer Science.

University of Udine, Udine (UD), Italy.

Title of the thesis: Una Base di Dati per la Condivisione di Film e Serie Televisive - A database for sharing movies and tv-series informations (Supervisor Prof. Angelo Montanari).

Publications

2020

  • Effectiveness Evaluation Without Human Relevance Judgments: A Systematic Analysis of Existing Methods and of their Combinations.

    Kevin Roitero, Andrea Brunello, Giuseppe Serra, and Stefano Mizzaro.

    Infomration Processing and Management (IPM).
    Url: https://www.sciencedirect.com/science/article/pii/S030645731930192X.

    Journal Paper.

  • Crowd Worker Strategies in Relevance Judgment Tasks.

    Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadiraju, Kevin Roitero, and Gianluca Demartini.

    Proceedings of the 13th ACM International Conference on Web Search and Data Mining, WSDM 2020, Houston, Texas. February 3-7, 2020.
    Url: .

    Conference Paper.

2019

  • The Impact of Task Abandonment in Crowdsourcing.

    Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini.

    IEEE Transactions on Knowledge and Data Engineering (TKDE).
    Url: https://ieeexplore.ieee.org/abstract/document/8873609/.

    Journal Paper.

  • Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models.

    Michael Soprano, Kevin Roitero, and Stefano Mizzaro.

    Proceedings of the 10th Italian Information Retrieval Workshop, Padua, Italy, September 16-18, 2019.
    Url: http://ceur-ws.org/Vol-2441/paper4.pdf.

    Conference Paper.

  • Towards Stochastic Simulations of Relevance Profiles.

    Kevin Roitero, Andrea Brunello, Julián Urbano and Stefano Mizzaro.

    Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM2019). Beijing, China. November 3-7, 2019.
    Url: https://dl.acm.org/citation.cfm?id=3358123.

    Conference Paper.

  • On Transforming Relevance Scales.

    Lei Han, Kevin Roitero, Eddy Maddalena, Stefano Mizzaro and Gianluca Demartini.

    Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM2019). Beijing, China. November 3-7, 2019.
    Url: https://dl.acm.org/citation.cfm?id=3357988.

    Conference Paper.

  • HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis.

    Michael Soprano, Kevin Roitero and Stefano Mizzaro.

    Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019).
    Url: http://ceur-ws.org/Vol-2414/paper7.pdf.

    Conference Paper.

  • Fewer Topics? A Million Topics? Both?! On Topics Subsets in Test Collections.

    Kevin Roitero, J. Shane Culpepper, Mark Sanderson, Falk Scholer, and Stefano Mizzaro

    Information Retrieval Journal (IRJ).
    Url: https://rdcu.be/bFuZq.

    Journal Paper.

  • On Topic Difficulty in IR Evaluation: The Effect of Corpora, Systems, and System Components.

    Fabio Zampieri, Kevin Roitero, Shane Culpepper, Oren Kurland, and Stefano Mizzaro.

    Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Paris, France. July 21-25, 2019.
    Url: https://dl.acm.org/citation.cfm?id=3331279.

    Conference Paper.

  • All Those Wasted Hours: On Task Abandonment in Crowdsourcing.

    Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena and Gianluca Demartini.

    ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, Australia.
    Url: https://dl.acm.org/citation.cfm?id=3291035.

    Conference Paper.

2018

  • How Many Truth Levels? Six? One Hundred? Even more? Validating Truthfulness of Statements via Crowdsourcing.

    Kevin Roitero, Damiano Spina, Gianluca Demartini, and Stefano Mizzaro.

    RDSM 2018 : Workshop on Rumours and Deception in Social Media.
    Url: https://www.damianospina.com/wp-content/uploads/2018/08/roitero2018how.pdf.

    Conference Paper.

  • Reproduce. Generalize. Extend. On Information Retrieval Evaluation without Relevance Judgments.

    Kevin Roitero, Marco Passon, Giuseppe Serra, and Stefano Mizzaro.

    ACM Journal of Data and Information Quality (JDIQ).
    Url: https://dl.acm.org/citation.cfm?id=3241064.

    Journal Paper.

  • Reproduce and Improve: An Evolutionary Approach to Select a Few Good Topics for Information Retrieval Evaluation.

    Kevin Roitero, Michael Soprano, Andrea Brunello, and Stefano Mizzaro.

    ACM Journal of Data and Information Quality (JDIQ).
    Url: https://dl.acm.org/citation.cfm?id=3239573.

    Journal Paper.

  • On Fine-Grained Relevance Scales.

    Kevin Roitero, Eddy Maddalena, Gianluca Demartini and Stefano Mizzaro.

    Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor Michigan, U.S.A. July 8-12, 2018.
    Url: https://dl.acm.org/citation.cfm?id=3210052.

    Conference Paper.

  • Query Performance Prediction and Effectiveness Evaluation Without Relevance Judgments: Two Sides of the Same Coin.

    Stefano Mizzaro, Josiane Mothe, Kevin Roitero, and Md Zia Ullah.

    Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor Michigan, U.S.A. July 8-12, 2018.
    Url: https://dl.acm.org/citation.cfm?id=3210146.

    Conference Paper.

  • IRevalOO: An Object Oriented Framework for Retrieval Evaluation.

    Kevin Roitero, Eddy Maddalena, Ponte Yannick and Stefano Mizzaro.

    Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor Michigan, U.S.A. July 8-12, 2018.
    Url: https://dl.acm.org/citation.cfm?id=3210084.

    Conference Paper.

  • Effectiveness Evaluation with a Subset of Topics: A Practical Approach.

    Kevin Roitero, Michael Soprano and Stefano Mizzaro.

    Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor Michigan, U.S.A. July 8-12, 2018.
    Url: https://dl.acm.org/citation.cfm?id=3210108.

    Conference Paper.

  • CHeap and Engineered Evaluation of Retrieval Systems.

    Kevin Roitero.

    Doctoral Consortium of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor Michigan, U.S.A. July 8-12, 2018.
    Url: https://dl.acm.org/citation.cfm?id=3210229.

    Conference Paper.

2017

  • Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing.

    Alessandro Checco, Kevin Roitero, Eddy Maddalena, Gianluca Demartini and Stefano Mizzaro.

    Proceedings of the The fifth AAAI Conference on Human Computation and Crowdsourcing, AAAI HCOMP2017. Quebec City, Canada. October 24-26 2017.
    Url: https://aaai.org/ocs/index.php/HCOMP/HCOMP17/paper/viewFile/15927/15258.

    Conference Paper.

  • Considering Assessor Agreement in IR Evaluation.

    Eddy Maddalena, Kevin Roitero, Gianluca Demartini and Stefano Mizzaro.

    Proceedings of the 3rd ACM International Conference on the Theory of Information Retrieval, ICTIR2017. Amsterdam, Netherlands. October 1-4 2017.
    Url: https://dl.acm.org/citation.cfm?id=3121060.

    Conference Paper.

  • Economic Evaluation of Recommender Systems: A Proposal.

    Kevin Roitero, Stefano Mizzaro, and Giuseppe Serra.

    Proceedings of the 8th Italian Information Retrieval Workshop, Switzerland. June 05-07, 2017.
    Url: http://ceur-ws.org/Vol-1911/8.pdf.

    Conference Paper.

  • Do Easy Topics Predict Effectiveness Better Than Difficult Topics?

    Kevin Roitero, Eddy Maddalena, and Stefano Mizzaro.

    Proceedings of the 39th European Conference on IR Research, ECIR 2017, Aberdeen, UK, April 8-13, 2017.
    pages: 605--611, isbn: 978-3-319-56608-5, doi: 10.1007/978-3-319-56608-5_55.
    Url: http://dx.doi.org/10.1007/978-3-319-56608-5_55.

    Conference Paper.

2016

Talks

  • 14 November 2019

    Magnitude Estimation for Relevance Assessment (and Other Applications).

    European Network for Combining Language Learning with Crowdsourcing Techniques, enetCollect. Sliema, Malta.

  • 25 July 2019

    HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis.

    ACM SIGIR Conference on Research and Development in Information Retrieval. Paris, France.

  • 7 June 2019

    Economical Evaluation of Systems: Users as a Signal.

    Spotify NYC.

  • 15 March 2019

    Let’s Agree to Disagree: Fixing Agreement Measures for Crowdsourcing.

    European Network for Combining Language Learning with Crowdsourcing Techniques, enetCollect. Lisbon.

  • 14 November 2018

    Internal Research Talk.

    Spotify. London, Great Britain.

  • 11 July 2018

    On fine Grained Relevance Scales.

    ACM SIGIR Conference on Research and Development in Information Retrieval. Ann Arbor Michigan, U.S.A.

  • 17 May 2018

    iPython notebooks for data analysis and plotting.

    RMIT University, Melbourne, Australia.

  • 10 May 2018

    Crowdsourcing relevance judgments and considering agreement in IR evaluation (with a new agreement measure)

    RMIT University, Melbourne, Australia.

  • 6 June 2017

    Economic Evaluation of Recommender Systems: A Proposal.

    IIR2017, Lugano. Switzerland.

  • 10 April 2017

    Do Easy Topics Predict Effectiveness Better Than Difficult Topics?

    ECIR2017, Aberdeen. UK.

  • 30 May 2016

    Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering?

    IIR 2016, Venezia. Italy.

  • 12 May 2016

    Improving the Efficiency of Retrieval Effectiveness Evaluation: Finding a Few Good Topics with Clustering and Dimensionality Reduction.

    MSc Course (Web Information Retrieval). University of Udine, Udine. Italy.

Honors, Awards, and Grants

  • Research Grant with title ``Crowdsourcing an Online Safety Benchmark for Fake News Identification''.

    Provided by Facebook Research. Call title: ``The Online Safety Benchmark''. Role: Participant.

  • Student Grant to attend ``ASIRF2019'', Schloss Dagstuhl, Germany.

    Provided by ASIRF.

  • Student Grant to attend ``SIGIR2019'', Paris, France.

    Provided by ACM SIGIR.

  • Participation in the 7th Heidelberg Laureate Forum.

    Provided by Heidelberg Laureate Forum Foundation (HLFF).

  • Invited Talk to European Network for Combining Language Learning with Crowdsourcing Techniques 2019.

    Provided by enetCollect.

  • Student Grant to attend ``SIGIR2018'', Ann Arbor, Michigan, U.S.A.

    Provided by ACM SIGIR.

  • Allowance to visit RMIT Information Storage, Analysis and Retrieval (ISAR) at RMIT University (the Royal Melbourne Institute of Technology), Melbourne, Victoria, Australia.

    Provided by RMIT Information Storage, Analysis and Retrieval (ISAR).

  • Student Grant to attend ``ICTIR2017'', Amsterdam, Netherlands.

    Provided by ACM SIGIR.

  • Allowance to visit Information School at University of Sheffield, United Kingdom.

    Provided by ERASMUS+ Traineeship Project.

  • Student Grant to attend ``ECIR2017'', Aberdeen, Scotland.

    Provided by BCS Information Retrieval Specialist Group (IRGS).

  • Allowance to visit Alpen-Adria-Universität Kalgenfurt, Klagenfurt, Austria.

    Provided by University of Udine, and ERAMUS Project.

Teaching

2019 - 2020

Modules of Social Computing Course.

Mascter Degree in Internet of Things, Big data and Web, University of Udine.

24 hours.

2016 - 2017

Operating Systems Lab.

Bachelor Degree in Computer Science, University of Udine.

24 hours.

Contacts

University of Udine
Dept. of Maths, Computer Science, and Physics.
Via delle Scienze 206
Udine (UD), Italy

Office: Rizzi Building, 2° floor, South Node, Room n° 3 (SN03)