Overview of the CLEF-2021 CheckThat! Lab Task 1 on check-worthiness estimation in tweets and political debates

Shaden Shaar; Maram Hasanain; Bayan Hamdan; Zien Sheikh Ali; Fatima Haouari; Alex Nikolov; Mucahid Kutlu; Yavuz Selim Kartal; Firoj Alam; Giovanni da San Martino; Alberto Barrón-Cedeño; Rubén Míguez; Javier Beltrán; Tamer Elsayed; Preslav Nakov

Overview of the CLEF-2021 CheckThat! Lab Task 1 on check-worthiness estimation in tweets and political debates

Shaden Shaar^*
, Maram Hasanain
, Bayan Hamdan
, Zien Sheikh Ali
, Fatima Haouari
, Alex Nikolov
, Mucahid Kutlu
, Yavuz Selim Kartal
, Firoj Alam
, Giovanni da San Martino
, Alberto Barrón-Cedeño
, Rubén Míguez
, Javier Beltrán
, Tamer Elsayed
, Preslav Nakov

^*Corresponding author for this work

Qatar Computing Research Institute

Hamad bin Khalifa University
Qatar University
Sofia University St. Kliment Ohridski
TOBB University of Economics and Technology
University of Padua
University of Bologna
Newtral Media Audiovisual

Research output: Contribution to journal › Conference article › peer-review

17 Citations (Scopus)

Abstract

We present an overview of Task 1 of the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The task asks to predict which posts in a Twitter stream are worth fact-checking, focusing on COVID-19 and politics in five languages: Arabic, Bulgarian, English, Spanish, and Turkish. A total of 15 teams participated in this task and most submissions managed to achieve sizable improvements over the baselines using Transformer-based models such as BERT and RoBERTa. Here, we describe the process of data collection and the task setup, including the evaluation measures, and we give a brief overview of the participating systems. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in check-worthiness estimation for tweets and political debates.

Original language	English
Pages (from-to)	369-392
Number of pages	24
Journal	CEUR Workshop Proceedings
Volume	2936
Publication status	Published - 2021
Event	22nd Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 - Virtual, Online, Romania Duration: 21 Sept 2021 → 24 Sept 2021

Keywords

COVID-19
Check-worthiness estimation
Computational journalism
Detecting previously fact-checked claims
Fact-checking
Social media verification
Veracity
Verified claims retrieval

Cite this

@article{c99761f936674b50a6ca91fd0ed71d18,

title = "Overview of the CLEF-2021 CheckThat! Lab Task 1 on check-worthiness estimation in tweets and political debates",

abstract = "We present an overview of Task 1 of the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The task asks to predict which posts in a Twitter stream are worth fact-checking, focusing on COVID-19 and politics in five languages: Arabic, Bulgarian, English, Spanish, and Turkish. A total of 15 teams participated in this task and most submissions managed to achieve sizable improvements over the baselines using Transformer-based models such as BERT and RoBERTa. Here, we describe the process of data collection and the task setup, including the evaluation measures, and we give a brief overview of the participating systems. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in check-worthiness estimation for tweets and political debates.",

keywords = "COVID-19, Check-worthiness estimation, Computational journalism, Detecting previously fact-checked claims, Fact-checking, Social media verification, Veracity, Verified claims retrieval",

author = "Shaden Shaar and Maram Hasanain and Bayan Hamdan and Ali, \{Zien Sheikh\} and Fatima Haouari and Alex Nikolov and Mucahid Kutlu and Kartal, \{Yavuz Selim\} and Firoj Alam and \{da San Martino\}, Giovanni and Alberto Barr{\'o}n-Cede{\~n}o and Rub{\'e}n M{\'i}guez and Javier Beltr{\'a}n and Tamer Elsayed and Preslav Nakov",

note = "Publisher Copyright: {\textcopyright} 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).; 22nd Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 ; Conference date: 21-09-2021 Through 24-09-2021",

year = "2021",

language = "English",

volume = "2936",

pages = "369--392",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

publisher = "CEUR-WS",

}

TY - JOUR

T1 - Overview of the CLEF-2021 CheckThat! Lab Task 1 on check-worthiness estimation in tweets and political debates

AU - Shaar, Shaden

AU - Hasanain, Maram

AU - Hamdan, Bayan

AU - Ali, Zien Sheikh

AU - Haouari, Fatima

AU - Nikolov, Alex

AU - Kutlu, Mucahid

AU - Kartal, Yavuz Selim

AU - Alam, Firoj

AU - da San Martino, Giovanni

AU - Barrón-Cedeño, Alberto

AU - Míguez, Rubén

AU - Beltrán, Javier

AU - Elsayed, Tamer

AU - Nakov, Preslav

PY - 2021

Y1 - 2021

N2 - We present an overview of Task 1 of the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The task asks to predict which posts in a Twitter stream are worth fact-checking, focusing on COVID-19 and politics in five languages: Arabic, Bulgarian, English, Spanish, and Turkish. A total of 15 teams participated in this task and most submissions managed to achieve sizable improvements over the baselines using Transformer-based models such as BERT and RoBERTa. Here, we describe the process of data collection and the task setup, including the evaluation measures, and we give a brief overview of the participating systems. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in check-worthiness estimation for tweets and political debates.

AB - We present an overview of Task 1 of the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The task asks to predict which posts in a Twitter stream are worth fact-checking, focusing on COVID-19 and politics in five languages: Arabic, Bulgarian, English, Spanish, and Turkish. A total of 15 teams participated in this task and most submissions managed to achieve sizable improvements over the baselines using Transformer-based models such as BERT and RoBERTa. Here, we describe the process of data collection and the task setup, including the evaluation measures, and we give a brief overview of the participating systems. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in check-worthiness estimation for tweets and political debates.

KW - COVID-19

KW - Check-worthiness estimation

KW - Computational journalism

KW - Detecting previously fact-checked claims

KW - Fact-checking

KW - Social media verification

KW - Veracity

KW - Verified claims retrieval

UR - https://www.scopus.com/pages/publications/85113440372

M3 - Conference article

AN - SCOPUS:85113440372

SN - 1613-0073

VL - 2936

SP - 369

EP - 392

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

T2 - 22nd Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021

Y2 - 21 September 2021 through 24 September 2021

ER -

Overview of the CLEF-2021 CheckThat! Lab Task 1 on check-worthiness estimation in tweets and political debates

Abstract

Keywords

Other files and links

Fingerprint

Cite this