Multi-dialect Arabic POS tagging: A CRF approach

Kareem Darwish; Hamdy Mubarak; Mohamed Eldesouki; Ahmed Abdelali; Younes Samih; Randah Alharbi; Mohammed Attia; Walid Magdy; Laura Kallmeyer

Multi-dialect Arabic POS tagging: A CRF approach

Kareem Darwish, Hamdy Mubarak, Mohamed Eldesouki, Ahmed Abdelali, Younes Samih, Randah Alharbi, Mohammed Attia, Walid Magdy, Laura Kallmeyer

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

34 Citations (Scopus)

Abstract

This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3%.

Original language	English
Title of host publication	LREC 2018 - 11th International Conference on Language Resources and Evaluation
Editors	Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, Takenobu Tokunaga
Publisher	European Language Resources Association (ELRA)
Pages	93-98
Number of pages	6
ISBN (Electronic)	9791095546009
Publication status	Published - 2018
Externally published	Yes
Event	11th International Conference on Language Resources and Evaluation, LREC 2018 - Miyazaki, Japan Duration: 7 May 2018 → 12 May 2018

Publication series

Name	LREC 2018 - 11th International Conference on Language Resources and Evaluation

Conference

Conference	11th International Conference on Language Resources and Evaluation, LREC 2018
Country/Territory	Japan
City	Miyazaki
Period	7/05/18 → 12/05/18

Keywords

Arabic dialects
CRF
POS tagging

Cite this

Darwish, K., Mubarak, H., Eldesouki, M., Abdelali, A., Samih, Y., Alharbi, R., Attia, M., Magdy, W., & Kallmeyer, L. (2018). Multi-dialect Arabic POS tagging: A CRF approach. In N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, & T. Tokunaga (Eds.), LREC 2018 - 11th International Conference on Language Resources and Evaluation (pp. 93-98). (LREC 2018 - 11th International Conference on Language Resources and Evaluation). European Language Resources Association (ELRA).

Darwish, Kareem ; Mubarak, Hamdy ; Eldesouki, Mohamed et al. / Multi-dialect Arabic POS tagging : A CRF approach. LREC 2018 - 11th International Conference on Language Resources and Evaluation. editor / Nicoletta Calzolari ; Khalid Choukri ; Christopher Cieri ; Thierry Declerck ; Sara Goggi ; Koiti Hasida ; Hitoshi Isahara ; Bente Maegaard ; Joseph Mariani ; Helene Mazo ; Asuncion Moreno ; Jan Odijk ; Stelios Piperidis ; Takenobu Tokunaga. European Language Resources Association (ELRA), 2018. pp. 93-98 (LREC 2018 - 11th International Conference on Language Resources and Evaluation).

@inproceedings{5d1f101b758146bfb2c84fa239fe7a83,

title = "Multi-dialect Arabic POS tagging: A CRF approach",

abstract = "This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3\%.",

keywords = "Arabic dialects, CRF, POS tagging",

author = "Kareem Darwish and Hamdy Mubarak and Mohamed Eldesouki and Ahmed Abdelali and Younes Samih and Randah Alharbi and Mohammed Attia and Walid Magdy and Laura Kallmeyer",

note = "Publisher Copyright: {\textcopyright} LREC 2018 - 11th International Conference on Language Resources and Evaluation. All rights reserved.; 11th International Conference on Language Resources and Evaluation, LREC 2018 ; Conference date: 07-05-2018 Through 12-05-2018",

year = "2018",

language = "English",

series = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",

publisher = "European Language Resources Association (ELRA)",

pages = "93--98",

editor = "Nicoletta Calzolari and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Koiti Hasida and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Asuncion Moreno and Jan Odijk and Stelios Piperidis and Takenobu Tokunaga",

booktitle = "LREC 2018 - 11th International Conference on Language Resources and Evaluation",

}

Darwish, K, Mubarak, H, Eldesouki, M, Abdelali, A, Samih, Y, Alharbi, R, Attia, M, Magdy, W & Kallmeyer, L 2018, Multi-dialect Arabic POS tagging: A CRF approach. in N Calzolari, K Choukri, C Cieri, T Declerck, S Goggi, K Hasida, H Isahara, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk, S Piperidis & T Tokunaga (eds), LREC 2018 - 11th International Conference on Language Resources and Evaluation. LREC 2018 - 11th International Conference on Language Resources and Evaluation, European Language Resources Association (ELRA), pp. 93-98, 11th International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, 7/05/18.

Multi-dialect Arabic POS tagging: A CRF approach. / Darwish, Kareem; Mubarak, Hamdy; Eldesouki, Mohamed et al.
LREC 2018 - 11th International Conference on Language Resources and Evaluation. ed. / Nicoletta Calzolari; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Koiti Hasida; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Asuncion Moreno; Jan Odijk; Stelios Piperidis; Takenobu Tokunaga. European Language Resources Association (ELRA), 2018. p. 93-98 (LREC 2018 - 11th International Conference on Language Resources and Evaluation).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Multi-dialect Arabic POS tagging

T2 - 11th International Conference on Language Resources and Evaluation, LREC 2018

AU - Darwish, Kareem

AU - Mubarak, Hamdy

AU - Eldesouki, Mohamed

AU - Abdelali, Ahmed

AU - Samih, Younes

AU - Alharbi, Randah

AU - Attia, Mohammed

AU - Magdy, Walid

AU - Kallmeyer, Laura

PY - 2018

Y1 - 2018

N2 - This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3%.

AB - This paper introduces a new dataset of POS-tagged Arabic tweets in four major dialects along with tagging guidelines. The data, which we are releasing publicly, includes tweets in Egyptian, Levantine, Gulf, and Maghrebi, with 350 tweets for each dialect with appropriate train/test/development splits for 5-fold cross validation. We use a Conditional Random Fields (CRF) sequence labeler to train POS taggers for each dialect and examine the effect of cross and joint dialect training, and give benchmark results for the datasets. Using clitic n-grams, clitic metatypes, and stem templates as features, we were able to train a joint model that can correctly tag four different dialects with an average accuracy of 89.3%.

KW - Arabic dialects

KW - CRF

KW - POS tagging

UR - https://www.scopus.com/pages/publications/85058117794

M3 - Conference contribution

AN - SCOPUS:85058117794

T3 - LREC 2018 - 11th International Conference on Language Resources and Evaluation

SP - 93

EP - 98

BT - LREC 2018 - 11th International Conference on Language Resources and Evaluation

A2 - Calzolari, Nicoletta

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Hasida, Koiti

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Moreno, Asuncion

A2 - Odijk, Jan

A2 - Piperidis, Stelios

A2 - Tokunaga, Takenobu

PB - European Language Resources Association (ELRA)

Y2 - 7 May 2018 through 12 May 2018

ER -

Darwish K, Mubarak H, Eldesouki M, Abdelali A, Samih Y, Alharbi R et al. Multi-dialect Arabic POS tagging: A CRF approach. In Calzolari N, Choukri K, Cieri C, Declerck T, Goggi S, Hasida K, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, Tokunaga T, editors, LREC 2018 - 11th International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA). 2018. p. 93-98. (LREC 2018 - 11th International Conference on Language Resources and Evaluation).

Multi-dialect Arabic POS tagging: A CRF approach

Abstract

Publication series

Conference

Keywords

Other files and links

Fingerprint

Cite this