A Challenging Data Set for Evaluating Part-of-speech Taggers
Paper in proceeding, 2024

We introduce a novel, challenging test set for part-of-speech (POS) tagging, consisting of sentences in which only one word is POS-tagged. First derived from Wiktionary, and then manually curated, it is intended as an out-of-sample test set for POS taggers trained over larger data sets. Sentences were selected such that at least one of four standard benchmark taggers would incorrectly tag the word under consideration for a given sentence, thus identifying challenging instances of POS tagging. Somewhat surprisingly, we find that the benchmark taggers often fail on rather straightforward instances of POS tagging, and we analyze these failures in some detail. We also compute the performance of a state-of-the-art DNN-based POS tagger over our set, obtaining an accuracy of around 0.87 for this out-of-sample test, far below its reported performance in the literature. Also for this tagger, we find instances of failure even in rather simple cases.

Natural Language Processing

Sequence Labeling

Part-of-speech Tagging

Author

Mattias Wahde

Chalmers, Mechanics and Maritime Sciences (M2), Vehicle Engineering and Autonomous Systems

Minerva Suvanto

Chalmers, Mechanics and Maritime Sciences (M2), Vehicle Engineering and Autonomous Systems

Marco L. Della Vedova

Chalmers, Mechanics and Maritime Sciences (M2), Vehicle Engineering and Autonomous Systems

International Conference on Agents and Artificial Intelligence

21843589 (ISSN) 2184433X (eISSN)

Vol. 2 79-86
978-989-758-680-4 (ISBN)

16th International Conference on Agents and Artificial Intelligence
Rome, Italy,

Subject Categories

Language Technology (Computational Linguistics)

Human Computer Interaction

Roots

Basic sciences

DOI

10.5220/0012307200003636

More information

Latest update

6/18/2024