A Challenging Data Set for Evaluating Part-of-speech Taggers
Paper i proceeding, 2024

We introduce a novel, challenging test set for part-of-speech (POS) tagging, consisting of sentences in which only one word is POS-tagged. First derived from Wiktionary, and then manually curated, it is intended as an out-of-sample test set for POS taggers trained over larger data sets. Sentences were selected such that at least one of four standard benchmark taggers would incorrectly tag the word under consideration for a given sentence, thus identifying challenging instances of POS tagging. Somewhat surprisingly, we find that the benchmark taggers often fail on rather straightforward instances of POS tagging, and we analyze these failures in some detail. We also compute the performance of a state-of-the-art DNN-based POS tagger over our set, obtaining an accuracy of around 0.87 for this out-of-sample test, far below its reported performance in the literature. Also for this tagger, we find instances of failure even in rather simple cases.

Natural Language Processing

Sequence Labeling

Part-of-speech Tagging

Författare

Mattias Wahde

Chalmers, Mekanik och maritima vetenskaper, Fordonsteknik och autonoma system

Minerva Suvanto

Chalmers, Mekanik och maritima vetenskaper, Fordonsteknik och autonoma system

Marco L. Della Vedova

Chalmers, Mekanik och maritima vetenskaper, Fordonsteknik och autonoma system

International Conference on Agents and Artificial Intelligence

21843589 (ISSN) 2184433X (eISSN)

Vol. 2 79-86
978-989-758-680-4 (ISBN)

16th International Conference on Agents and Artificial Intelligence
Rome, Italy,

Ämneskategorier

Språkteknologi (språkvetenskaplig databehandling)

Människa-datorinteraktion (interaktionsdesign)

Fundament

Grundläggande vetenskaper

DOI

10.5220/0012307200003636

Mer information

Senast uppdaterat

2024-06-18