D 2023

Resources and Few-shot Learners for In-context Learning in Slavic Languages

ŠTEFÁNIK, Michal; Marek KADLČÍK; Piotr GRAMACKI and Petr SOJKA

Basic information

Original name

Resources and Few-shot Learners for In-context Learning in Slavic Languages

Authors

ŠTEFÁNIK, Michal (703 Slovakia, guarantor, belonging to the institution); Marek KADLČÍK (203 Czech Republic, belonging to the institution); Piotr GRAMACKI (616 Poland) and Petr SOJKA (203 Czech Republic, belonging to the institution)

Edition

Dubrovnik, Croatia, Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), p. 94-105, 12 pp. 2023

Publisher

Association for Computational Linguistics

Other information

Language

English

Type of outcome

Proceedings paper

Country of publisher

United States of America

Confidentiality degree

is not subject to a state or trade secret

Publication form

electronic version available online

References:

RIV identification code

RIV/00216224:14330/23:00130911

Organization

Fakulta informatiky – Repository – Repository

ISBN

978-1-959429-57-9

EID Scopus

2-s2.0-85161966608

Keywords in English

in-context learning; low-resource; Slavic languages; datasets

Links

MUNI/A/1339/2022, interní kód Repo. Czech-BioImaging II, large research infrastructures.
Changed: 27/4/2024 04:19, RNDr. Daniel Jakubík

Abstract

V originále

Despite the rapid recent progress in creating accurate and compact in-context learners, most recent work focuses on in-context learning (ICL) for tasks in English. However, the ability to interact with users of languages outside English presents a great potential for broadening the applicability of language technologies to non-English speakers. In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages. Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models that we train on the collected resources and compare their performance to previous work. We find that ICL models tuned in English are also able to learn some tasks from non-English contexts, but multilingual instruction fine-tuning consistently improves the ICL ability. We also find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.

Files attached