2023
pdf
bib
abs
GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration
Aleksandra Piktus
|
Odunayo Ogundepo
|
Christopher Akiki
|
Akintunde Oladipo
|
Xinyu Zhang
|
Hailey Schoelkopf
|
Stella Biderman
|
Martin Potthast
|
Jimmy Lin
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)
Noticing the urgent need to provide tools for fast and user-friendly qualitative analysis of large-scale textual corpora of the modern NLP, we propose to turn to the mature and well-tested methods from the domain of Information Retrieval (IR) - a research field with a long history of tackling TB-scale document collections. We discuss how Pyserini - a widely used toolkit for reproducible IR research can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts. We leverage the existing functionalities of both platforms while proposing novel features further facilitating their integration. Our goal is to give NLP researchers tools that will allow them to develop retrieval-based instrumentation for their data analytics needs with ease and agility. We include a Jupyter Notebook-based walk through the core interoperability features, available on GitHub:
https://github.com/huggingface/gaia. We then demonstrate how the ideas we present can be operationalized to create a powerful tool for qualitative data analysis in NLP. We present GAIA Search - a search engine built following previously laid out principles, giving access to four popular large-scale text collections. GAIA serves a dual purpose of illustrating the potential of methodologies we discuss but also as a standalone qualitative analysis tool that can be leveraged by NLP researchers aiming to understand datasets prior to using them in training. GAIA is hosted live on Hugging Face Spaces:
https://huggingface.co/spaces/spacerini/gaia.
pdf
bib
abs
Evaluating Embedding APIs for Information Retrieval
Ehsan Kamalloo
|
Xinyu Zhang
|
Odunayo Ogundepo
|
Nandan Thakur
|
David Alfonso-hermelo
|
Mehdi Rezagholizadeh
|
Jimmy Lin
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)
The ever-increasing size of language models curtails their widespread access to the community, thereby galvanizing many companies and startups into offering access to large language models through APIs. One particular API, suitable for dense retrieval, is the semantic embedding API that builds vector representations of a given text. With a growing number of APIs at our disposal, in this paper, our goal is to analyze semantic embedding APIs in realistic retrieval scenarios in order to assist practitioners and researchers in finding suitable services according to their needs. Specifically, we wish to investigate the capabilities of existing APIs on domain generalization and multilingual retrieval. For this purpose, we evaluate the embedding APIs on two standard benchmarks, BEIR, and MIRACL. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective on English, in contrast to the standard practice, i.e., employing them as first-stage retrievers. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best albeit at a higher cost. We hope our work lays the groundwork for thoroughly evaluating APIs that are critical in search and more broadly, in information retrieval.
pdf
bib
abs
Cross-lingual Open-Retrieval Question Answering for African Languages
Odunayo Ogundepo
|
Tajuddeen Gwadabe
|
Clara Rivera
|
Jonathan Clark
|
Sebastian Ruder
|
David Adelani
|
Bonaventure Dossou
|
Abdou Diop
|
Claytone Sikasote
|
Gilles Hacheme
|
Happy Buzaaba
|
Ignatius Ezeani
|
Rooweither Mabuya
|
Salomey Osei
|
Chris Emezue
|
Albert Kahira
|
Shamsuddeen Muhammad
|
Akintunde Oladipo
|
Abraham Owodunni
|
Atnafu Tonja
|
Iyanuoluwa Shode
|
Akari Asai
|
Anuoluwapo Aremu
|
Ayodele Awokoya
|
Bernard Opoku
|
Chiamaka Chukwuneke
|
Christine Mwase
|
Clemencia Siro
|
Stephen Arthur
|
Tunde Ajayi
|
Verrah Otiende
|
Andre Rubungo
|
Boyd Sinkala
|
Daniel Ajisafe
|
Emeka Onwuegbuzia
|
Falalu Lawan
|
Ibrahim Ahmad
|
Jesujoba Alabi
|
Chinedu Mbonu
|
Mofetoluwa Adeyemi
|
Mofya Phiri
|
Orevaoghene Ahia
|
Ruqayya Iro
|
Sonia Adhiambo
Findings of the Association for Computational Linguistics: EMNLP 2023
African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems – those that retrieve answer content from other languages while serving people in their native language—offer a means of filling this gap. To this end, we create Our Dataset, the first cross-lingual QA dataset with a focus on African languages. Our Dataset includes 12,000+ XOR QA examples across 10 African languages. While previous datasets have focused primarily on languages where cross-lingual QA augments coverage from the target language, Our Dataset focuses on languages where cross-lingual answer content is the only high-coverage source of answer content. Because of this, we argue that African languages are one of the most important and realistic use cases for XOR QA. Our experiments demonstrate the poor performance of automatic translation and multilingual retrieval methods. Overall, Our Dataset proves challenging for state-of-the-art QA models. We hope that the dataset enables the development of more equitable QA technology.
pdf
bib
abs
Better Quality Pre-training Data and T5 Models for African Languages
Akintunde Oladipo
|
Mofetoluwa Adeyemi
|
Orevaoghene Ahia
|
Abraham Owodunni
|
Odunayo Ogundepo
|
David Adelani
|
Jimmy Lin
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
In this study, we highlight the importance of enhancing the quality of pretraining data in multilingual language models. Existing web crawls have demonstrated quality issues, particularly in the context of low-resource languages. Consequently, we introduce a new multilingual pretraining corpus for 16 African languages, designed by carefully auditing existing pretraining corpora to understand and rectify prevalent quality issues. To compile this dataset, we undertake a rigorous examination of current data sources for thirteen languages within one of the most extensive multilingual web crawls, mC4, and extract cleaner data through meticulous auditing and improved web crawling strategies. Subsequently, we pretrain a new T5-based model on this dataset and evaluate its performance on multiple downstream tasks. Our model demonstrates better downstream effectiveness over existing pretrained models across four NLP tasks, underscoring the critical role data quality plays in pretraining language models in low-resource scenarios. Specifically, on cross-lingual QA evaluation, our new model is more than twice as effective as multilingual T5. All code, data and models are publicly available at https://github.com/castorini/AfriTeVa-keji.
pdf
bib
abs
Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face
Christopher Akiki
|
Odunayo Ogundepo
|
Aleksandra Piktus
|
Xinyu Zhang
|
Akintunde Oladipo
|
Jimmy Lin
|
Martin Potthast
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
We present Spacerini, a tool that integrates the Pyserini toolkit for reproducible information retrieval research with Hugging Face to enable the seamless construction and deployment of interactive search engines. Spacerini makes state-of-the-art sparse and dense retrieval models more accessible to non-IR practitioners while minimizing deployment effort. This is useful for NLP researchers who want to better understand and validate their research by performing qualitative analyses of training corpora, for IR researchers who want to demonstrate new retrieval models integrated into the growing Pyserini ecosystem, and for third parties reproducing the work of other researchers. Spacerini is open source and includes utilities for loading, preprocessing, indexing, and deploying search engines locally and remotely. We demonstrate a portfolio of 13 search engines created with Spacerini for different use cases.
pdf
bib
MasakhaNEWS: News Topic Classification for African languages
David Ifeoluwa Adelani
|
Marek Masiak
|
Israel Abebe Azime
|
Jesujoba Alabi
|
Atnafu Lambebo Tonja
|
Christine Mwase
|
Odunayo Ogundepo
|
Bonaventure F. P. Dossou
|
Akintunde Oladipo
|
Doreen Nixdorf
|
Chris Chinenye Emezue
|
Sana Al-azzawi
|
Blessing Sibanda
|
Davis David
|
Lolwethu Ndolela
|
Jonathan Mukiibi
|
Tunde Ajayi
|
Tatiana Moteu
|
Brian Odhiambo
|
Abraham Owodunni
|
Nnaemeka Obiefuna
|
Muhidin Mohamed
|
Shamsuddeen Hassan Muhammad
|
Teshome Mulugeta Ababu
|
Saheed Abdullahi Salahudeen
|
Mesay Gemeda Yigezu
|
Tajuddeen Gwadabe
|
Idris Abdulmumin
|
Mahlet Taye
|
Oluwabusayo Awoyomi
|
Iyanuoluwa Shode
|
Tolulope Adelani
|
Habiba Abdulganiyu
|
Abdul-Hakeem Omotayo
|
Adetola Adeeko
|
Abeeb Afolabi
|
Anuoluwapo Aremu
|
Olanrewaju Samuel
|
Clemencia Siro
|
Wangari Kimotho
|
Onyekachi Ogbu
|
Chinedu Mbonu
|
Chiamaka Chukwuneke
|
Samuel Fanijo
|
Jessica Ojo
|
Oyinkansola Awosan
|
Tadesse Kebede
|
Toadoum Sari Sakayo
|
Pamela Nyatsine
|
Freedmore Sidume
|
Oreen Yousuf
|
Mardiyyah Oduwole
|
Kanda Tshinu
|
Ussen Kimanuka
|
Thina Diko
|
Siyanda Nxakama
|
Sinodos Nigusse
|
Abdulmejid Johar
|
Shafie Mohamed
|
Fuad Mire Hassan
|
Moges Ahmed Mehamed
|
Evrard Ngabire
|
Jules Jules
|
Ivan Ssenkungu
|
Pontus Stenetorp
Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
pdf
bib
abs
MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Xinyu Zhang
|
Nandan Thakur
|
Odunayo Ogundepo
|
Ehsan Kamalloo
|
David Alfonso-Hermelo
|
Xiaoguang Li
|
Qun Liu
|
Mehdi Rezagholizadeh
|
Jimmy Lin
Transactions of the Association for Computational Linguistics, Volume 11
MIRACL is a multilingual dataset for ad hoc retrieval across 18 languages that collectively encompass over three billion native speakers around the world. This resource is designed to support monolingual retrieval tasks, where the queries and the corpora are in the same language. In total, we have gathered over 726k high-quality relevance judgments for 78k queries over Wikipedia in these languages, where all annotations have been performed by native speakers hired by our team. MIRACL covers languages that are both typologically close as well as distant from 10 language families and 13 sub-families, associated with varying amounts of publicly available resources. Extensive automatic heuristic verification and manual assessments were performed during the annotation process to control data quality. In total, MIRACL represents an investment of around five person-years of human annotator effort. Our goal is to spur research on improving retrieval across a continuum of languages, thus enhancing information access capabilities for diverse populations around the world, particularly those that have traditionally been underserved. MIRACL is available at http://miracl.ai/.
2022
pdf
bib
abs
AfriCLIRMatrix: Enabling Cross-Lingual Information Retrieval for African Languages
Odunayo Ogundepo
|
Xinyu Zhang
|
Shuo Sun
|
Kevin Duh
|
Jimmy Lin
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Language diversity in NLP is critical in enabling the development of tools for a wide range of users.However, there are limited resources for building such tools for many languages, particularly those spoken in Africa.For search, most existing datasets feature few or no African languages, directly impacting researchers’ ability to build and improve information access capabilities in those languages.Motivated by this, we created AfriCLIRMatrix, a test collection for cross-lingual information retrieval research in 15 diverse African languages.In total, our dataset contains 6 million queries in English and 23 million relevance judgments automatically mined from Wikipedia inter-language links, covering many more African languages than any existing information retrieval test collection.In addition, we release BM25, dense retrieval, and sparse–dense hybrid baselines to provide a starting point for the development of future systems.We hope that these efforts can spur additional work in search for African languages.AfriCLIRMatrix can be downloaded at https://github.com/castorini/africlirmatrix.