Srini Narayanan

Also published as: Srinivas Narayanan


2024

pdf bib
UGIF-DataSet: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI
Sagar Gubbi Venkatesh | Partha Talukdar | Srini Narayanan
Findings of the Association for Computational Linguistics: NAACL 2024

Help documents are supposed to aid smartphone users in resolving queries such as “How to block calls from unknown numbers?”. However, given a query, identifying the right help document, understanding instructions from the document, and using them to resolve the issue at hand is challenging. The user experience may be enhanced by converting the instructions in the help document to a step-by-step tutorial overlaid on the phone UI. Successful execution of this task requires overcoming research challenges in retrieval, parsing, and grounding in the multilingual-multimodal setting. For example, user queries in one language may have to be matched against instructions in another language, which in turn needs to be grounded in a multimodal UI in yet another language. Moreover, there isn’t any relevant dataset for such a task. In order to bridge this gap, we introduce UGIF-DataSet, a multi-lingual, multi-modal UI grounded dataset for step-by-step task completion on the smartphone, containing 4,184 tasks across 8 languages. The instruction steps in UGIF-DataSet are available only in English, so the challenge involves operations in the cross-modal, cross-lingual setting. We compare the performance of different large language models for this task and find that the end-to-end task completion rate drops from 48% in English to 32% for other languages, demonstrating significant overall headroom for improvement. We are hopeful that UGIF-DataSet and our analysis will aid further research on the important problem of sequential task completion in the multilingual and multimodal setting.

2023

pdf bib
A Benchmark for Reasoning with Spatial Prepositions
Iulia Comsa | Srini Narayanan
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Spatial reasoning is a fundamental building block of human cognition, used in representing, grounding, and reasoning about physical and abstract concepts. We propose a novel benchmark focused on assessing inferential properties of statements with spatial prepositions. The benchmark includes original datasets in English and Romanian and aims to probe the limits of reasoning about spatial relations in large language models. We use prompt engineering to study the performance of two families of large language models, PaLM and GPT-3, on our benchmark. Our results show considerable variability in the performance of smaller and larger models, as well as across prompts and languages. However, none of the models reaches human performance.

2022

pdf bib
MiQA: A Benchmark for Inference on Metaphorical Questions
Iulia Comșa | Julian Eisenschlos | Srini Narayanan
Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)

We propose a benchmark to assess the capability of large language models to reason with conventional metaphors. Our benchmark combines the previously isolated topics of metaphor detection and commonsense reasoning into a single task that requires a model to make inferences by accurately selecting between the literal and metaphorical register. We examine the performance of state-of-the-art pre-trained models on binary-choice tasks and find a large discrepancy between the performance of small and very large models, going from chance to near-human level. We also analyse the largest model in a generative setting and find that although human performance is approached, careful multiple-shot prompting is required.

2018

pdf bib
Points, Paths, and Playscapes: Large-scale Spatial Language Understanding Tasks Set in the Real World
Jason Baldridge | Tania Bedrax-Weiss | Daphne Luong | Srini Narayanan | Bo Pang | Fernando Pereira | Radu Soricut | Michael Tseng | Yuan Zhang
Proceedings of the First International Workshop on Spatial Language Understanding

Spatial language understanding is important for practical applications and as a building block for better abstract language understanding. Much progress has been made through work on understanding spatial relations and values in images and texts as well as on giving and following navigation instructions in restricted domains. We argue that the next big advances in spatial language understanding can be best supported by creating large-scale datasets that focus on points and paths based in the real world, and then extending these to create online, persistent playscapes that mix human and bot players, where the bot players must learn, evolve, and survive according to their depth of understanding of scenes, navigation, and interactions.

2017

pdf bib
Multilingual Metaphor Processing: Experiments with Semi-Supervised and Unsupervised Learning
Ekaterina Shutova | Lin Sun | Elkin Darío Gutiérrez | Patricia Lichtenstein | Srini Narayanan
Computational Linguistics, Volume 43, Issue 1 - April 2017

Highly frequent in language and communication, metaphor represents a significant challenge for Natural Language Processing (NLP) applications. Computational work on metaphor has traditionally evolved around the use of hand-coded knowledge, making the systems hard to scale. Recent years have witnessed a rise in statistical approaches to metaphor processing. However, these approaches often require extensive human annotation effort and are predominantly evaluated within a limited domain. In contrast, we experiment with weakly supervised and unsupervised techniques—with little or no annotation—to generalize higher-level mechanisms of metaphor from distributional properties of concepts. We investigate different levels and types of supervision (learning from linguistic examples vs. learning from a given set of metaphorical mappings vs. learning without annotation) in flat and hierarchical, unconstrained and constrained clustering settings. Our aim is to identify the optimal type of supervision for a learning algorithm that discovers patterns of metaphorical association from text. In order to investigate the scalability and adaptability of our models, we applied them to data in three languages from different language groups—English, Spanish, and Russian—achieving state-of-the-art results with little supervision. Finally, we demonstrate that statistical methods can facilitate and scale up cross-linguistic research on metaphor.

2014

pdf bib
Bridging Text and Knowledge with Frames
Srini Narayanan
Proceedings of Frame Semantics in NLP: A Workshop in Honor of Chuck Fillmore (1929-2014)

2006

pdf bib
Catching Metaphors
Matt Gedigian | John Bryant | Srini Narayanan | Branimir Ciric
Proceedings of the Third Workshop on Scalable Natural Language Understanding

2004

pdf bib
Answering Questions Using Advanced Semantics and Probabilistic Inference
Srini Narayanan | Sanda Harabagiu
Proceedings of the Workshop on Pragmatics of Question Answering at HLT-NAACL 2004

pdf bib
Question Answering Based on Semantic Structures
Srini Narayanan | Sanda Harabagiu
COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics

2003

pdf bib
Putting FrameNet Data into the ISO Linguistic Annotation Framework
Srinivas Narayanan | Miriam R. L. Petruck | Collin F. Baker | Charles J. Fillmore
Proceedings of the ACL 2003 Workshop on Linguistic Annotation: Getting the Model Right

pdf bib
Semantic Extraction with Wide-Coverage Lexical Resources
Behrang Mohit | Srini Narayanan
Companion Volume of the Proceedings of HLT-NAACL 2003 - Short Papers

2002

pdf bib
Putting Frames in Perspective
Nancy Chang | Srini Narayanan | Miriam R.L. Petruck
COLING 2002: The 19th International Conference on Computational Linguistics