Yanai Elazar

Publications

2025

Chain-of-Thought Is Not Explainability

Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Damiano Fornasiere, John Yan, Yanai Elazar, Yoshua Bengio
arxiv
paper bibtex

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback

*Lester James V. Miranda, *Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi
ACL 2025
paper bibtex code resource blog

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
ACL system demonstrations 2025 🏆 Best Demo Award
paper bibtex demo

Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases

Shanshan Xu, T.Y.S.S Santosh, Yanai Elazar, Quirin Vogel, Barbara Plank, Matthias Grabmair
The First Workshop on Large Language Model Memorization @ ACL 2025
paper bibtex

On Linear Representations and Pretraining Data Frequency in Language Models

Jack Merullo, Noah A. Smith, *Sarah Wiegreffe, *Yanai Elazar
ICLR 2025
paper bibtex

Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data

*Xinyi Wang, *Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, William Yang Wang
ICLR 2025
paper bibtex code

Calibrating Large Language Models with Sample Consistency

*Qing Lyu, *Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, Chris Callison-Burch
AAAI 2025
paper bibtex code

2024

GRADE: Quantifying Sample Diversity in Text-to-Image Models

Royi Rassin, Aviv Slobodkin, Shauli Ravfogel, Yanai Elazar, Yoav Goldberg
arxiv
paper bibtex code webpage

How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation Threshold

Sahil Verma, Royi Rassin, Arnav Das, Gantavya Bhatt, Preethi Seshadri, Chirag Shah, Jeff Bilmes, Hannaneh Hajishirzi, Yanai Elazar
arxiv
paper bibtex code webpage blog

Paloma: A Benchmark for Evaluating Language Model Fit

Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, Jesse Dodge
NeurIPS 2024 Track Datasets and Benchmarks
paper bibtex code resource

Evaluating \( n \)-Gram Novelty of Language Models Using Rusty-DAWG

William Merrill, Noah A. Smith, Yanai Elazar
EMNLP 2024
paper bibtex long code

Detection and Measurement of Syntactic Templates in Generated Text

Chantal Shaib, Yanai Elazar, Junyi Jessy Li, Byron C. Wallace
EMNLP 2024
paper bibtex long code webpage

Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation

Bar Iluz, Yanai Elazar, Asaf Yehudai, Gabriel Stanovsky
EMNLP 2024
paper bibtex short

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals

Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Raghavi, Vivek Srikumar, Sameer Singh, Noah A. Smith
Findings of EMNLP 2024
paper bibtex long

Data Contamination Report from the 2024 CONDA Shared Task

Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang
CONDA Workshop @ ACL 2024
paper bibtex resource

OLMo: Accelerating the Science of Language Models

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi
ACL 2024 🏆 Best Theme Paper
paper bibtex long code resource models
Press: TechCrunch Axios Forbes GeekWire SD Times VentureBeat Fast Company

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, Kyle Lo
ACL 2024 🏆 Best Resource Paper
paper bibtex long code resource

The Bias Amplification Paradox in Text-to-Image Generation

Preethi Seshadri, Sameer Singh, Yanai Elazar
NAACL 2024
paper bibtex long code poster

A Survey on Data Selection for Language Models

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang
TMLR 2024
paper bibtex resource

Backtracking Mathematical Reasoning of Language Models to the Pretraining Data

*Yasaman Razeghi, *Hamish Ivison, Sameer Singh, Yanai Elazar
Tiny Papers, ICLR 2024
paper bibtex workshop

What's In My Big Data?

Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, Jesse Dodge
ICLR 2024, spotlight
paper bibtex code demo 🤗 HF demo poster
Press: Marktechpost Washington Post

Estimating the Causal Effect of Early ArXiving on Paper Acceptance

*Yanai Elazar, *Jiayao Zhang, *David Wadden, Bo Zhang, Noah A. Smith
CLeaR 2024
paper bibtex code poster
Press: Causal Bandits podcast

2023

A taxonomy and review of generalization research in NLP

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Rita Frieske, Ryan Cotterell, Zhijing Jin
Nature Machine Intelligence 2023
paper bibtex journal project-page

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, Yanai Elazar
Findings of ACL 2023
paper bibtex long code poster

CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm

Hongming Zhang, Yintong Huo, Yanai Elazar, Yangqiu Song, Yoav Goldberg, Dan Roth
Findings of EACL 2023
paper bibtex long code

2022

Lexical Generalization Improves with Larger Models and Longer Training

Elron Bandel, Yoav Goldberg, Yanai Elazar
Findings of EMNLP 2022
paper bibtex short code poster

Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, Yoav Goldberg
arxiv
paper bibtex models

Text-based NP Enrichment

*Yanai Elazar, *Victoria Basmov, Yoav Goldberg, Reut Tsarfaty
TACL 2022
paper bibtex journal code resource demo project-page slides video

2021

Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes

Ofer Sabo, Yanai Elazar, Yoav Goldberg, Ido Dagan
TACL 2021
paper bibtex journal code video

Back to Square One: Bias Detection, Training and Commonsense Disentanglement in the Winograd Schema

Yanai Elazar, Hongming Zhang, Yoav Goldberg, Dan Roth
EMNLP 2021
paper bibtex long code slides video

Contrastive Explanations for Model Interpretability

Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, Yoav Goldberg
EMNLP 2021
paper bibtex long code video

Measuring and Improving Consistency in Pretrained Language Models

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
TACL 2021
paper bibtex journal code resource slides video

First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT

Benjamin Muller, Yanai Elazar, Benoît Sagot and Djamé Seddah
EACL 2021
paper bibtex short code

*Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

Yanai Elazar, Shauli Ravfogel, Alon Jacovi, Yoav Goldberg
TACL 2021
(*) previous version that appeared on arxiv was named: "When Bert Forgets How To POS: Amnesic Probing of Linguistic Properties and MLM Predictions", which we changed to the current title to better reflect our contributions.
paper bibtex journal code slides video

2020

At Your Fingertips: Extracting Piano Fingering Instructions from Videos

Amit Moryossef, Yanai Elazar, Yoav Goldberg
arxiv
paper bibtex code

It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT

Hila Gonen, Shauli Ravfogel, Yanai Elazar, Yoav Goldberg
Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, at EMNLP 2020
paper bibtex long code poster

The Extraordinary Failure of Complement Coercion Crowdsourcing

Yanai Elazar, Victoria Basmov, Shauli Ravfogel, Yoav Goldberg, Reut Tsarfaty
Workshop on Insights from Negative Results in NLP, EMNLP 2020
paper bibtex short slides video

Do Language Embeddings Capture Scales?

Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, Dan Roth
Findings of EMNLP 2020
paper bibtex long code

Unsupervised Distillation of Syntactic Information from Contextualized Word Representations

*Shauli Ravfogel, *Yanai Elazar, Jacob Goldberger, Yoav Goldberg
Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, at EMNLP 2020
paper bibtex long code slides

Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg
ACL 2020
paper bibtex long code video

Evaluating Models' Local Decision Boundaries via Contrast Sets

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
Findings of EMNLP 2020
paper bibtex long resource

oLMpics -- On what Language Model Pre-training Captures

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant
TACL 2020 (presented at EMNLP 2020)
paper bibtex journal code video

2019

Adversarial Removal of Demographic Attributes Revisited

Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, Anders Søgaard
EMNLP 2019
paper bibtex short

How Large Are Lions? Inducing Distributions over Quantitative Attributes

Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, Dan Roth
ACL 2019
paper bibtex long code resource demo slides

Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolution

Yanai Elazar, Yoav Goldberg
TACL 2019 (presented at EMNLP 2019)
paper bibtex journal code resource demo slides video

Privacy and Fairness in Recommender Systems via Adversarial Training of User Representations

Yehezkel S. Resheff, Yanai Elazar, Moni Shahar, Oren Sar Shalom
ICPRAM 2019
paper bibtex long

2018

Adversarial Removal of Demographic Attributes from Text Data

Yanai Elazar, Yoav Goldberg
EMNLP 2018
paper bibtex long code slides video