Postdoctoral Researcher
AI2 & University of Washington

Yanai Elazar

I am a postdoctoral researcher on the AllenNLP team at AI2, and the University of Washington, working with Noah A. Smith and Hannaneh Hajishirzi. I also work closely with Sameer Singh and his lab at UCI. I'm fortunate to be a Rothschild Fellow. Before joining AI2 and UW, I completed my PhD (2022) in Computer Science in the NLP lab at Bar-Ilan University, where I was lucky to be a Google PhD Fellow, and worked with Yoav Goldberg.

Broadly, I'm interested in Natural Language Processing and Machine Learning. I'm particularly interested in the Science of Language Models, where I develop methods for understanding what makes models work, how, and why. These days, I focus on the data on which such models are trained and draw connections between the data and model behavior.


I'm happy to talk about research in general, and my own work in particular. If you have any questions about one of my papers, or my overall research, feel free to reach out!

I am on the academic job market!

News

Interviewed for the Causal Bandits podcast
October 2024
OLMo and Dolma got best paper awards at ACL 2024
August 2024
Co-Organized the 1st Data Contamination Workshop at ACL 2024
August 2024
I was interviewed for an article on Science News
July 2024
Giving talks at the Technion, Bar-Ilan University, and IBM
May 2024
Attending a Dagstuhl Seminar on 'Trustworthiness and Responsibility in AI - Causality, Learning, and Verification'
March 2024
Giving talks at LMU Munich and Edinburgh NLP
March 2024
Giving talks at USC, UCLA, and UCSB
Febrauary 2024
IAAI Best Thesis Award Runner Up
December 2023
Co-Organized The Big Picture Workshop at EMNLP 2023
December 2023
Attending SoCal NLP. Come say hi!
November 2023
Invited talk at EPFL
November 2023
Wrote a 502 page about my academic failures
October 2023
Our work (turned into WIMBD) was featured in The Washington Post
April 2023

Publications

2024

GRADE: Quantifying Sample Diversity in Text-to-Image Models
Royi Rassin, Aviv Slobodkin, Shauli Ravfogel, Yanai Elazar, Yoav Goldberg
arxiv
paper code webpage

Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
*Xinyi Wang, *Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, William Yang Wang
arxiv
paper

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
*Lester James V. Miranda, *Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi
arxiv
paper code resource blog

How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation Threshold
Sahil Verma, Royi Rassin, Arnav Das, Gantavya Bhatt, Preethi Seshadri, Chirag Shah, Jeff Bilmes, Hannaneh Hajishirzi, Yanai Elazar
arxiv
paper code webpage blog

Paloma: A Benchmark for Evaluating Language Model Fit
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, Jesse Dodge
NeurIPS 2024 Track Datasets and Benchmarks
paper code resource

Evaluating \( n \)-Gram Novelty of Language Models Using Rusty-DAWG
William Merrill, Noah A. Smith, Yanai Elazar
EMNLP 2024
paper long code

Detection and Measurement of Syntactic Templates in Generated Text
Chantal Shaib, Yanai Elazar, Junyi Jessy Li, Byron C. Wallace
EMNLP 2024
paper long

Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation
Bar Iluz, Yanai Elazar, Asaf Yehudai, Gabriel Stanovsky
EMNLP 2024
paper short

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Raghavi, Vivek Srikumar, Sameer Singh, Noah A. Smith
Findings of EMNLP 2024
paper long

Data Contamination Report from the 2024 CONDA Shared Task
Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang
CONDA Workshop @ ACL 2024
paper resource

Calibrating Large Language Models with Sample Consistency
*Qing Lyu, *Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, Chris Callison-Burch
arxiv
paper

OLMo: Accelerating the Science of Language Models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi
ACL 2024 🏆 Best Theme Paper
paper long code resource models
Press: TechCrunch Axios Forbes GeekWire SD Times VentureBeat Fast Company

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, Kyle Lo
ACL 2024 🏆 Best Resource Paper
paper long code resource

The Bias Amplification Paradox in Text-to-Image Generation
Preethi Seshadri, Sameer Singh, Yanai Elazar
NAACL 2024
paper long code poster

A Survey on Data Selection for Language Models
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang
TMLR 2024
paper resource

Backtracking Mathematical Reasoning of Language Models to the Pretraining Data
*Yasaman Razeghi, *Hamish Ivison, Sameer Singh, Yanai Elazar
Tiny Papers, ICLR 2024
paper workshop

What's In My Big Data?
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, Jesse Dodge
ICLR 2024, spotlight
paper code demo 🤗 HF demo poster
Press: Marktechpost Washington Post

Estimating the Causal Effect of Early ArXiving on Paper Acceptance
*Yanai Elazar, *Jiayao Zhang, *David Wadden, Bo Zhang, Noah A. Smith
CLeaR 2024
paper code poster
Press: Causal Bandits podcast

2023

A taxonomy and review of generalization research in NLP
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Rita Frieske, Ryan Cotterell, Zhijing Jin
Nature Machine Intelligence 2023
paper journal project-page

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, Yanai Elazar
Findings of ACL 2023
paper long code poster

CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm
Hongming Zhang, Yintong Huo, Yanai Elazar, Yangqiu Song, Yoav Goldberg, Dan Roth
Findings of EACL 2023
paper long code

2022

Lexical Generalization Improves with Larger Models and Longer Training
Elron Bandel, Yoav Goldberg, Yanai Elazar
Findings of EMNLP 2022
paper short code poster

Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, Yoav Goldberg
arxiv
paper models

Text-based NP Enrichment
*Yanai Elazar, *Victoria Basmov, Yoav Goldberg, Reut Tsarfaty
TACL 2022
paper journal code resource demo project-page slides video

2021

Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes
Ofer Sabo, Yanai Elazar, Yoav Goldberg, Ido Dagan
TACL 2021
paper journal code video

Back to Square One: Bias Detection, Training and Commonsense Disentanglement in the Winograd Schema
Yanai Elazar, Hongming Zhang, Yoav Goldberg, Dan Roth
EMNLP 2021
paper long code slides video

Contrastive Explanations for Model Interpretability
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel, Yanai Elazar, Yejin Choi, Yoav Goldberg
EMNLP 2021
paper long code video

Measuring and Improving Consistency in Pretrained Language Models
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
TACL 2021
paper journal code resource slides video

First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT
Benjamin Muller, Yanai Elazar, Benoît Sagot and Djamé Seddah
EACL 2021
paper short code

*Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, Yoav Goldberg
TACL 2021
(*) previous version that appeared on arxiv was named: "When Bert Forgets How To POS: Amnesic Probing of Linguistic Properties and MLM Predictions", which we changed to the current title to better reflect our contributions.
paper journal code slides video

2020

At Your Fingertips: Extracting Piano Fingering Instructions from Videos
Amit Moryossef, Yanai Elazar, Yoav Goldberg
arxiv
paper code

It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT
Hila Gonen, Shauli Ravfogel, Yanai Elazar, Yoav Goldberg
Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, at EMNLP 2020
paper long code poster

The Extraordinary Failure of Complement Coercion Crowdsourcing
Yanai Elazar, Victoria Basmov, Shauli Ravfogel, Yoav Goldberg, Reut Tsarfaty
Workshop on Insights from Negative Results in NLP, EMNLP 2020
paper short slides video

Do Language Embeddings Capture Scales?
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, Dan Roth
Findings of EMNLP 2020
paper long code

Unsupervised Distillation of Syntactic Information from Contextualized Word Representations
*Shauli Ravfogel, *Yanai Elazar, Jacob Goldberger, Yoav Goldberg
Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, at EMNLP 2020
paper long code slides

Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg
ACL 2020
paper long code video

Evaluating Models' Local Decision Boundaries via Contrast Sets
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
Findings of EMNLP 2020
paper long resource

oLMpics -- On what Language Model Pre-training Captures
Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant
TACL 2020 (presented at EMNLP 2020)
paper journal code video

2019

Adversarial Removal of Demographic Attributes Revisited
Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, Anders Søgaard
EMNLP 2019
paper short

How Large Are Lions? Inducing Distributions over Quantitative Attributes
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, Dan Roth
ACL 2019
paper long code resource demo slides

Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolution
Yanai Elazar, Yoav Goldberg
TACL 2019 (presented at EMNLP 2019)
paper journal code resource demo slides video

Privacy and Fairness in Recommender Systems via Adversarial Training of User Representations
Yehezkel S. Resheff, Yanai Elazar, Moni Shahar, Oren Sar Shalom
ICPRAM 2019
paper long

2018

Adversarial Removal of Demographic Attributes from Text Data
Yanai Elazar, Yoav Goldberg
EMNLP 2018
paper long code slides video

Posts

Meta-Reviewing for ACL-ARR (EMNLP)

My experience from meta-reviewing for ARR, and on some reviewer's fallacies

From Interviewee To Interviewer

Behind the scences of the interviewing process

Attending ACL 2020

My strategy for attending my first virtual conference.

Remote Servers

How to setup your environment to seemingly work with remote servers.