# Marco Tulio Ribeiro

**Source**: https://homes.cs.washington.edu/~marcotcr/
**Parent**: https://idl.uw.edu/papers

I'm a researcher at Google DeepMind. I am also an Affiliate Assistant Professor at the University of Washington, where I was previously a Ph.D student advised by [Carlos Guestrin](http://homes.cs.washington.edu/~guestrin) and
[Sameer Singh](http://sameersingh.org/).

My research is mostly on helping humans interact with machine learning models meaningfully. That involves interpretability, trust, debugging, feedback, etc.

Despite various attempts, I haven't made much progress on the (much harder) problem of getting a particular group of humans to all look at a camera at the same time →→→

### Blog

I wrote these posts on how to pick a project for an intern, but I figured others might be interested too:

- [Part 1: Coming up with research ideas](https://medium.com/@marcotcr/coming-up-with-research-ideas-3032682e5852)
- [Part 2: Organizing and evaluating research ideas](https://medium.com/@marcotcr/organizing-and-evaluating-research-ideas-e137637b599e)

I also wrote this one on writing:

- [Writing, part 1 — the process](https://medium.com/@marcotcr/writing-part-1-the-process-6bb92cb522eb)

### Publications

- [Sparks of artificial general intelligence: Early experiments with GPT-4](https://arxiv.org/pdf/2303.12712.pdf)\
  Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, **Marco Tulio Ribeiro**, Yi Zhang
- [ScatterShot: Interactive In-context Example Curation for Text Transformation](https://dl.acm.org/doi/pdf/10.1145/3581641.3584059)\
  Tongshuang Wu, Hua Shen, Daniel Weld, Jeffrey Heer, **Marco Tulio Ribeiro**\
  *In: International Conference on Intelligent User Interfaces (IUI), 2023*\
  **Best Paper Honorable mention**
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://arxiv.org/abs/2303.09014)\
  Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi,\
  Luke Zettlemoyer, **Marco Tulio Ribeiro**\
  *In submission*
- [Editing Models with Task Arithmetic](https://arxiv.org/pdf/2212.04089.pdf)\
  Gabriel Ilharco, **Marco Tulio Ribeiro**, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi\
  *In: International Conference on Learning Representations (ICLR), 2023*\
  [[code]](https://github.com/mlfoundations/task_vectors)
- [Adaptive Testing and Debugging of NLP Models](https://aclanthology.org/2022.acl-long.230.pdf)\
  **Marco Tulio Ribeiro**\*, Scott Lundberg\* (Equal contribution)\
  *In: Association for Computational Linguistics (ACL), 2022*\
  [[code]](https://github.com/microsoft/adatest)
  [[bibtex]](http://homes.cs.washington.edu/~marcotcr/acl22_adatest.bib)
- [Fixing Model Bugs with Natural Language Patches](http://homes.cs.washington.edu/~marcotcr/emnlp22_patches.pdf)\
  Shikhar Murty, Christopher D. Manning, Scott Lundberg, **Marco Tulio Ribeiro**\
  *In: Empirical Methods in Natural Language Processing (EMNLP), 2022*
- [Finding and Fixing Spurious Patterns with Explanations](https://openreview.net/pdf?id=whJPugmP5I)\
  Gregory Plumb, **Marco Tulio Ribeiro**, Ameet Talwalkar\
  *In: Transactions on Machine Learning Research (TMLR), 2022*
- [ExSum: From Local Explanations to Model Understanding](https://arxiv.org/pdf/2205.00130.pdf)\
  Yilun Zhou, **Marco Tulio Ribeiro**, Julie Shah\
  *In: Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), 2022*\
  [[code]](https://github.com/YilunZhou/ExSum)
- [What Did My AI Learn? How Data Scientists Make Sense of Model Behavior](https://dl.acm.org/doi/pdf/10.1145/3542921)\
  Ángel Alexander Cabrera, **Marco Tulio Ribeiro**, Bongshin Lee, Rob DeLine, Adam Perer, Steven M. Drucker\
  *In: ACM Transactions on Computer-Human Interaction (TOCHI), 2022*\
  [[bibtex]](https://dl.acm.org/doi/10.1145/3542921)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://arxiv.org/pdf/2104.14403.pdf)\
  Yilun Zhou, Serena Booth, **Marco Tulio Ribeiro**, Julie Shah\
  *In: AAAI Conference on Artificial Intelligence (AAAI), 2022*\
  [[code]](https://github.com/YilunZhou/feature-attribution-evaluation)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://aiweb.cs.washington.edu/ai/pubs/bansal-chi21.pdf) \
  Gagan Bansal\*, Tongshuang Wu\*, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, **Marco Tulio Ribeiro**, Daniel S. Weld\
  *In: CHI 2021: the 2021 Conference on Human Factors in Computing Systems*
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](http://homes.cs.washington.edu/~marcotcr/acl21_polyjuice.pdf)\
  Tongshuang Wu, **Marco Tulio Ribeiro**, Jeffrey Heer, Daniel Weld\
  *In: Association for Computational Linguistics (ACL), 2021*\
  [[code]](https://github.com/tongshuangwu/polyjuice) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/acl21_counterfactuals.bib)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](http://homes.cs.washington.edu/~marcotcr/acl20_checklist.pdf)\
  **Marco Tulio Ribeiro**, Tongshuang Wu, Carlos Guestrin, Sameer Singh.\
  *In: Association for Computational Linguistics (ACL), 2020*\
  **Best Paper Award**\
  [[code]](https://github.com/marcotcr/checklist) [[talk]](https://www.youtube.com/watch?v=VqiTtdY58Ts) [[slides]](https://drive.google.com/file/d/1j5UlxoqQYLA_lqD5q5JT4jJn-IMnbamp/view?usp=sharing) [[longer slides]](https://drive.google.com/file/d/1x3zyMBShLN5QM86TmcPMf-RlrHzNypm-/view?usp=sharing) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/acl20_checklist.bib)
- [SQuINTing at VQA Models: Interrogating VQA Models with Sub-Questions](https://arxiv.org/abs/2001.06927)\
  Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz,\
  **Marco Tulio Ribeiro**, Besmira Nushi, Ece Kamar.\
  *In: Conference on Computer Vision and Pattern Recognition (CVPR), 2020*\
  [[bibtex]](http://homes.cs.washington.edu/~marcotcr/cvpr20_squint.bib)
- [Errudite: Scalable, Reproducible, and Testable Error Analysis](http://homes.cs.washington.edu/~marcotcr/acl19_errudite.pdf)\
  Tongshuang Wu, **Marco Tulio Ribeiro**, Jeffrey Heer, Daniel Weld.\
  *In: Association for Computational Linguistics (ACL), 2019*\
  [[code]](https://github.com/uwdata/errudite) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/acl19_errudite.bib) [[blog]](https://medium.com/@uwdata/errudite-55d5fbf3232e)
- [Are Red Roses Red? Evaluating Consistency of Question-Answering Models](http://homes.cs.washington.edu/~marcotcr/acl19_implication.pdf)\
  **Marco Tulio Ribeiro**, Carlos Guestrin, Sameer Singh.\
  *In: Association for Computational Linguistics (ACL), 2019*\
  [[code]](https://github.com/marcotcr/qa_consistency) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/acl19_implication.bib)
- [Semantically Equivalent Adversarial Rules for Debugging NLP Models](http://homes.cs.washington.edu/~marcotcr/acl18.pdf)\
  **Marco Tulio Ribeiro**, Sameer Singh, Carlos Guestrin.\
  *In: Association for Computational Linguistics (ACL), 2018*\
  **Honorable mention for best paper award**\
  [[code]](https://github.com/marcotcr/sears) [[talk]](https://vimeo.com/285801102) [[slides]](https://drive.google.com/open?id=1rDkGPgyd6RkIZD0v4umHYFlUSXlbt-cn) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/acl18.bib)
- [Anchors: High-Precision Model-Agnostic Explanations](http://homes.cs.washington.edu/~marcotcr/aaai18.pdf)\
  **Marco Tulio Ribeiro**, Sameer Singh, Carlos Guestrin.\
  *In: AAAI Conference on Artificial Intelligence (AAAI), 2018*\
  [[code]](https://github.com/marcotcr/anchor) [[slides]](https://drive.google.com/open?id=1QdBRUCn_yYtUlJaTyWghcp9XJZ4K2kY9) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/aaai18.bib)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](http://arxiv.org/abs/1602.04938)\
  **Marco Tulio Ribeiro**, Sameer Singh, Carlos Guestrin.\
  *In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016*\
  **Audience appreciation award** [[video]](https://www.youtube.com/watch?v=hUnRCxnydCc)\
  [[code]](https://github.com/marcotcr/lime) [[talk]](https://www.youtube.com/watch?v=KP7-JtFMLo4&t=1s) [[slides]](https://drive.google.com/open?id=0ByblrZgHugfYZ0ZCSWNPWFNONEU) [[bibtex]](http://homes.cs.washington.edu/~marcotcr/kdd16.bib) [[blog]](https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime)
- [Model-Agnostic Interpretability of Machine Learning](https://arxiv.org/pdf/1606.05386v1.pdf)\
  **Marco Tulio Ribeiro**, Sameer Singh, Carlos Guestrin.\
  *In: ICML Workshop on Human Interpretability in Machine Learning (WHI), 2016*\
  **Best paper award**\
  [[bibtex]](http://homes.cs.washington.edu/~marcotcr/whi2016.bib)