Marco Tulio Ribeiro

Source: https://homes.cs.washington.edu/~marcotcr/ Parent: https://idl.uw.edu/papers

I'm a researcher at Google DeepMind. I am also an Affiliate Assistant Professor at the University of Washington, where I was previously a Ph.D student advised by Carlos Guestrin and Sameer Singh.

My research is mostly on helping humans interact with machine learning models meaningfully. That involves interpretability, trust, debugging, feedback, etc.

Despite various attempts, I haven't made much progress on the (much harder) problem of getting a particular group of humans to all look at a camera at the same time →→→

Blog

I wrote these posts on how to pick a project for an intern, but I figured others might be interested too:

• Part 1: Coming up with research ideas
• Part 2: Organizing and evaluating research ideas

I also wrote this one on writing:

• Writing, part 1 — the process

Publications

• Sparks of artificial general intelligence: Early experiments with GPT-4\ Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Yi Zhang
• ScatterShot: Interactive In-context Example Curation for Text Transformation\ Tongshuang Wu, Hua Shen, Daniel Weld, Jeffrey Heer, Marco Tulio Ribeiro\ In: International Conference on Intelligent User Interfaces (IUI), 2023\ Best Paper Honorable mention
• ART: Automatic multi-step reasoning and tool-use for large language models\ Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi,\ Luke Zettlemoyer, Marco Tulio Ribeiro\ In submission
• Editing Models with Task Arithmetic\ Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi\ In: International Conference on Learning Representations (ICLR), 2023\ [code]
• Adaptive Testing and Debugging of NLP Models\ Marco Tulio Ribeiro*, Scott Lundberg* (Equal contribution)\ In: Association for Computational Linguistics (ACL), 2022\ [code] [bibtex]
• Fixing Model Bugs with Natural Language Patches\ Shikhar Murty, Christopher D. Manning, Scott Lundberg, Marco Tulio Ribeiro\ In: Empirical Methods in Natural Language Processing (EMNLP), 2022
• Finding and Fixing Spurious Patterns with Explanations\ Gregory Plumb, Marco Tulio Ribeiro, Ameet Talwalkar\ In: Transactions on Machine Learning Research (TMLR), 2022
• ExSum: From Local Explanations to Model Understanding\ Yilun Zhou, Marco Tulio Ribeiro, Julie Shah\ In: Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), 2022\ [code]
• What Did My AI Learn? How Data Scientists Make Sense of Model Behavior\ Ángel Alexander Cabrera, Marco Tulio Ribeiro, Bongshin Lee, Rob DeLine, Adam Perer, Steven M. Drucker\ In: ACM Transactions on Computer-Human Interaction (TOCHI), 2022\ [bibtex]
• Do Feature Attribution Methods Correctly Attribute Features?\ Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, Julie Shah\ In: AAAI Conference on Artificial Intelligence (AAAI), 2022\ [code]
• Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance \ Gagan Bansal*, Tongshuang Wu*, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld\ In: CHI 2021: the 2021 Conference on Human Factors in Computing Systems
• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models\ Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel Weld\ In: Association for Computational Linguistics (ACL), 2021\ [code] [bibtex]
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList\ Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh.\ In: Association for Computational Linguistics (ACL), 2020\ Best Paper Award\ [code] [talk] [slides] [longer slides] [bibtex]
• SQuINTing at VQA Models: Interrogating VQA Models with Sub-Questions\ Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz,\ Marco Tulio Ribeiro, Besmira Nushi, Ece Kamar.\ In: Conference on Computer Vision and Pattern Recognition (CVPR), 2020\ [bibtex]
• Errudite: Scalable, Reproducible, and Testable Error Analysis\ Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel Weld.\ In: Association for Computational Linguistics (ACL), 2019\ [code] [bibtex] [blog]
• Are Red Roses Red? Evaluating Consistency of Question-Answering Models\ Marco Tulio Ribeiro, Carlos Guestrin, Sameer Singh.\ In: Association for Computational Linguistics (ACL), 2019\ [code] [bibtex]
• Semantically Equivalent Adversarial Rules for Debugging NLP Models\ Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.\ In: Association for Computational Linguistics (ACL), 2018\ Honorable mention for best paper award\ [code] [talk] [slides] [bibtex]
• Anchors: High-Precision Model-Agnostic Explanations\ Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.\ In: AAAI Conference on Artificial Intelligence (AAAI), 2018\ [code] [slides] [bibtex]
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier\ Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.\ In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016\ Audience appreciation award [video]\ [code] [talk] [slides] [bibtex] [blog]
• Model-Agnostic Interpretability of Machine Learning\ Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin.\ In: ICML Workshop on Human Interpretability in Machine Learning (WHI), 2016\ Best paper award\ [bibtex]