Publications

Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards.
Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christopher A. Choquette-Choo, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, Ion Stoica, Florian Tramer, Chiyuan Zhang.
Preprint, 2025. arXiv:2501.07493

Balls-and-Bins Sampling for DP-SGD.
Lynn Chua, Badih Ghazi, Charlie Harrison, Ethan Leeman, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang.
International Conference on Artificial Intelligence and Statistics (AISTATS), 2025. arXiv:2412.16802

Unlearn and Burn: Adversarial Machine Unlearning Requests Destroy Model Accuracy.
Yangsibo Huang, Daogao Liu, Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Milad Nasr, Amer Sinha, Chiyuan Zhang.
International Conference on Learning Representations (ICLR), 2025. arXiv:2410.09591

MUSE: Machine Unlearning Six-Way Evaluation for Language Models.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, Chiyuan Zhang.
International Conference on Learning Representations (ICLR), 2025. arXiv:2407.06460 project website

Fantastic Copyrighted Beasts and How (Not) to Generate Them.
Luxi He, Yangsibo Huang, Weijia Shi, Tinghao Xie, Haotian Liu, Yue Wang, Luke Zettlemoyer, Chiyuan Zhang, Danqi Chen, Peter Henderson.
International Conference on Learning Representations (ICLR), 2025. arXiv:2406.14526 project website

On Memorization of Large Language Models in Logical Reasoning.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, Ravi Kumar.
Preprint, 2024. arXiv:2410.23123 website code data

Scalable DP-SGD: Shuffling vs. Poisson Subsampling.
Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang.
Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2411.04205

Evaluating Copyright Takedown Methods for Language Models.
Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A. Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, Peter Henderson.
Advances in Neural Information Processing Systems, Datasets and Benchmarks Track (NeurIPS), 2024. arXiv:2406.18664 project website

On Convex Optimization with Semi-Sensitive Features.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Raghu Meka, Chiyuan Zhang.
Conference on Learning Theory (COLT), 2024. arXiv:2406.19040

Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models.
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chulin Xie, Chiyuan Zhang.
Preprint, 2024. arXiv:2406.16135 code

Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning.
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Daogao Liu, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang.
Conference on Language Modeling (COLM), 2024. arXiv:2406.14322

How Private are DP-SGD Implementations?
Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang.
International Conference on Machine Learning (ICML, Oral), 2024. arXiv:2403.17673

Localizing Paragraph Memorization in Language Models?
Niklas Stoehr, Mitchell Gordon, Chiyuan Zhang, Owen Lewis.
Preprint, 2024. arXiv:2403.19851

LabelDP-Pro: Learning with Label Differential Privacy via Projections.
Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Chiyuan Zhang.
International Conference on Learning Representations (ICLR), 2024. PDF

Training Differentially Private Ad Prediction Models with Semi-Sensitive Features.
Lynn Chua, Qiliang Cui, Badih Ghazi, Charlie Harrison, Pritish Kamath, Walid Krichene, Ravi Kumar, Pasin Manurangsi, Krishna Giri Narra, Amer Sinha, Avinash Varadarajan, Chiyuan Zhang.
The 5th AAAI Workshop on Privacy-Preserving Artificial Intelligence (PPAI), 2024. arXiv:2401.15246

Optimal Unbiased Randomizers for Regression with Label Differential Privacy.
Ashwinkumar Badanidiyuru, Badih Ghazi, Pritish Kamath, Ravi Kumar, Ethan Leeman, Pasin Manurangsi, Avinash V Varadarajan, Chiyuan Zhang.
Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2312.05659

Sparsity-Preserving Differentially Private Training of Large Embedding Models.
Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang.
Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2311.08357

User-Level Differential Privacy With Few Examples Per User.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Raghu Meka, Chiyuan Zhang.
Advances in Neural Information Processing Systems (NeurIPS, Oral), 2023. arXiv:2309.12500

Counterfactual Memorization in Neural Language Models.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, Nicholas Carlini.
Advances in Neural Information Processing Systems (NeurIPS, Spotlight), 2023. arXiv:2112.12938

Ticketed Learning-Unlearning Schemes.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Ayush Sekhari, Chiyuan Zhang.
Conference on Learning Theory (COLT), 2023. arXiv:2306.15744

Can Neural Network Memorization Be Localized?
Pratyush Maini, Michael C. Mozer, Hanie Sedghi, Zachary Lipton, Zico Kolter, Chiyuan Zhang.
International Conference on Machine Learning (ICML), 2023. arXiv:2307.09542

On User-Level Private Convex Optimization.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Raghu Meka, Pasin Manurangsi, Chiyuan Zhang.
International Conference on Machine Learning (ICML), 2023. arXiv:2305.04912

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy.
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, Nicholas Carlini.
International Natural Language Generation Conference (INLG), 2023. arXiv:2210.17546

Private Ad Modeling with DP-SGD.
Carson Denison, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Krishna Giri Narra, Amer Sinha, Avinash Varadarajan, Chiyuan Zhang.
AdKDD, 2023. arXiv:2211.11896

Regression with Label Differential Privacy.
Badih Ghazi, Pritish Kamath, Ravi Kumar, Ethan Leeman, Pasin Manurangsi, Avinash Varadarajan, Chiyuan Zhang.
The International Conference on Learning Representations (ICLR), 2023. arXiv:2212.06074

Quantifying Memorization Across Neural Language Models.
Nicholas Carlini♮, Daphne Ippolito♮, Matthew Jagielski♮, Katherine Lee♮, Florian Tramer♮, Chiyuan Zhang♮. ♮equal contribution.
The International Conference on Learning Representations (ICLR, top-25%), 2023. arXiv:2202.07646 Data

Measuring Forgetting of Memorized Training Examples.
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, Chiyuan Zhang.
The International Conference on Learning Representations (ICLR), 2023. arXiv:2207.00099

Just Fine-tune Twice: Selective Differential Privacy for Large Language Models.
Weiyan Shi, Si Chen, Chiyuan Zhang, Ruoxi Jia, Zhou Yu.
Empirical Methods in Natural Language Processing (EMNLP), 2022. arXiv:2204.07667

The Privacy Onion Effect: Memorization is Relative.
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, Florian Tramer.
Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2206.10469

Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures?
Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia, Jon Kleinberg, Aryo Lotfi, Maithra Raghu, Chiyuan Zhang.
Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2205.13647

Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation.
Yao Qin, Chiyuan Zhang, Ting Chen, Balaji Lakshminarayanan, Alex Beutel, Xuezhi Wang.
Advances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2110.07858

Are All Layers Created Equal?
Chiyuan Zhang, Samy Bengio, Yoram Singer.
Journal of Machine Learning Research (JMLR), 2022. arXiv:1902.01996 JMLR

Deduplicating Training Data Makes Language Models Better.
Katherine Lee♮, Daphne Ippolito♮, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini. ♮equal contribution.
The 60th Annual Meeting of the Association for Computational Linguistics (ACL, Oral), 2022. arXiv:2107.06499 code

Do Vision Transformers See Like Convolutional Neural Networks?
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy.
Advances in Neural Information Processing Systems (NeurIPS), 2021. arXiv:2108.08810

Deep Learning with Label Differential Privacy.
Badih Ghazi♮, Noah Golowich♮, Ravi Kumar♮, Pasin Manurangsi♮, Chiyuan Zhang♮. ♮equal contribution.
Advances in Neural Information Processing Systems (NeurIPS), 2021. arXiv:2102.06062 code

Characterizing Structural Regularities of Labeled Data in Overparameterized Models.
Ziheng Jiang♮, Chiyuan Zhang♮, Kunal Talwar, Michael C. Mozer. ♮equal contribution.
International Conference on Machine Learning (ICML, Long presentation (3%)), 2021. arXiv:2002.03206 project website code

Understanding invariance via feedforward inversion of discriminatively trained classifiers.
Piotr Teterwak, Chiyuan Zhang, Dilip Krishnan, Michael C. Mozer.
International Conference on Machine Learning (ICML), 2021. arXiv:2103.07470 project website

Understanding Deep Learning (Still) Requires Rethinking Generalization.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals.
Communications of the ACM, March 2021, Vol. 64 No. 3, Pages 107-115. (Republication of our ICLR 2017 papar as CACM Research Highlights). Full Article Media Technical Perspective

What is being transferred in transfer learning?.
Behnam Neyshabur♮, Hanie Sedghi♮, Chiyuan Zhang♮. ♮equal contribution.
Advances in Neural Information Processing Systems (NeurIPS), 2020. arXiv:2008.11687

What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation.
Vitaly Feldman♮, Chiyuan Zhang♮. ♮equal contribution.
Advances in Neural Information Processing Systems (NeurIPS, Spotlight (4%)), 2020. arXiv:2008.03703 project website

Identity Crisis: Memorization and Generalization under Extreme Overparameterization.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, Yoram Singer.
The International Conference on Learning Representations (ICLR), 2020. arXiv:1902.04698

Transfusion: Understanding Transfer Learning for Medical Imaging.
Maithra Raghu♮, Chiyuan Zhang♮, Jon Kleinberg♭, Samy Bengio♭. ♮equal contribution; ♭equal contribution.
Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1902.07208

Unrestricted Adversarial Examples.
Tom B. Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, Ian Goodfellow.
Preprint 2018. arXiv:1809.08352

A Study on Overfitting in Deep Reinforcement Learning.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, Samy Bengio.
Preprint 2018. arXiv:1804.06893

Machine Theory of Mind.
Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S.M. Ali Eslami, Matthew Botvinick.
International Conference on Machine Learning (ICML), 2018. arXiv:1802.07740

Automated fault detection without seismic processing.
Mauricio Araya-Polo, Taylor Dahlke, Charlie Frogner, Chiyuan Zhang, Tomaso Poggio, and Detlef Hohl.
The Leading Edge (TLE), Society of Exploration Geophysicists (SEG), 2017. DOI:10.1190/tle36030208.1

Understanding Deep Learning Requires Rethinking Generalization.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals.
International Conference on Learning Representations (ICLR), Best Paper Award, 2017. arXiv:1611.03530 code

Training Deep Nets with Sublinear Memory Cost.
Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin.
Preprint 2016. arXiv:1604.06174 code

Learning with a Wasserstein Loss.
Charlie Frogner♮, Chiyuan Zhang♮, Hossein Mobahi, Mauricio Araya-Polo, Tomaso Poggio. ♮equal contribution.
Advances in Neural Information Processing Systems (NeurIPS), 2015. arXiv:1506.05439 project website MIT News

MXNet: A Distributed Deep Learning Framework for Efficiency and Flexibility.
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang.
NeurIPS Workshop on LearningSys, 2015. arXiv:1512.01274 code

A-Optimal Projection for Image Representation.
Xiaofei He, Chiyuan Zhang, Lijun Zhang, Xuelong Li.
IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), 2015.

Discriminative Template Learning in Group-Convolutional Networks for Invariant Speech Representations.
Chiyuan Zhang, Stephen Voinea, Georgios Evangelopoulos, Lorenzo Rosasco, Tomaso Poggio.
INTERSPEECH 2015.

Phone Classification by a Hierarchy of Invariant Representation Layers.
Chiyuan Zhang, Stephen Voinea, Georgios Evangelopoulos, Lorenzo Rosasco, Tomaso Poggio.
INTERSPEECH 2014.

Word-level Invariant Representations from Acoustic Waveforms.
Stephen Voinea, Chiyuan Zhang, Georgios Evangelopoulos, Lorenzo Rosasco, Tomaso Poggio.
INTERSPEECH, Best Student Paper, 2014.

Learning An Invariant Speech Representation.
Georgios Evangelopoulos, Stephen Voinea, Chiyuan Zhang, Lorenzo Rosasco, Tomaso Poggio.
CBMM Memo No. 22, 2014. arXiv:1406.3884

A Deep Representation for Invariance and Music Classification.
Chiyuan Zhang, Georgios Evangelopoulos, Stephen Voinea, Lorenzo Rosasco, Tomaso Poggio.
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.

Machine-learning Based Automated Fault Detection in Seismic Traces.
Chiyuan Zhang, Charlie Frogner, Mauricio Araya-Polo, Detlef Hohl.
Proceedings of 76th European Association of Geoscientists and Engineers Conference & Exhibition (EAGE), 2014. PDF

Parallel Vector Field Embedding.
Binbin Lin, Xiaofei He, Chiyuan Zhang, Ming Ji.
Journal of Machine Learning Research (JMLR), 2013. JMLR

Image Compression by Learning to Minimize the Total Error.
Chiyuan Zhang, Xiaofei He.
IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2013.

Multi-task Vector Field Learning.
Binbin Lin, Sen Yang, Chiyuan Zhang, Jieping Ye, Xiaofei He.
Advances in Neural Information Processing Systems (NeurIPS), 2012. paper

Semi-supervised Regression via Parallel Field Regularization.
Binbin Lin, Chiyuan Zhang, Xiaofei He.
Advances in Neural Information Processing Systems (NeurIPS), 2011. paper

A Variance Minimization Criterion to Feature Selection using Laplacian Regularization.
Xiaofei He, Ming Ji, Chiyuan Zhang.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2011.

Unsupervised feature selection for multi-cluster data.
Deng Cai, Chiyuan Zhang, Xiaofei He.
Proc. of the 16th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD), 2010.