Papers

• Lancet: Accelerating Mixture-of-Experts Training by Overlapping Weight Gradient Computation and All-to-All Communication
  Chenyu Jiang, Ye Tian, Zhen Jia, Chuan Wu, Yida Wang, Shuai Zheng
  The 7th Annual Conference on Machine Learning and Systems (MLSys), Santa Clara, CA, USA, May 2024
  

• DISTMM: Accelerating Distributed Multi-modal Model Training
  Jun Huang, Zhen Zhang, Feng Qin, Yida Wang, Shuai Zheng
  The 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI), Santa Clara, CA, USA, April 2024
  

• VCC: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens
  Zhanpeng Zeng, Cole Hawkins, Mingyi Hong, Aston Zhang, Nikolaos Pappas, Vikas Singh, Shuai Zheng
  The 37th Conference on Neural Information Processing Systems (NeurIPS), New Orleans, USA, Dec 2023
  

• Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition
  Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang, Shuai Zheng, Mu Li, Alex Smola, Xu Sun
  The 37th Conference on Neural Information Processing Systems (NeurIPS), New Orleans, USA, Dec 2023
  

• DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines
  Chenyu Jiang, Zhen Jia, Shuai Zheng, Yida Wang, Chuan Wu
  The 19th European Conference on Computer Systems (EuroSys), Athens, Greece, April 2024
  

• GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints
  Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, Yida Wang
  The 29th ACM Symposium on Operating Systems Principles (SOSP), Koblenz, Germany, October 2023
  

• Slapo: A Schedule Language for Efficient Progressive Optimization of Large Deep Learning Model Training
  Hongzheng Chen, Cody Hao Yu, Shuai Zheng, Zhen Zhang, Zhiru Zhang, Yida Wang
  The ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), San Diego, CA, USA, April 2024
  

• MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud [Amazon blog] [SageMaker]
  Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, Xin Jin
  The 49th International Conference on Very Large Data Bases (VLDB), Vancouver, Canada, August 2023
  

• SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning
  M Saiful Bari, Aston Zhang, Shuai Zheng, Xingjian Shi, Yi Zhu, Shafiq Joty, Mu Li
  arXiv:2212.10929, Dec 2022
  

• SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
  Chaoyang He, Shuai Zheng, Aston Zhang, George Karypis, Trishul Chilimbi, Mahdi Soltanolkotabi, Salman Avestimehr
  arXiv:2212.05191, Dec 2022
  

• Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition
  Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex Smola, Zhangyang Wang
  The 39th International Conference on Machine Learning (ICML, Long Oral), Baltimore, Maryland, USA, July 2022
  

• Removing Batch Normalization Boosts Adversarial Training
  Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, Zhangyang Wang
  The 39th International Conference on Machine Learning (ICML), Baltimore, Maryland, USA, July 2022
  

• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
  Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi, Jin Cao, Rakesh Chada, Amit Chauhan, Luoxin Chen, Anurag Dwarakanath, Satyam Dwivedi, Turan Gojayev, Karthik Gopalakrishnan, Thomas Gueudre, Dilek Hakkani-Tur, Wael Hamza, Jonathan Hueser, Kevin Martin, Jose Haidar Khan, Beiye Liu, Jianhua Lu, Alessandro Manzotti, Pradeep Natarajan, Karolina Owczarzak, Gokmen Oz, Enrico Palumbo, Charith Peris, Chandana Satya Prakash, Stephen Rawls, Andy Rosenbaum, Anjali Shenoy, Saleh Soltan, Mukund Harakere Sridhar, Liz Tan, Fabian Triefenbach, Pan Wei, Haiyang Yu, Shuai Zheng, Gokhan Tur, Prem Natarajan
  The 28th ACM SIGKDD Conference (KDD), Washington DC, August, 2022
  

• DCAF-BERT: A Distilled Cachable Adaptable Factorized Model For Improved Ads CTR Prediction
  Aashiq Muhamed, Jaspreet Singh, Shuai Zheng, Iman Keivanloo, Sujan Perera, Jame Mracek, Yi Xu, Qingjun Cui, Sunny Rajagopalan, Belinda Zeng, Trishul Chilimbi
  The 31st ACM Web Conference (WWW), Lyon, France, April 2022
  

• Contractive Error Feedback for Gradient Compression
  Bingcong Li, Shuai Zheng, Parameswaran Raman, Anshumali Shrivastava, Georgios B. Giannakis
  Preprint 2021
  

• Context, Language Modeling, and Multimodal Data in Finance
  Sanjiv Das, Connor Goggins, John He, George Karypis, Sandeep Krishnamurthy, Mitali Mahajan, Nagpurnanand Prabhala, Shenghua Yue, Dylan Slack, Rob van Dusen, Sheng Zha, Shuai Zheng
  Authors listed in alphabetic order
  The Journal of Financial Data Science, Summer 2021
  

• Compressed Communication for Distributed Training: Adaptive Methods and System [code]
  Yuchen Zhong, Cong Xie, Shuai Zheng, Haibin Lin
  Preprint arXiv:2105.07829, May 2021
  

• CSER: Communication-efficient SGD with Error Reset
  Cong Xie, Shuai Zheng, Oluwasanmi Koyejo, Indranil Gupta, Mu Li, Haibin Lin
  The 34th Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, Dec 2020
  

• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
  Shuai Zheng, Haibin Lin, Sheng Zha, Mu Li
  Preprint arXiv:2006.13484, June 2020
  

• GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
  Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, Shuai Zheng, Yi Zhu
  Authors listed in alphabetic order
  Journal of Machine Learning Research (JMLR), Feb 2020
  

• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback [code]
  Shuai Zheng, Ziyue Huang, James T. Kwok
  The 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, Dec 2019
  

• Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
  Shuai Zheng, James T. Kwok
  Preprint arXiv:1905.09899, May 2019
  

• Lightweight Stochastic Optimization for Minimizing Finite Sums with Infinite Data
  Shuai Zheng, James T. Kwok
  The 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, July 2018
  

• Follow the Moving Leader in Deep Learning [supplementary]
  Shuai Zheng, James T. Kwok
  The 34th International Conference on Machine Learning (ICML), Sydney, Australia, August 2017
  

• Fast-and-Light Stochastic ADMM [supplementary] [longer arxiv version]
  Shuai Zheng, James T. Kwok
  The 25th International Joint Conference on Artificial Intelligence (IJCAI), New York, New York, USA, July 2016
  

• Fast Nonsmooth Regularized Risk Minimization with Continuation [supplementary]
  Shuai Zheng, Ruiliang Zhang, James T. Kwok
  The 30th AAAI Conference on Artificial Intelligence (AAAI), Phoenix, Arizona, USA, Feb 2016
  

• Asynchronous Distributed Semi-Stochastic Gradient Optimization [supplementary]
  Ruiliang Zhang, Shuai Zheng, James T. Kwok
  The 30th AAAI Conference on Artificial Intelligence (AAAI), Phoenix, Arizona, USA, Feb 2016
  

• Accurate Integration of Aerosol Predictions by Smoothing on a Manifold [code][data]
  Shuai Zheng, James T. Kwok
  The 28th AAAI Conference on Artificial Intelligence (AAAI), Quebec City, Canada, July 2014
  

• Flexible Navigation in Smartphones and Tablets using Scalable Storyboards
  Shuai Zheng, Luis Herranz, Shuqiang Jiang
  The 3rd ACM International Conference on Multimedia Retrieval (ICMR), Dallas, Texas, USA, April 2013