Shuai Zheng

Shuai Zheng

Senior Applied Scientist
Amazon Web Services
2795 Augustine Dr
Santa Clara, CA 95054, US
szhengac@connect.ust.hk
Github

About Me

I am a Senior Applied Scientist at Amazon Web Services. I received my Ph.D. degree in computer science from the Hong Kong University of Science and Technology. My team works on large-scale distributed machine learning system, compiler and algorithm, with a focus on optimizing the training and inference performance for state-of-the-art deep learning models on big multi-modal data spanning over text, image, tabular, and graph. The team provides system support for many projects at Amazon. We empowered trillion-parameter model training on AWS. Previously, we also explored how cutting-edge natural language processing techniques can advance online advertising and financial services. Our goal is to democratize the creation of state-of-the-art machine learning models while reducing associated human labeling and model training and inference costs.
We have full-time and internship openings for distributed deep learning system, compiler, and algorithm. Do drop me a line if you are interested.

Research Interests

  • Distributed System
  • Large-scale Distributed Algorithm
  • Deep Learning
  • Natural Language Processing

Working Experience

  • Senior Applied Scientist, AWS Deep Learning, Amazon AI
    East Palo Alto, CA, USA, Sep 2019 - Present
  • Applied Scientist Intern, AWS Deep Learning, Amazon AI
    East Palo Alto, CA, USA, Feb 2018 - Aug 2018
  • Research Intern, VIPL Group, Institute of Computing Technology, Chinese Academy of Sciences
    Beijing, China, August 2012 - April 2013

Open Source Software

  • MXNet: A deep learning framework that mixes symbolic and imperative programming to maximize efficiency and productivity.
  • Gluon NLP: GluonNLP is a toolkit that enables easy text preprocessing, datasets loading and neural models building to help you speed up your Natural Language Processing (NLP) research.
  • Slapo: Slapo is a schedule language for progressive optimization of large deep learning model training.

Publications

  1. Decoupled Model Schedule for Deep Learning Training
    Hongzheng Chen, Cody Hao Yu, Shuai Zheng, Zhen Zhang, Zhiru Zhang, Yida Wang
    arXiv:2302.08005, Feb 2023

  2. MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud [Amazon blog] [SageMaker]
    Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, Xin Jin
    To appear in the 49th International Conference on Very Large Data Bases (VLDB), Vancouver, Canada, August 2023

  3. SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning
    M Saiful Bari, Aston Zhang, Shuai Zheng, Xingjian Shi, Yi Zhu, Shafiq Joty, Mu Li
    arXiv:2212.10929, Dec 2022

  4. SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routing
    Chaoyang He, Shuai Zheng, Aston Zhang, George Karypis, Trishul Chilimbi, Mahdi Soltanolkotabi, Salman Avestimehr
    arXiv:2212.05191, Dec 2022

  5. Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition
    Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex Smola, Zhangyang Wang
    The 39th International Conference on Machine Learning (ICML, Long Oral), Baltimore, Maryland, USA, July 2022

  6. Removing Batch Normalization Boosts Adversarial Training
    Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, Zhangyang Wang
    The 39th International Conference on Machine Learning (ICML), Baltimore, Maryland, USA, July 2022

  7. Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
    Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi, Jin Cao, Rakesh Chada, Amit Chauhan, Luoxin Chen, Anurag Dwarakanath, Satyam Dwivedi, Turan Gojayev, Karthik Gopalakrishnan, Thomas Gueudre, Dilek Hakkani-Tur, Wael Hamza, Jonathan Hueser, Kevin Martin, Jose Haidar Khan, Beiye Liu, Jianhua Lu, Alessandro Manzotti, Pradeep Natarajan, Karolina Owczarzak, Gokmen Oz, Enrico Palumbo, Charith Peris, Chandana Satya Prakash, Stephen Rawls, Andy Rosenbaum, Anjali Shenoy, Saleh Soltan, Mukund Harakere Sridhar, Liz Tan, Fabian Triefenbach, Pan Wei, Haiyang Yu, Shuai Zheng, Gokhan Tur, Prem Natarajan
    The 28th ACM SIGKDD Conference (KDD), Washington DC, August, 2022

  8. DCAF-BERT: A Distilled Cachable Adaptable Factorized Model For Improved Ads CTR Prediction
    Aashiq Muhamed, Jaspreet Singh, Shuai Zheng, Iman Keivanloo, Sujan Perera, Jame Mracek, Yi Xu, Qingjun Cui, Sunny Rajagopalan, Belinda Zeng, Trishul Chilimbi
    The 31st ACM Web Conference (WWW), Lyon, France, April 2022

  9. Contractive Error Feedback for Gradient Compression
    Bingcong Li, Shuai Zheng, Parameswaran Raman, Anshumali Shrivastava, Georgios B. Giannakis
    Preprint 2021

  10. Context, Language Modeling, and Multimodal Data in Finance
    Sanjiv Das, Connor Goggins, John He, George Karypis, Sandeep Krishnamurthy, Mitali Mahajan, Nagpurnanand Prabhala, Shenghua Yue, Dylan Slack, Rob van Dusen, Sheng Zha, Shuai Zheng
    Authors listed in alphabetic order
    The Journal of Financial Data Science, Summer 2021

  11. Compressed Communication for Distributed Training: Adaptive Methods and System [code]
    Yuchen Zhong, Cong Xie, Shuai Zheng, Haibin Lin
    Preprint arXiv:2105.07829, May 2021

  12. CSER: Communication-efficient SGD with Error Reset
    Cong Xie, Shuai Zheng, Oluwasanmi Koyejo, Indranil Gupta, Mu Li, Haibin Lin
    The 34th Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, Dec 2020

  13. Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
    Shuai Zheng, Haibin Lin, Sheng Zha, Mu Li
    Preprint arXiv:2006.13484, June 2020

  14. GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
    Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, Aston Zhang, Hang Zhang, Zhi Zhang, Zhongyue Zhang, Shuai Zheng, Yi Zhu
    Authors listed in alphabetic order
    Journal of Machine Learning Research (JMLR), Feb 2020

  15. Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback [code]
    Shuai Zheng, Ziyue Huang, James T. Kwok
    The 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, Dec 2019

  16. Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
    Shuai Zheng, James T. Kwok
    Preprint arXiv:1905.09899, May 2019

  17. Lightweight Stochastic Optimization for Minimizing Finite Sums with Infinite Data
    Shuai Zheng, James T. Kwok
    The 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, July 2018

  18. Follow the Moving Leader in Deep Learning [supplementary]
    Shuai Zheng, James T. Kwok
    The 34th International Conference on Machine Learning (ICML), Sydney, Australia, August 2017

  19. Fast-and-Light Stochastic ADMM [supplementary] [longer arxiv version]
    Shuai Zheng, James T. Kwok
    The 25th International Joint Conference on Artificial Intelligence (IJCAI), New York, New York, USA, July 2016

  20. Fast Nonsmooth Regularized Risk Minimization with Continuation [supplementary]
    Shuai Zheng, Ruiliang Zhang, James T. Kwok
    The 30th AAAI Conference on Artificial Intelligence (AAAI), Phoenix, Arizona, USA, Feb 2016

  21. Asynchronous Distributed Semi-Stochastic Gradient Optimization [supplementary]
    Ruiliang Zhang, Shuai Zheng, James T. Kwok
    The 30th AAAI Conference on Artificial Intelligence (AAAI), Phoenix, Arizona, USA, Feb 2016

  22. Accurate Integration of Aerosol Predictions by Smoothing on a Manifold [code][data]
    Shuai Zheng, James T. Kwok
    The 28th AAAI Conference on Artificial Intelligence (AAAI), Quebec City, Canada, July 2014

  23. Flexible Navigation in Smartphones and Tablets using Scalable Storyboards
    Shuai Zheng, Luis Herranz, Shuqiang Jiang
    The 3rd ACM International Conference on Multimedia Retrieval (ICMR), Dallas, Texas, USA, April 2013

Awards

  • Top Reviewer, NeurIPS 2019, ICML 2020
  • Postgraduate Studentship, HKUST 2015-2019
  • Travel Award, AAAI 2014, IJCAI 2016, ICML (2017, 2018)
  • Undergraduate Scholarship, BJTU 2012

Academic Services

  • PC member of AAAI Conference on Artificial Intelligence (AAAI) 2019 - 2021
  • PC member of Asian Conference on Machine Learning (ACML) 2018 - 2020
  • PC member of ACM SIGKDD Conference (KDD) 2022
  • Reviewer of International Conference on Machine Learning (ICML) 2017 - 2022
  • Reviewer of Neural Information Processing Systems (NeurIPS) 2018 - 2022
  • Reviewer of International Conference on Artificial Intelligence and Statistics (AISTATS) 2019 - 2023
  • Reviewer of International Conference on Learning Representations (ICLR) 2020 - 2022
  • Reviewer of Machine Learning Journal (MLJ)
  • Reviewer of Transactions on Machine Learning Research (TMLR)
  • Reviewer of IEEE Access
  • Reviewer of IEEE Transactions on Automatic Control (IEEE TAC)
  • Reviewer of IEEE Transactions on Signal Processing (IEEE TSP)
  • Reviewer of ACM Transactions on Knowledge Discovery from Data (TKDD)
  • Reviewer of IEEE/ACM Transactions on Networking (TON)
  • Reviewer of IEEE Transactions on Signal and Information Processing over Networks (IEEE TSIPN)
  • Reviewer of IEEE Transactions on Neural Networks and Learning Systems (IEEE TNNLS)
  • Reviewer of IEEE Transactions on Emerging Topics in Computational Intelligence (IEEE TETCI)