Why should researchers care about MLOps?

Let’s chat about the importance of Machine Learning Operations (MLOps) for folks like us who primarily focus on models. Why, you ask? Well, let me break it down.

AI development has three phases:

Model-Centric Era:

Also known as the Kaggle period, this phase involves downloading data and initiating development, primarily focusing on creating high-performance models.

Data-Centric Era:

We are currently in this phase, where we emphasize collecting and providing massive amounts of high-quality data to drive machine learning models forward.

AI Operations Optimization Centric Era:

The upcoming phase will concentrate on operating AI systems at scale, focusing on safety, performance monitoring, and bias reduction. In this era, multiple models in production should be modularly integrated, with advantages derived from seamless operations and infrastructure optimization.

Why should researchers care about MLOps?

MLOps is like DevOps but for machine learning models. Its importance for production and industrial use cases is obvious, but why is it important for researchers, also?

• Reproducibility: MLOps keeps things consistent, making it easier to reproduce results and validate findings.
• Collaboration: MLOps helps researchers, data scientists, and engineers work together smoothly, preventing miscommunications.
• Scalability: MLOps lets researchers quickly scale their models and experiments, which is crucial when dealing with large datasets and complex models.
• Efficiency: MLOps saves time and resources by automating many aspects of the machine learning workflow.
• Model Management: MLOps offers tools and techniques to manage the lifecycle of machine learning models, from development to deployment and monitoring.
• Continuous improvement: MLOps encourages a learning culture by providing a framework for iterating on models and experiments.
• Quality assurance: MLOps promotes best practices like thorough testing and validation, ensuring high-quality, reliable models.
• Bias mitigation: MLOps helps identify and mitigate biases in data and models.
• Transparency: MLOps increases transparency by promoting versioning, documentation, and collaboration.
• Inclusiveness: MLOps creates an environment where diverse perspectives can be included in the development process.
• Accountability: MLOps encourages monitoring and auditing tools to track AI systems’ performance and impact.
• Education and awareness: MLOps raises awareness about potential biases and ethical issues associated with AI.
• Policy and regulation alignment: MLOps can help AI systems comply with existing policies and ethical standards.

In short, MLOps helps researchers maintain reproducibility, collaboration, scalability, efficiency, model management, continuous improvement, and quality. MLOps also addresses AI ethicists’ concerns and promotes fairness.

Do you convince that MLOps is important?

So, if you’re unfamiliar with MLOps, don’t worry! It’s easier than you think. Check out some of these resources I found:

Online courses:

• Stanford Machine Learning Systems Design
• Machine Learning Engineering for Production (MLOps) Specialization by Andrew Ng
• Ducke Open Source Platforms for MLOps
• Ducke Python Essentials for MLOps
• Azure ML ops tutorial
• Google MLops tutorial
• Amazon MLops tutorial

Books:

MLOps text books

• Practical MLOps : Operationalizing Machine Learning Models by Gift, Noah; Deza, Alfredo
• Designing Machine Learning Systems: An Iterative Process for Production-ready by Chip Huyen
• Python for Devops: Learn Ruthlessly Effective Automation by Noah Gift , Kennedy Behrman, Alfredo Deza , Grig Gherghiu
• Introducing MLOps: How to Scale Machine Learning in the Enterprise by Mark Treveil , Nicolas Omont , Clement Stenac , Kenji Lefevre, Du Phan
• Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and Mlops by Valliappa Lakshmanan

Podcasts:

The MLOps Podcast: Weekly talks and fireside chats about everything that has to do with getting Machine Learning going in the real world.

Papers:

• Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?
• The Pipeline for the Continuous Development of Artificial Intelligence Models — Current State of Research and Practice
• MLOps: A Primer for Policymakers on a New Frontier in Machine Learning
• Machine Learning Operations (MLOps): Overview, Definition, and Architecture
• Operationalizing AI in Future Networks: A Bird’s Eye View from the System Perspective
• MLOps with enhanced performance control and observability
• Share

Talks About MLOps

Edited by Dr. Larysa Visengeriyeva

1. “MLOps: Automated Machine Learning” by Emmanuel Raj
2. DeliveryConf 2020. “Continuous Delivery For Machine Learning: Patterns And Pains” by Emily Gorcenski
3. MLOps Conference: Talks from 2019
4. Kubecon 2019: Flyte: Cloud Native Machine Learning and Data Processing Platform
5. Kubecon 2019: Running LargeScale Stateful workloads on Kubernetes at Lyft
6. A CI/CD Framework for Production Machine Learning at Massive Scale (using Jenkins X and Seldon Core)
7. MLOps Virtual Event (Databricks)
8. MLOps NY conference 2019
9. MLOps.community YouTube Channel
10. MLinProduction YouTube Channel
11. Introducing MLflow for End-to-End Machine Learning on Databricks. Spark+AI Summit 2020. Sean Owen
12. MLOps Tutorial #1: Intro to Continuous Integration for ML
13. Machine Learning At Speed: Operationalizing ML For Real-Time Data Streams (2019)
14. Damian Brady — The emerging field of MLops
15. MLOps — Entwurf, Entwicklung, Betrieb (INNOQ Podcast in German)
16. Instrumentation, Observability & Monitoring of Machine Learning Models
17. Efficient ML engineering: Tools and best practices
18. Beyond the jupyter notebook: how to build data science products
19. An introduction to MLOps on Google Cloud (First 19 min are vendor-, language-, and framework-agnostic. @visenger)
20. How ML Breaks: A Decade of Outages for One Large ML Pipeline
21. Clean Machine Learning Code: Practical Software Engineering
22. Machine Learning Engineering: 10 Fundamentale Praktiken
23. Architecture of machine learning systems (3-part series)
24. Machine Learning Design Patterns
25. The laylist that covers techniques and approaches for model deployment on to production
26. ML Observability: A Critical Piece in Ensuring Responsible AI (Arize AI at Re-Work)
27. ML Engineering vs. Data Science (Arize AI Un/Summit)
28. SRE for ML: The First 10 Years and the Next 10
29. Demystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
30. Apply Conf 2022
31. Databricks’ Data + AI Summit 2022
32. RE•WORK MLOps Summit 2022
33. Annual MLOps World Conference

Case studies (by Chip Huyen)

Using Machine Learning to Predict Value of Homes On Airbnb (Robert Chang, Airbnb Engineering & Data Science, 2017)

1. In this detailed and well-written blog post, Chang described how Airbnb used machine learning to predict an important business metric: the value of homes on Airbnb. It walks you through the entire workflow: feature engineering, model selection, prototyping, moving prototypes to production. It’s completed with lessons learned, tools used, and code snippets too.
2. Using Machine Learning to Improve Streaming Quality at Netflix (Chaitanya Ekanadham, Netflix Technology Blog, 2018)
3. As of 2018, Netflix streams to over 117M members worldwide, half of those living outside the US. This blog post describes some of their technical challenges and how they use machine learning to overcome these challenges, including to predict the network quality, detect device anomaly, and allocate resources for predictive caching.
4. To understand Netflix’s infrastructure for machine learning, check out Ville Tuulos’s talk Human-Centric Machine Learning Infrastructure @Netflix.
5. 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com (Bernardi et al., KDD, 2019)
6. As of 2019, Booking.com has around 150 machine learning models in production. These models solve a wide range of prediction problems (e.g. predicting users’ travel preferences and how many people they travel with) and optimization problems (e.g.optimizing the background images and reviews to show for each user). Adrian Colyer gave a good summary of the six lessons learned here:

• Machine learned models deliver strong business value.
• Model performance is not the same as business performance.
• Be clear about the problem you’re trying to solve.
• Prediction serving latency matters.
• Get early feedback on model quality.
• Test the business impact of your models using randomized controlled trials.

1. Machine Learning-Powered Search Ranking of Airbnb Experiences (Mihajlo Grbovic, Airbnb Engineering & Data Science, 2019)
2. This article walks you step by step through a canonical example of the ranking and recommendation problem. The four main steps are system design, personalization, online scoring, and business aspect. The article explains which features to use, how to collect data and label it, why they chose Gradient Boosted Decision Tree, which testing metrics to use, what heuristics to take into account while ranking results, how to do A/B testing during deployment. Another wonderful thing about this post is that it also covers personalization to rank results differently for different users.
3. From shallow to deep learning in fraud (Hao Yi Ong, Lyft Engineering, 2018)
4. Fraud detection is one of the industry’s earliest use cases of machine learning. This article explores the evolution of fraud detection algorithms used at Lyft. At first, an algorithm as simple as logistic regression with engineered features was enough to catch most fraud cases. Its simplicity allowed the team to understand the importance of different features. Later, more complex models are required when fraud techniques have become too sophisticated. This article explores the tradeoff between complexity and interpretability, performance and ease of deployment.
5. Space, Time and Groceries (Jeremy Stanley, Tech at Instacart, 2017)
6. Instacart uses machine learning to solve the task of path optimization: how to most efficiently assign tasks for multiple shoppers and find the optimal paths for them. The article explains the system design process, from framing the problem to collecting data, algorithm, and metric selection, topped with a tutorial for beautiful visualization.
7. Creating a Modern OCR Pipeline Using Computer Vision and Deep Learning (Brad Neuberg, Dropbox Engineering, 2017)
8. An application as simple as a document scanner has two distinct components: optical character recognition and word detector. Each requires its own production pipeline, and the end-to-end system requires additional steps for training and tuning. This article also goes into detail the team’s effort to collect data, which includes building their own data annotation platform.
9. Scaling Machine Learning at Uber with Michelangelo (Jeremy Hermann and Mike Del Balso, Uber Engineering, 2019)
10. Uber uses extensive machine learning in their production. This article gives an impressive overview of their end-to-end workflow, where machine learning is being applied at Uber, and how their teams are organized.
11. How we grew from 0 to 4 million women on our fashion app, with a vertical machine learning approach (Gabriel Aldamiz, HackerNoon, 2018)
12. To offer automated outfit advice, Chicisimo tried to qualify people’s fashion tastes using machine learning. Due to the ambiguous nature of the task, the biggest challenges are framing the problem and collecting the data for it, both challenges are addressed by the article. It also covers every consumer app’s problem: user retention.