Building an MLOps Pipeline in 3 Weeks — a Paint by Numbers Approach

I will present you with a general framework for how to build up your own MLOps Pipeline, maybe not in 3 weeks, but in a scalable way that might have its first MVP in 3 weeks.

This article is split in two, we will begin with the outline, a gradual discovery of the general concept which is followed by a look at the implementation we achieved at a client.

Scope

MLOps is a gigantic topic, combining DevOps with all its concepts and ML as a new and rapidly developing field. It is therefor important to focus intently. In this article we will:

look at:

• the infrastructure components
• general processes
• example for the components we require

ignore:

• details of software frameworks
• code examples

I aim to provide you with a general framework for MLOps, onto which you can build your future knowledge and fill all the gaps that interest you.

Outline

The Basics

Every MLOps pipeline needs to implement (at a minimum) these 3 stages which will repeat with every iteration:

1. build the dataset
2. train the model
3. deploy the model

Each of these stages again consist of smaller steps.

When building your dataset you need to fetch the data, clean it and prepare it for your specific model.

Your training should also contain a step to evaluate and ensure you only deploy models that perform better than your previous model and lastly your deployment. It can be either a permanent deployment or batch inference, but should also include monitoring, as your model might perform well on the current requests and given dataset but might perform worse over time (data drift).

It is important to note that we have 2 lifecycles in this simple diagram.

One data lifecycle containing the fetch, build and prepare steps and a model lifecycle including the training and deployment of your model.

The Concept

In this approach we want one source of behaviour, the version control system in combination with an orchestration solution. As data changes with time, but the application only changes with commits, and we only want to deploy changed artefacts it is sensible to have have two separate lifecycles for the data and the model.

Each lifecycle iteration is started by a different event, the data changes with time, where as the application only changes with commits, but both end in the generation of new artefacts. For this reason we have two pipelines in our orchestration solution. The data pipeline can be triggered periodically and in turn trigger the complete downstream model pipeline and the trainings pipeline can additionally be triggered by merges to the main branch, following a feature branch approach.

In advanced setups you might choose to even deploy your model fully automatically, this however is often just an aspirational goal for companies, as you need extremely high confidence in your models, engineers, and testing pipeline. Instead you will often trigger the deployment of carefully selected models manually.

You will also need:

1. a tool or tool collection to generate your dataset
2. a tool to store and version your datasets
3. an ML runtime; somewhere to train your models and run your inferences, which ideally handles the provisioning of resources as well and has wide hardware support, this means integrating with your cloud / compute provider.
4. a metrics server to ensure your models quality
5. a model registry to version your promoted models and ensure a continued lineage
6. a docker registry is a de facto prerequisite, but you will also need it to version your model runtime during the metric collection to again ensure an unbroken lineage.

Putting all this together we arrive at something like the image below:

In our paint by numbers approach above is our outline, now we need a palette and a selection of colours / tools. Our palette might look like this, by far not complete, but enough to demonstrate the concept:

How you select components from this list highly depends on your specific organisational needs. In our case the goal was to build something very quickly with few new resources that could be built out later, so reusing or even misusing technologies to move very quickly was acceptable, if it meant reducing the number of new tools we needed to deploy.

Mini Case Study

The following is not necessarily a best practice, but simply a showcase of what has worked in the past and the reasoning behind it. As we worked on AWS and were already using GitLab, ECR, Sagemaker, S3, Athena and Glue we arrived at this set of tools

Note that we are using GitlabCI as an orchestration server. This is not ideal, but it can function to a limited degree as one, you're missing out on a lot of features of other orchestration services, but it allows you to start and manage tasks in your ML runtime environment, which is all we need.

It was however the first component replaced later.

This meant we only need DVC as an additional component, which is a tool that uses GIT and some blob storage (S3 in our case) to version datasets and does not require any kind of infrastructure deployment. Everything else was already in use and deployed which is what made our timeline possible.

And so we arrive at a high-level overview like this:

Two pipelines, triggered independently and only interacting with each other by first generating and then utilising a shared dataset. The data pipeline uses AWS S3, Glue and Athena to generate a dataset, which is versioned and stored in DVC. This dataset is then retrieved after a commit is made on the main branch of our model repository. This triggers a sagemaker trainings job, which pulls the current code from git, to train our new model and compare the metrics against previous runs. If the new model performs better than the old one, the runtime (docker image) is tagged with the id of the current pipeline, as is the model itself and pushed to the sagemaker model registry. And finally the new model is deployed into production.

This high level of automation allowed the team, consisting of 3 members, to continuously train and deploy better version of their two productive services with confidence and still develop new services at the same time.

This article was originally published on Medium — AWS in Plain English in October 2023.