How to Address the ML Deployment Pipeline Bottleneck at Enterprise Organizations
As an Enterprise Data Scientist for a large organization, your AI roadmaps and project timelines to deliver on end to end AI solutions will depend on your organization’s ML deployment pipeline.
I wrote and shared another related article on LinkedIn, on the need for data scientists at large organizations to focus on building complete ML systems. One executive responded by agreeing with this reframing the focus from ML algorithms to building complete ML systems. But they pointed out that they see problems with the dev and engineering teams building the ML deployment pipeline, holding up AI projects looking to deploy their ML models in production, at scale. This bottleneck in turn dominates their AI roadmaps and increases the timelines to deliver on end to end AI solutions.
But this is a huge problem, especially at large organizations, this ML deployment pipeline. Building a complete ML system, from the technology perspective, faces further barriers when the organization doesn’t have an adequate data science stack/platform on which to develop, test, and deploy ML models in production. Serving ML models at scale is a huge endeavour indeed.
Rather than building an entire data science platform from scratch, one way to build a data science platform quickly is to utilize one of the major cloud vendors, AWS, Azure, or GCP, and utilize their Platform as a Service (PaaS). You can leverage their PaaS, where they do all of the heavy lifting and deliver the hardware and software to users via the cloud, and scale your infrastructure and capacity according to the needs of your organization. I detail the guiding principles for building an Enterprise Data Science platform here.
Once you procure a cloud vendor, then it’s time for the company data scientist, ML engineer, and ML architect to work with the vendor cloud service provider to architect the high level conceptual diagrams to include the following:
Data pipeline (data engineer)
ML pipeline (data scientist, ML engineer)
Deployment pipeline (ML engineer, cloud engineer, MLOps engineer)
It is advisable to base your data pipeline on a data lakehouse, which combines the capabilities of a data lake and data warehouse. Then you can add the various Big Data ETL and processing services, ML services, and AI services (prebuilt models), customized for your company use cases. If the AI/ML use cases are not well delineated, then just build a Minimum Viable Platform, focusing on just the ML services (or even a VM with data science tools) and foregoing the more expensive AI services. With a Minimum Viable Platform, you can always circle back and add services/components when the need arises with future AI/ML use cases.
Once you have your organization’s data science platform in the cloud, then you can start building AI prototypes on it, bringing company use cases and data, and exploring the capabilities. This AI prototyping will get you 2 out of the 3 pipelines you will need for end to end AI solutions: it will give you the 1) data pipeline for data movement, data provisioning, and data processing, and the 2) ML pipeline for feature engineering, model training, and model metrics/testing.
When you’re at this stage, pat yourselves and the executives on the back, as you have made progress towards your goal of building your Enterprise Data Science platform. So the only thing you’re missing is the deployment pipeline. At this point, the data scientist lead should work closely with the ML engineer, cloud engineer, and MLOps engineer to serve your ML models in production, at scale.
You will also need an application team to build a system to consume the outputs of your ML model in production. You can either serve it to a web application as an interactive dashboard for your clients, or have a system consume the outputs to make automated decisions.
In the cloud, this last step, the deployment pipeline, is much easier to build, as you have PaaS and let the vendor do all of the heavy lifting. If you stay on premises and build your deployment pipeline from scratch, you run the risk of never building one, as it requires deep technical knowledge and the infrastructure required will never keep up with increasing demand/use by the data scientists in your company. Even if you were successful at building the deployment pipeline on prem, you will always be behind the 8 ball, as you will never have enough horsepower, and your AI tools/packages will constantly be outdated.
A good way to ensure your dev and engineering teams are in the loop for model guardrails, endpoints, training metrics, and deployment metrics, is to embed a senior data scientist on that IT team. This data scientist will guide the developers and engineers as they move from one end of the pipeline, to the other end, ensuring the integrity of the data science lifecycle, and the integrity of the deployed ML model.
In summary, to address the ML deployment pipeline bottleneck, build your data science platform on the cloud with PaaS, reuse the data pipeline and ML pipeline you already built for the AI prototypes you built successfully on your cloud platform, and embed a senior data scientist on your IT dev and engineering teams. Enterprise Data Scientists are not just for modeling- they can also guide an IT dev and engineering team as they build end to end AI solutions.
If you're looking for support from me, here are a few options:
Enterprise Data Science Consultancy: With my consult team comprised of a Senior Data Scientist/ML Engineer, Senior Data Engineer, and Senior Cloud Engineer, we will help you architect and build your Enterprise Data Science platform, and transfer knowledge to your IT team to maintain and optimize it. If you don’t have an MLOps team, I will help you build one. Please get in touch about this consultancy here
Coaching and Mentorship: I offer coaching and mentorship; book a coaching session here