While Uber, Google, Facebook and other big tech companies seem to be succeeding with getting machine learning models into production as useful guidance for internal and external users, many smaller companies struggle. They invest significant dollars and time in data science teams but then fail to see anything useful make it to production.

In this article, I outline a common mode of machine learning / data science failure: when machine learning models get only as far as PowerPoint.

What a machine learning lifecycle should look like

Here’s a theoretical machine learning development, deployment, and improvement lifecycle:

In this idealized lifecycle, a domain expert or product manager will identify up front what sort of guidance your ML-powered capability is intended to provide. Predictions on their own are typically not very useful. This first step must be informed by data availability: no data, no guidance. You will survey internal and external sources of data to evaluate whether you have the data to get started.

If you don’t have appropriate data already, you can design the system to bootstrap itself, starting with heuristic-based guidance and then generating a labeled dataset as users interact with the new feature.

Data science teams tend to excel at the second two steps in this process: preparing data and training algorithms. No need to dwell on this.

Deployment of the model involves more than just serving predictions. It will be wrapped up in a guidance-oriented feature. Predictions must be processed to be useful to people, and must be served at the right time in the right way.

But wait! You’re not done when the model has been deployed. Everyone knows the model may drift, and you will want to retrain it with updated data at regular intervals. Also, you’ll want to design in capture and curation of the labels on your dataset. Note the feedback loop in this process. That need for model and dataset feedback makes ML-powered features significantly more complex than building traditional software features.

This lifecycle is largely theoretical. It is complex to get right, and most businesses starting out with machine learning don’t hire the right staff or organize them in the right way to enable it. More on that in a later post.

PowerPoint deployment?

Unfortunately, in the real world here’s what machine learning development and “deployment” often looks like:

In this mode, your data science team finds a labeled dataset that they can make predictions from, trains predictive models using a variety of algorithms, and then shares results via PowerPoint. These presentations may actually be pretty interesting. You’ll get a read on how much signal there is in the dataset for producing predictions and sometimes the underlying model can provide some insight into important correlational patterns in your business. But rarely do these models ever get wrapped up into guidance in your application. And even more rarely are feedback loops for capturing new training data and new training data labels implemented.

Your data scientists may bring a lot of creativity and machine learning expertise to their model training. Data science as craft drives the powerpoint model failure mode.

Succeeding with ML-powered features: Let’s find a better way

We’re looking for a new path to success with machine learning: using relatively off-the-shelf modeling capabilities where model training and accuracy evaluation is handled by the computer, not by a data science artisan. We believe that the more important steps in producing machine learning to create real value are in defining the guidance that the ML model will provide, and in designing in feedback loops so that the ML capability can produce better and better guidance over time.

If you want to follow us on our quest to liberate the power of machine learning from the confines of PowerPoint, sign up for our mailing list: