An Engineer’s View: How to Pick an Automated Machine Learning Product

You are the Director of Technology at a Fortune 500 company and have been recently put in charge of product selections. You are asked to select a vendor product for a relatively new, yet fast maturing technology called machine learning — specifically for a product that does automated machine learning. This blog, written from an engineer’s perspective, will walk you through some useful steps to assist you in the product selection process.

An automated machine learning product needs to bring high value to your company.

Two important formulae:

1. Value from any product = Basic Value + Extra Value

• Rationale: We expect a Basic Value when we purchase a product. For example, a TV is expected to deliver crisp images and delightful audio — those are two of its basic values. Extra Value is what makes a product even more appealing. In our TV example, a TV that is curved has the extra value because it better suits your vision.

    2. Overall Value of a product to your company = ROI to company + ROI to users

• Rationale: ROI (Return on Investment) to a company is important because the company could use those funds elsewhere. ROI to product users is also important because a product with no user adoption is of little use.

These two formulae are discussed in more detail below.

Value from an automated machine learning product

An automated machine learning product needs to bring high value to your company.
Basic Value:

• Provides value to a large number of machine learning needs in your organization.
• A product that is highly engaging for your users so that they will invest their time on it.

Extra Value:

• This product helps your company propel to the forefront of predictive technology.

ROI to the company

You want to derive good value from automated machine learning; usually a lot more than what you invested. Let us call this its “Machine Learning Value.”  You can measure this Machine Learning Value using an ROI formula.

It is partly your job to carefully document these value creations as you evaluate various products.

These formulas vary from customer-to-customer and many companies may not even have a formal process to calculate the ROI. Some companies might calculate this in a very indirect manner. For example, you might want to look at projects you couldn’t have seen otherwise. You will then attribute the value coming out of it, and subtract  expenses in order to get the Machine Learning Value from this product. Your expenses would be the time you spend learning about this product, the subscription fee, and the opportunity cost, as you will miss out on other machine learning products by purchasing a given product.
You may also want to look at it as your investment into emerging technologies that will produce value over several years. In such a case, you don’t need to directly calculate an ROI, but can measure other tangible outcomes from using this product — how much of your workforce will be trained in this new technology while using this product, how this might help your company or even your group get new funding, show your shareholders (if you are a public company) your adoption of new and valuable technology, and so on. It is partly your job to carefully document this value creation as you evaluate various products.

Machine Learning Value = Value from all items discussed above- product cost – opportunity cost

ROI to company = Machine Learning Value / (product cost + opportunity cost)

ROI to users

Before you create any value in your company, you need to create value for each user at your company. Let us call this “Value to User”. Value creation happens as a user experiences the product. This is where user journey comes into play.

Overall, a product that brings measurable, high value to your company and its users will be your best choice.

I describe various users’ journeys below, but you need to figure out which user personas you have at your company that will derive value out of an automated machine learning product.
All user journeys referred to below were based off of my user interactions on several machine learning modeling projects with a large client. You can substitute these personas with personas at your own company when you conduct your analysis.

User Journey 1 (Persona = Analyst):

1. This user is an Analyst who read about machine learning and sees this as an opportunity to learn more about this exciting technology.
2. This user gets an opportunity to work on the automated machine learning product, but has little experience with machine learning.
3. Once he starts using the product, he might run into problems:
   • Not having enough data
   • Doesn’t know where to get the data from, as his group never had this need
   • Once he gets the data, he doesn’t know how to frame the data, especially for the use cases that are not so straightforward
   • Once the data is all set, and an initial model is built, he doesn’t know how to improve it further

Derivation: make sure this product offers good training and support

User Journey 2  (Persona = Business User):

1. This user works on the business side, and is asked by his boss to use this automated machine learning product
2. The user starts working on it mainly to please the boss
3. She takes a formal class offered by the product’s company
4. Once she starts using the product she:
   • Doesn’t find time to use it, but tries doing something with it
   • Tries getting some answers or results to show them to her boss
   • Her take on it is, “If I get good results that will be great. It will help me create value, and I can also show my boss that I used it. If I don’t get good results, it’s also ok., I can still show my boss that I tried this product.”

Derivation: make sure this product company offers their own data scientists as consultants, so that this business user gets a helping hand
User Journey 3 (Persona = Data Scientist):

1. This user is a data scientist
2. The user wants to see how much this product can increase his productivity over his current methods
3. He takes a formal class offered by the product’s company
4. Once he starts using the product he:
   • Will try benchmarking how this product can do more than what he is already currently doing on his own
   • Is unsatisfied until he sees substantially higher values from this product beyond his present capacity
   • Might worry a bit if this kind of product will make him jobless one day, without realizing that a product like this could help him become more efficient at his job

Derivation: make sure this product offers advanced ML features that blows away even a seasoned data scientist
User Journey 4 (Persona = IT person):

1. This user is a technologist whom you can potentially convert into a great user of the product with minimal training

Derivation: make sure this product has a short learning curve, especially for IT people

User Value = Value from all Personas – Learning curve on the product

ROI to User = User Value / Cost to User in terms of his time and attention

Overall, a product that brings measurable, high value to your company and its users will be your best choice.
DataRobot is an automated machine learning company with client’s ROI deeply embedded in the DataRobot platform. This is evident from the large number of clients they serve today in multiple continents. DataRobot is easy to navigate for an analyst or an IT person, and also offers advanced features such as Reason Codes and Model X-Ray to support and enchant the more advanced users. DataRobot offers training through DataRobot University, as well as support and data science consultants through their AI services offering.

About the Author:
Raju Penmatcha, PhD works as a Customer Facing Data Scientist at DataRobot. Prior to becoming a data scientist he was (and still is) an engineer.
 

About the author

Raju Penmatcha

PhD, Customer Facing Data Scientist – DataRobot

Meet Raju Penmatcha