How to Run a Machine Learning Technical Due Diligence

This post is NOT aimed only towards Machine Learning (ML) experts that run a Tech DD on a startup for a Venture Capital (VC). If you are a Data Scientist, ML Engineer or a Researcher considering joining an existing ML team at any company, you should read this post to gauge the level of the ML foundations and take a decision that matches your career goals.

In recent years, VCs turn to me for running DD on startups that heavily rely on Machine Learning in their “claim to fame”. The goal of the DD is to estimate the depth of the ML technology and the team’s capability to deliver their “claim”, and win the “fame”.

I chose to write this post because I was looking for a ML tech DD document template online and didn’t find any, so I figured it will be a good idea to share mine. Moreover, I use this set of guidelines whenever I choose my next workplace or when I give others advice on how to choose theirs. So here we go.

Part I — Company Overview

As a Data Scientist I know that Data Scientists and ML researchers are often perceived as employees that are very disconnected from the business. I believe that such teams are doomed to fail and thus, before diving into the details of the ML system, we need to understand the business.

The company

An overview of the company, the vision and goals.

The products

A description of the product/s that the company is currently developing, with emphasis on their ML aspect. If applicable, a description of possible future products that rely on this infrastructure.

The competition

A list of competitor products and technology, and the competitive advantage over them.

The team

A summary of the ML leaders (managers, tech leads, etc.) background with relation to the relevant academic and industry experience.

Part II — ML Technology Deep Dive

It is important to note that a good ML system is not only measured by the depth of the research or by using the state of the art algorithms. It must have also solid engineering foundations to allow scaling, easy deployment and a smooth management of the Data Science work.

AI system design

A description of the main AI components, data stores, flow and architecture. In addition, a review of the development roadmap for the coming year. Does the team have a clear view of how the final system should look like?

• Serving — system throughput, average time per prediction, average cost per prediction and how it’s monitored
• Training — average cost of training a model, frequency of training in production and how it’s being monitored

The data

• Data sources — how do you get the data and the relevant labels?
• Manual annotation — what is the amount of manual annotation and curation work? Do you have plans to reduce it?
• Data quality — do you preform regular QA actions on the data, to make sure it’s not corrupted and that it’s accurately annotated?
• Privacy issues — how do you make sure that privacy is not compromised?

The models

• In-depth description of the models — including models that are currently in use, those that are under development, and the roadmap for the coming year. Note, that the models may be simple, even rule-based if the company is in early stages — but it is important to assess what exists, what is the short-term plan and what is the full vision. Are the people ML-savvy and understand the bits-and-bytes? Are they up-to-date with relevant SOTA (state-of-the-art) models? This part plays a major role in evaluating the ML capabilities and trajectory.
• Models measurements — this is a tricky part. It is different for every business problem, but basically, I am trying to assess whether the team focuses on the customer experience when measuring their models. This approach imposes not only the choice of success measurements but also the sampling methods.
• Models monitoring — including service and data monitoring and alerting
• The process of locating and dealing with prediction mistakes
• Models development process — this part is all about the team’s day to day work processes

Discussion

• The alternatives to the current solution and why this solution was chosen
• The advantage the system has over competitors

Part III — Summary and Recommendation

This part is dedicated to the final evaluation of the AI technology. I focus on the following main criteria:

1. Technological depth and unique IP (Intellectual Property) — which doesn’t have to be backed by a patent, but rather demonstrate innovative ideas.
2. Scalability —making sure the engineering infrastructure supports scale and robustness.
3. Team — I provide an evaluation of the team’s ML understanding and ability to execute.

To summarize, I write my personal opinion and recommendation including the technological and practical challenges and concerns.

After writing this post, I have realized that many of the points I raised here serve me when I interview experienced Data Scientists and ML engineers about their past projects (each has a different focus, of course).

And if you’re on the other side, that is, interviewing for a ML team, and the interviewer is surprising you with the “do you have anything you’d like to ask me?” question, I think you have a nice set of options to choose from, too :)