How to Run a Machine Learning Technical Due Diligence

This post is NOT aimed only towards Machine Learning (ML) experts that run a Tech DD on a startup for a Venture Capital (VC). If you are a Data Scientist, ML Engineer or a Researcher considering joining an existing ML team at any company, you should read this post to gauge the level of the ML foundations and take a decision that matches your career goals.

Part I — Company Overview

As a Data Scientist I know that Data Scientists and ML researchers are often perceived as employees that are very disconnected from the business. I believe that such teams are doomed to fail and thus, before diving into the details of the ML system, we need to understand the business.

The company

An overview of the company, the vision and goals.

The products

A description of the product/s that the company is currently developing, with emphasis on their ML aspect. If applicable, a description of possible future products that rely on this infrastructure.

The competition

A list of competitor products and technology, and the competitive advantage over them.

The team

A summary of the ML leaders (managers, tech leads, etc.) background with relation to the relevant academic and industry experience.

Part II — ML Technology Deep Dive

It is important to note that a good ML system is not only measured by the depth of the research or by using the state of the art algorithms. It must have also solid engineering foundations to allow scaling, easy deployment and a smooth management of the Data Science work.

AI system design

A description of the main AI components, data stores, flow and architecture. In addition, a review of the development roadmap for the coming year. Does the team have a clear view of how the final system should look like?

  • Serving — system throughput, average time per prediction, average cost per prediction and how it’s monitored
  • Training — average cost of training a model, frequency of training in production and how it’s being monitored

The data

  • Data sources — how do you get the data and the relevant labels?
  • Manual annotation — what is the amount of manual annotation and curation work? Do you have plans to reduce it?
  • Data quality — do you preform regular QA actions on the data, to make sure it’s not corrupted and that it’s accurately annotated?
  • Privacy issues — how do you make sure that privacy is not compromised?

The models

  • In-depth description of the models — including models that are currently in use, those that are under development, and the roadmap for the coming year. Note, that the models may be simple, even rule-based if the company is in early stages — but it is important to assess what exists, what is the short-term plan and what is the full vision. Are the people ML-savvy and understand the bits-and-bytes? Are they up-to-date with relevant SOTA (state-of-the-art) models? This part plays a major role in evaluating the ML capabilities and trajectory.
  • Models measurements — this is a tricky part. It is different for every business problem, but basically, I am trying to assess whether the team focuses on the customer experience when measuring their models. This approach imposes not only the choice of success measurements but also the sampling methods.
  • Models monitoring — including service and data monitoring and alerting
  • The process of locating and dealing with prediction mistakes
  • Models development process — this part is all about the team’s day to day work processes

Discussion

  • The alternatives to the current solution and why this solution was chosen
  • The advantage the system has over competitors

Part III — Summary and Recommendation

This part is dedicated to the final evaluation of the AI technology. I focus on the following main criteria:

  1. Technological depth and unique IP (Intellectual Property) — which doesn’t have to be backed by a patent, but rather demonstrate innovative ideas.
  2. Scalability —making sure the engineering infrastructure supports scale and robustness.
  3. Team — I provide an evaluation of the team’s ML understanding and ability to execute.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Inbal Budowski-Tal

PhD in Computer Science, loves Machine Learning and curious about bringing ML to prod and creating real business value by researchers. Oh, and I love dancing!