🤔Should we use AI?

Just because its easier than ever to build AI models does not mean you should.

This is a fairly common question, and it’s best answered by analyzing the role AI can play and what cost it introduces to your users and your product.

To start, I want to touch upon the role “predictions” play in products. Unless we’re dealing with a very simple, single purpose product (such as a torch app), every product makes use of predictions in some form or another to provide a better UX.

• Apps that involve inboxes (such as an E-mail client or messaging app) predict that the user is most likely to engage with the most recent thread, and hence sort the inbox chronologically.

• The camera app on your phone predicts that you’re more likely to take a photograph than a video by default opening into that mode.

However, simple predictions such as chronological feeds don’t scale well to all use cases, such as when the feed contains a lot of content coming in. This forces you to think of second-order signals of importance – people the user has in their contacts, people the user frequently communicates with, etc.

At this point, you want to consider alternatives for simplifying your users’ lives. There are two ways you can go about it, using a movie streaming service as an example:

• Use heuristics: this could mean grouping movies together by genres, then recommending genres based on what the user recently watched, and movies inside the genre by popularity. You can exclude movies the user has already watched from the recommendations. The user can perceive how the recommendations work given the consistency.

• Use machine learning: this could using multiple signals such as how often a user watches movies of a particular genre, director or actor, the way they’ve rated movies, etc. The user is no longer able to perceive how the recommendations work as the system is now effectively a black box.

As with all things in computing, the above choices come with tradeoffs. If you’ve built a high quality machine learning model, it is likely that the model will perform better (for a movie streaming service, this might mean more movies streamed). However, it introduces complexity for your users who sense a loss of control over the content that they see.

So how do you make a decision on how to proceed?

First, you should carefully consider whether it’s ok for your product to introduce such complexity and take away control from your users. It often isn’t. For instance, an E-mail client that sorts your inbox with some black-box algorithm will arguably make it difficult to get things done for your users than it is to help them¹.

Regardless of your decision, it is still worthwhile to start with a heuristic-based system.This is because heuristics are likely to have a net positive impact on your users while still being explainable to them and likely less costly to build. This system can now serve as a valuable baseline for all future iterations.

At this point, you’re in a good position to start prototyping an ML-model driven solution. Utilize A/B testing tools to study how valuable the ML model really is to your users over and above the heuristics-based changes, and how they’re reacting to the changes. It’s at this stage that you can truly measure the ROI of using AI in your product, ideally using an objective decision making framework to guide you.

1. This doesn’t mean that you cannot solve your users’ problems using some form of prediction – you may just want to think of a different way of doing this. An example is marking E-mails as spam, auto-categorizing them into certain folders, etc to help the user keep their inbox manageable.