This is part 3 in our Field Guide to Machine Learning Product Development: What Product Managers Need to Know series. Read the Introduction Field Guide to Machine Learning Product Development: What Product Managers Need to Know, learn about how to manage data throughout the machine learning product lifecycle, and stay tuned for more deep dives into the six main stages of the machine learning PM process, starting with Discovery below.

Field Guide to Machine Learning Product Development: Discovery

The development of a machine learning product should always start with discovery. As a product manager, I am constantly asking myself “are we building the right thing?”. It’s a question that overrides all other concerns I might have.

I’m sure I’m not alone: we product managers (PMs) are responsible for ensuring the products our teams build deliver value to our users and our business. We are responsible for ensuring that our clients not only use but choose our product. We are responsible for delivering a quality solution in a timely manner.

Ultimately, answering that question often boils down to ensuring we’ve identified a problem that is not only worth the effort of solving, but also one that our team can solve best. Identifying such a problem is at the core of product discovery. In this post, I share my perspective on product discovery, drawing from experiences as a PM at an AI-first organization that emphasizes product innovation.

Model Validation: a vital tool for building trust in AI
Image - PMBS_Discovery_1.png

Focus on the problem space

Just as you wouldn’t want to set off down a long highway in the wrong direction, product discovery is critical to setting the right direction through the product development lifecycle. Head off in the wrong direction and you could find yourself far from your destination as weeks or even months are lost while the team solves for the wrong problem. Nobody wants that. So, it is critical for product discovery to focus on clearly


identifying the right problem to solve to create real value for the users.

But what is the ‘right problem’ and how does a product team find the right problem?

There are already a lot of good resources out there that explore product discovery in a more traditional setting (i.e., without the complexity of incorporating ML, let alone bespoke, novel ML solutions, into products).

For example, I like how Silicon Valley Product Group’s Marty Cagan lays out 5 areas of risk to explore during product discovery: user value risk, usability risk, feasibility risk, business viability risk, and ethics risk. Product discovery coach Teresa Torres also has a great framework to evaluate opportunities and other useful tips around how to organize findings from the discovery process.

There is also a lot of good content available on how to talk with users and customers to uncover pain points, how to test assumptions, and how to build POCs and prototypes quickly. Indeed, much of the traditional product discovery process, which focuses on the problem space (versus solutioning), remains applicable to a machine learning PM in the innovation space.

Image - PMBS_Discovery_7.png

What to do when identifying the problem space is not enough

For those PMs who are focused on applying ML techniques in novel and innovative ways, there are two additional considerations from the solution space that need to come into the mix when identifying the ‘right problem’ during discovery.

The first is to consider whether ML is actually the best solution to solve the problem. It is far too easy to get caught up in the excitement of ML and want to apply it to every problem we come across (I’ve been in those conversations before). But an ML solution is relatively complex and comes with its own set of risks and challenges. An ML solution is something you apply only when non-ML techniques are insufficient for solving the problem. Simply put,


just because ML can solve the problem does not mean it’s the best solution for the problem.

When evaluating the applicability of ML to a given problem, I often look at three main axes:

  1. Data. ML can amplify existing challenges in the input data – “garbage in, garbage out” – so the more comprehensive, unbiased, and clean data that the product team can access, the more accurate and reliable the ML solution will be. And just because data exists doesn’t mean it is accessible or usable for the intended solution. For example, if your solution relies on user feedback and data but your customers are disinclined to give you access to their data, machine learning may not be a good fit. Similarly, if the only data available is known to be biased, you may want to consider alternatives to ML (for more on the data challenges across the product development lifecycle, check out this blog by my colleague Khaled Ammar).
  2. Generalizability. Is there a need for some kind of generalized behaviour or capability? Are there too many instances to program manually or to write rules for? If yes, the problem is probably a good candidate for ML. Machine learning is powerful, in part, because it can account for new instances that were not explicitly seen in the training data. In other words, you didn’t need to have come across that specific example in order for the ML algorithm to handle it. However, the ability to generalize well remains a tough problem in ML and is still not perfect, which leads me to the third axis…
  3. Tolerance for error. There is always a chance, especially when ML is applied in unanticipated situations, that the algorithm might make a mistake (like when legitimate emails occasionally wind up in your spam box) or create an unwanted outcome (anyone remember Microsoft’s Tay chatbot?). Knowing that, we must always consider the likelihood and potential downsides of such mistakes and unwanted outcomes. In situations where there is very low tolerance for error, like in a medical life-or-death situation, ML may play a role, but it will likely be accompanied by many non-ML safeguards as part of the solution. In cases where the tolerance for errors is not as low – like a book recommendation engine, for example – an ML-only solution may be suitable.

The second consideration in identifying the ‘right problem’ is the opportunity for unique ML contributions. You may find it is possible to use ML in your product without any novel ML research. But for PMs at organizations with teams of researchers aiming to develop novel ML techniques (like those at Borealis AI), we must also evaluate whether there is an opportunity for unique ML contributions in the products we develop.

It is a tricky consideration for PMs to balance. On the one hand, we know that ‘cool research’ should never be the rationale for creating a product; discovery is about understanding the user and business problems. But on the other hand, PMs should also avoid taking too narrow a lens by focusing solely on the immediate problem, and think more broadly about how our users, businesses and problem space will evolve. This can often uncover even bigger problems that can lead to greater innovation and space for novel research to be applied, to deilver better returns for the users and for the business down the road.

Image - PMBS_Discovery_2.png

Building trust is key to product discovery

At Borealis AI, we believe that one of the keys to product discovery comes down to the ‘people’ side of the equation. While Borealis AI has expertise in product, engineering, and ML research, we work closely with partners in the specific lines of businesses at RBC to leverage their business domain expertise. Successful product discovery (recognizing the considerations above) often relies on establishing trust with our business partners and a focus on long-term relationships.

One of ways Borealis has successfully built trust with our business partners is by starting with a focus on delivering shorter-term value for our business partners and their users at the expense of novel technology and high risks, but with the clear intention to collaborate with partners on more innovative high-risk, high-reward solutions over time. We also often create POCs and prototypes that can be helpful in building trust with our partners.

Over time, and as trust with our business partners builds, the ability to discover more innovative product opportunities grows. We understand more about the pain points and trends of the business domain and the users. Our partners understand more about ML, its limitations and its risks. They gain confidence in ML solutions. And that leads to greater mutual trust as we work together to incorporate novel ML research into products that require longer timelines and carry greater risk (though those are also great opportunities for greater reward!).

Image - PMBS_Discovery_5.png

Always Be Discovering

At Borealis AI, our culture is centered around novel research and innovation, deeply intertwined with an ambitious product spirit. As product managers, our challenge is to discover the right problem: one that addresses a meaningful pain point for users and the businesses we collaborate with; that requires a complex technology like ML; and where there is opportunity to incorporate novel ML techniques and research over time.

Discovery of the ‘right problem’ to solve relies on the strength of our business partnerships, where trust needs to be established. As the business partnerships develop and evolve, continuous product discovery leads to more opportunities for innovation and novel ML contributions.

So, product managers out there: are we building the right thing? It’s hard to know for sure. But a good product discovery process is an important piece to answering that question.

 

 

We're Hiring!

We are scientists, engineers and product experts dedicated to forging new frontiers of AI in finance, backed by RBC.  The team is growing! Please view the open roles, including Product Manager, Business Development Lead, Research Engineer – Data Systems, and more. 

View all jobs