The role "product data scientist" (DS) doesn't have quite the same clarity as some of the other roles in tech: Engineers build the product. Designers tell the engineers what the product should be. User researchers tell the designers what the users need. And the product manager tells them all to hurry up. So, what does a DS do?
My old boss used to say "we keep the product manager standing upright". As a product manager without a strong narrative is buffeted this way and that by the many concerned voices in their company. And who better to shape that narrative than a statistician with their hands on the product data?
That narrative can be broken down into a few key questions, and the activities a DS will do to answer them:
- What should we make next? Volume metrics show how much something is used, performance metrics how well it is working. Areas with high volume and low performance are opportunities for development. Combining this with previous product change assessments, we can estimate the impact of specific potential product changes & inform prioritisation.
- Should we optimise it with machine learning? Automated, but uncertain decisions that depend on large data volumes are opportunities for ML. They can be tested with offline models to achieve relatively inexpensive & accurate impact estimates. Promising solutions can then be prioritised & developed with engineering.
- What did our product change do? Using AB testing we can compare the product with and without our change. This allows for a direct measurement of the business impact, making the roll out decision straight forward, as well as validating our product strategy.
Possibly barring the ML question, these are the same questions a user researcher, designer and product manager will be asking. Close collaboration between these disciplines is essential to form a common view that is acted upon. This makes the conception of a DS as antisocial problematic; their ability to collaborate is paramount.
There is much foundational work needed to enable this, largely done in partnership with engineering: The creation of product data sets, metrics, ML pipelines & the capability to AB test. This tends to be an ongoing effort as requirements will change as the product evolves.
So, what does a product data scientist do? We keep the product manager standing upright