As product managers, we constantly hear stories of how our products - and our lives - can be enhanced and improved with big data. But what about the dark side of big data?
This is exactly what Cathy O'Neil explores in her book Weapons of Math Destruction. This book completely blew my mind, and is one of the most important books for product managers - and everyone who works in tech - to read and understand.
I recently released a podcast where I talked about Weapons of Math Destruction with one of my machine learning engineer friends. You can listen to the episode for the whole review, but here are the 3 takeaways from the book that I found most relevant to product management.
1 - Transparency
How can someone fight an unfair algorithm, if they don’t understand the criteria fed into the algorithm, or how the algorithm works?
Here's an example: The model used to 'identify and reward the best teachers.' Many teachers who were scored poorly by the algorithm sensed there was something wrong - but how could they challenge the results of the algorithm, without understanding how the results were calculated?
Such a lack of transparency means whatever is going on inside the algorithm is taken as truth, without challenge.
2 - Feedback Loops
When we get results for an algorithm, initial results are often fed into a loop that determines future results. This can become very problematic, and can make algorithms biased.
Here's an example: Let's talk about the prison system. What makes it more likely that a felon will commit another crime? The justice system only looks at data points external to the prison system in order to determine if a prisoner will re-commit. If we compare this with companies like Amazon, who want to retain customers inside their ecosystem, actions people take inside their ecosystem are taken into account. Similarly, the justice system should be testing factors internal to prisons.
What happens if we give prisoners more sunshine? Does that affect their return rate? What about less solitary confinement? These data points aren't used as part of the feedback loop.
3 - Error Feedback
How do you test if the feedback you're getting from your product is accurate? With many machine learning and other models used in software products today, there is no way to determine if results are errors, or results are accurate.
Here's an example: Most companies use HR software that picks out the best resumes among the huge pool of applicants for job listings. Recruiters take the results returned by the software (without question) and proceed with the recommended applicants. But how do we know the software actually picked the best applicants?
We aren't looking at the rejects, so how do we know the best applicants weren't rejected?
Many of these use cases are pretty scary, since they affect all of us, and the options we have, or will have, in our lives.
I encourage product managers to read the book, understand the responsibility we have in dealing with data in our own products, and what red flags to look out for as we develop the next generation of software products.
You can listen to further discussion on this topic in the podcast episode - stream below.