#236: Building ML Products at Compare the Market, with Conor O'Neill

This week on the Data Futurology podcast, we have an in-depth conversation with Conor O’Neill, the Head of Data Science at Compare The Market exploring his career journey and current role leveraging data and innovating with machine learning.

When O’Neill landed at Compare The Market, he quickly found himself in a senior data role within an organisation that needed to both transform and mature its approach to data. On the podcast, O’Neill walks through the various stages of transformation, and getting the rest of the organisation aligned with that vision.

He also shares some use cases that Compare The Market is effectively leveraging data for, as well as how they have been building ML products. He explains how he involves data scientists in this process and offers advice on building ML as a product when it comes to planning, delivery and infrastructure.

Finally, O’Neill shares some thoughts on the difference between a data scientist’s role and that of a senior manager, and how this shifts the perspective and how a data professional will look at projects. He then rounds out the conversation with some thoughts about where data science is heading as a profession.

For anyone interested in data science, O’Neill’s unconventional journey into and through the profession is both interesting and inspiring. Enjoy the show!

https://youtu.be/Qpf6PbmZQx0

Connect with Conor: https://www.linkedin.com/in/conoroneill1/

Thank you to our sponsor, Talent Insights Group!

Join us for our next events Advancing AI and Data Engineering Sydney (5-7 September): https://www.datafuturology.com/events 

Join our Slack Community: https://join.slack.com/t/datafuturologycircle/shared_invite/zt-z19cq4eq-ET6O49o2uySgvQWjM6a5ng

“If you have an approach to data that produces a prediction that a person then reviews, that means you have to upskill those people and make sure that they're understanding everything at a much deeper level. Whereas, if it's all done through systems, you don't have to do that so much.”

—   Conor O’Neill, the Head of Data Science at Compare The Market

WHAT WE DISCUSSED

2:26: Felipe introduces Conor O’Neill.

3:23: O’Neill shares his journey from astrophysics to data science.

6:49: In astrophysics, the data sets that scientists work on are massive. O’Neill shares some insights about how he managed data in that role.

8:40: O’Neill shares his journey at Compare The Market so far.

12:04: O’Neill shares some information about a current data project that he and his team are working on.

18:08: Compare The Market had to do significant foundational work in transformation. O’Neill shares insights into that process.

21:18: O’Neill shares his experience in getting the Compare The Market organisation aligned behind their data vision.

25:12: O’Neill explains the value of having data scientists involved at the earliest stages of transformation design. 

28:44: O’Neill describes his experience in moving from a data scientist role to heading a team, and the differences between these roles. 

32:56: O’Neill explains some of the thinking that goes into reusing data projects, as well as how they decide the projects to not follow through.

34:04: Getting a model in front of the end users and driving adoption is a critical step – O’Neill explains how he has approached it for Compare The Market.

37:54: O’Neill overviews the various consumers of the work done by the data team, and how the data team needs to think about each of them.

40:51: Tips and guidance for creating ML as a product to be consumed internally

45:48: O’Neill shares some thoughts on how the data science industry is evolving.

EPISODE HIGHLIGHTS

• “We’ve been on a transformational journey now for a little over a year, and that’s been really good. We’ve been migrating off our legacy on-prem stack to Databricks. We’ve also been focused on getting the right people, and then also establishing a process, because if you just change the tool, you haven't fixed the issues, typically.”

• “You don't want your control group to be too large and you then miss opportunities. But you also don't want it to be so small that you don't get sufficient data. That's where the algorithm behind our recommendation system controls that, to optimise according to our confidence that we are or are not exceeding the required threshold, and adjust the weighting of the control group accordingly.”

• “Initially, we thought this would be a pretty straightforward process, because there are some similar processes in play, but with the changes in the architecture and how we want to build and enhance products, there was a lot of coordination required with the architecture teams, the tech teams, the DevOps teams and security teams. This is especially true because the Databricks real-time ML endpoint actually doesn't sit within your own environment. It's on Databricks environment, which means you have to traverse the Internet to make use of it.”

• “We all know data scientists have an ill-defined remit at the best of times, they can be many different things.”

• “We just recently went through FY24 planning, and a big part of that was getting all these different groups coordinated, discussing the improved processes that we're doing as an organisation to make sure dependencies are managed well. It’s a really difficult thing to get right.”

• “The data scientist might be responsible for building a great model that achieves the goals it's meant to do. And they're not necessarily responsible for the end-to-end project success. Now, however, I need to make sure that that gets used in the right way by the business.”

• “If you have an approach to data that produces a prediction that a person then reviews, that means you have to upskill those people and make sure that they're understanding everything at a much deeper level. Whereas, if it's all done through systems, you don't have to do that so much.”

• “I'm definitely a fan of product-driven development for the sole purpose that it coordinates all the different groups together to make sure that everyone's focused on delivering the same thing, and reduces the number of conflicting projects that might come in from the side to take away one group of resources to work on something else and makes delivery a bit more confusing.”

• “I've had people wonder whether data science is going down, because there's a big push for data engineering. I actually don't think that's the case. I think it's just standard maturing, the value is still there, but the industry is realising what it takes to deliver properly.”

---

At Data Futurology, we are always working to bring you use cases, new approaches and everything related to the most relevant topics in data science to help you get the most value out of these technologies! Check out our upcoming events for more amazing content. And as always, we appreciate your Reviews, Follows, Likes, Shares and Ratings. It really helps new data scientists find us.