#202 Building A Unified and Uniform Approach To Data And Data Teams With Nathan Steiner, Director of Field Engineering, ANZ, at Databricks

Later this month, Nathan Steiner, the Director of Field Engineering, ANZ, at Databricks, will give a presentation at the Data Engineering Summit. There he will talk about the “habits” of data-driven organisations, and the importance of an open architecture that combines the best elements of data lakes and data warehouses.

Steiner kindly appeared on this episode of the Data Futurology podcast to talk about this, and further discuss the Databricks vision for data-driven workspaces.

“Historically, you look at data engineers, data analysts, AI, machine learning and data scientists, they were focused on different types of data, so you had your data engineers focused on your siloed and disparate ADW enterprise data warehousing, relational database structured systems, and you had your data scientists looking at predominantly real time data,” he says during the wide-ranging conversation.

The solution, to Steiner’s and Databricks’ vision, is bringing those data resources together and making for a more collaborative data environment. “It’s more pragmatic and effective for these job roles to be working from a single uniform platform,” he says.

As Steiner notes during the conversation, the personalisation that is so important to modern business is driven from being able to make the data resources collaborative. He highlights the example of a financial services company that wants to be able to issue credit within five minutes from an application via a smartphone. “In the back end, it's AI, and ML that is doing the credit risk assessment frameworks of that particular individual and creating that value customer experience,” he says.

Finally, Steiner considers the governance implications of the Databricks lakehouse, and the advantages of having a uniform and unified approach when it comes to governance.

For more insights on breaking down data silos and unifying data teams, be sure to tune in to the podcast!

Enjoy the show!

Learn more about Databricks

Learn more about Nathan Steiner

Thank you to you our sponsor, Talent Insights Group!

Join us at the Data Engineering Summit: https://www.datafuturology.com/data-engineering-summit

Join our Slack Community: https://join.slack.com/t/datafuturologycircle/shared_invite/zt-z19cq4eq-ET6O49o2uySgvQWjM6a5ng

“We look at data bricks as a company, simple, open, multi-cloud platform that’s really driven and passionate about democratizing AI and ML, and putting it in the hands of consumers, citizens and customers.”

— Nathan Steiner, Director of Field Engineering, ANZ, at Databricks

WHAT WE DISCUSSED

00:00 Introduction
04:07 Nathan tells us about his role and remit and introduces Databricks.
06:43 Where do you see the challenges at the moment that people are faced with?
11:52 In the last couple of years you’ve added Delta Lake, how did you see that development internally, what type of customer requests or challenges in the market were you responding to by making that move?19:08 Nathan discusses the three different functions -  the data engineers, the analysts and the data scientists and how they could interact with the lake house, using their own set of tools.
25:44 Can you talk about the democratization of AI and ML and what are the things have you been doing in that space?
30:16 How does governance come into play within Databricks within the lake house?
35:35 How is Databricks managing the skills shortage and how do you go about finding talent?
40:43 How do people get certified on Databricks? Is there any training that they could look at?
42:32 What can people expect to see from Databricks at Advancing AI & Data Engineering Summit in Sydney in August?

EPISODE HIGHLIGHTS

• You'll often hear our CEO talk about software eating, you know, eating the world, AI eating the world, we are, very much on that early precipice of the value that it can continue to bring to organizations. And that's where we are today and helping our customers solve those challenges.

• You had very large historical systems of record singing enterprise data warehouses, which were historically on-prem. And the twain didn't meet.

• At the heart of building a simple open and multi-cloud, AI machine learning data analytics platform, it really does all start with the data engineers.

• Data is growing, it's becoming more critical. It sprawling exists in a vast set of structured and unstructured data sets and enterprise applications, to get it into a single enterprise data warehouse, that's where the data engineers come in.

• Then you move into the data analysts, and they're the ones that start to move slightly closer to the business, they have a very good understanding of the business processes, and their business functions.

• You need to be very open and flexible, depending on the organization. You know, you cannot lock into a single unit, to a single set of tooling.

• When you start looking at the smartphone, that changed the game in terms of expectations of citizens, consumers and customers, and the way in which they consumed data, the way in which they consume services, unlocking the power of personalization, and just what it meant pushing data further to the edge of that device.

• I think that that's the true power of democratization the consumption and production of personalized information underpinned by data analytics, AI and ML, but we don't see it. It's all in the background, but the simplicity with which it's delivered is what we talked about in terms of democratization.

• You also need to have a very strong passionate focus on investment in delivering skills. So we talk about democratization, one of the big approaches that we're driving from a skills perspective is training.

• When we talk about skills, training and enablement and certification is a really big part of creating that, that momentum around education and enablement, because that is really the only way that you get it, get adoption, in democratizing is the ability to use it.

• So it doesn't matter what you're looking for, whether it's helping manage delivery risk through a partner, whether it's helped data bricks, helping you on where to start giving you more insight into aspects of our future releases and capabilities as part of the platform or, you know, as a customer or prospect, you know, having a look at you know, where we're at data bricks can kick it off. We will be there helping.

---

At Data Futurology, we are always working to bring you use cases, new approaches and everything related to the most relevant topics in data science to help you get the most value out of these technologies! Check out our upcoming events for more amazing content.