Online Events

Due to time zones, events presented by American speakers will be spread over more days, and will take place in the afternoon from 2 pm to 6 pm Italian time

Real-Time Big Data Systems with Spark Streaming and Kafka

ONLINE LIVE STREAMING

Oct 11 - Oct 12, 2022

By: Jesse Anderson

Practical Guidelines for Designing Modern Data Architectures

ONLINE LIVE STREAMING

Oct 13 - Oct 14, 2022

By: Rick van der Lans

Creating Data Products in a Data Mesh, Data Lake or Lakehouse for use in Analytics

ONLINE LIVE STREAMING

Oct 17 - Oct 18, 2022

By: Mike Ferguson

Centralised Data Governance of a Distributed Data Landscape

ONLINE LIVE STREAMING

Oct 19 - Oct 20, 2022

By: Mike Ferguson

Understanding DataOps to Deliver Analytics Better and Faster

ONLINE LIVE STREAMING

Oct 21, 2022

By: John O'Brien

Agile Data Science 2.0

ONLINE LIVE STREAMING

Oct 24 - Oct 28, 2022

By: Russell Jurney

Artificial Intelligence, Machine Learning and Data Management

ONLINE LIVE STREAMING

Nov 03 - Nov 04, 2022

By: Derek Strauss

User Experience Design

ONLINE LIVE STREAMING

Nov 07 - Nov 10, 2022

By: James Hobart

Designing, developing and deploying a Microservices Architecture

ONLINE LIVE STREAMING

Nov 11, 2022

By: Sander Hoogendoorn

Free article of the month

Jesse Anderson
June 2022

Upcoming events by this speaker:

Data Engineering Technology Tree

What we know is a drop, what we don’t know is an ocean.
   Isaac Newton

Data engineering is one of the disciplines where you just know a drop. Some companies are saying it’s easy, and you just need to know a drop. My experience in the field and teaching tells me otherwise. A data engineer needs to learn many different technologies and possess in-depth knowledge of big data.

To help you sort it out, I want to help you imagine the skills as a technology tree. You might have played the Civilization series at some point and maybe even spent way too much time (just one more turn). If you aren’t familiar with it, here is Civilization 6’s technology tree.

You’ll notice that you start with the most basic technologies in the world, such as pottery or animal husbandry. As you begin to research those technologies, you unlock more technologies. Each of these technologies takes a certain amount of turns to research, and the number of turns is based on the science your civilization produces.

If you didn’t know, you can try to skip researching technologies. Instead of gaining all of the foundational knowledge, the player can try to skip ahead. Skipping technologies causes all kinds of problems in-game, just like we’re about to see in our real-life example.

Let’s imagine data engineering as a technology tree. I think it all starts with a specialization in technology and branches out from there. These branches are systems, programming, and architecture. Looking at the diagram below, you can see various relationships.

                                                                                                                         Continued to read…

Subscribe to our newsletter