Technology Transfer - since 1986
Leading Edge Information Technology Education
First Class Speakers
Top Venue
A splendid location in the center of Rome close to Piazza del Popolo, Piazza di Spagna, Trevi Fountain, Piazza Navona and the Pantheon
Online Events
Due to time zones, events presented by American speakers will be spread over more days, and will take place in the afternoon from 2 pm to 6 pm Italian time
Real-Time Big Data Systems with Spark Streaming and Kafka
ONLINE LIVE STREAMING
Oct 11 - Oct 12, 2022
By: Jesse Anderson
Practical Guidelines for Designing Modern Data Architectures
ONLINE LIVE STREAMING
Oct 13 - Oct 14, 2022
By: Rick van der Lans
Creating Data Products in a Data Mesh, Data Lake or Lakehouse for use in Analytics
ONLINE LIVE STREAMING
Oct 17 - Oct 18, 2022
By: Mike Ferguson
Centralised Data Governance of a Distributed Data Landscape
ONLINE LIVE STREAMING
Oct 19 - Oct 20, 2022
By: Mike Ferguson
Understanding DataOps to Deliver Analytics Better and Faster
ONLINE LIVE STREAMING
Oct 21, 2022
By: John O'Brien
Artificial Intelligence, Machine Learning and Data Management
ONLINE LIVE STREAMING
Nov 03 - Nov 04, 2022
By: Derek Strauss
Designing, developing and deploying a Microservices Architecture
ONLINE LIVE STREAMING
Nov 11, 2022
By: Sander Hoogendoorn
Upcoming Events
Free article of the month

June 2022
Upcoming events by this speaker:
Jun 20 – Jun 21 2022:
Real-Time Big Data Systems with Spark Streaming and Kafka
Dec 15 – Dec 16 2022:
Modern Application Architectures
Data Engineering Technology Tree
“What we know is a drop, what we don’t know is an ocean.”
Isaac Newton
Data engineering is one of the disciplines where you just know a drop. Some companies are saying it’s easy, and you just need to know a drop. My experience in the field and teaching tells me otherwise. A data engineer needs to learn many different technologies and possess in-depth knowledge of big data.
To help you sort it out, I want to help you imagine the skills as a technology tree. You might have played the Civilization series at some point and maybe even spent way too much time (just one more turn). If you aren’t familiar with it, here is Civilization 6’s technology tree.
You’ll notice that you start with the most basic technologies in the world, such as pottery or animal husbandry. As you begin to research those technologies, you unlock more technologies. Each of these technologies takes a certain amount of turns to research, and the number of turns is based on the science your civilization produces.
If you didn’t know, you can try to skip researching technologies. Instead of gaining all of the foundational knowledge, the player can try to skip ahead. Skipping technologies causes all kinds of problems in-game, just like we’re about to see in our real-life example.
Let’s imagine data engineering as a technology tree. I think it all starts with a specialization in technology and branches out from there. These branches are systems, programming, and architecture. Looking at the diagram below, you can see various relationships.