The new challenge for data architectures
Organizations store more and more data in ever-larger volumes. However, most of that data is not new or original, but copied. Companies excel at duplicating data. For example, information about a customer is stored in a CRM system, a staging area, a Data Warehouse, several Data Marts, and a Data Lake. Even within one database, data is stored multiple times to support different users. In addition, copies of data are stored in development and test environments. And don’t forget the users who copy data from central databases to private files and spreadsheets. There is also data redundancy between organizations when exchanging data. Usually, the receiving organization stores the data in its own systems, resulting in even more copies.
The unrestrained duplication of data has many disadvantages and challenges:
Higher data latency, Missed opportunities, Complex data synchronization, More complex data security, More complex data privacy, Higher development costs, Higher maintenance costs, Higher technology costs, More complex database administration, More complex metadata administration, Reduced data quality.
Data minimization is therefore one of the most important preconditions for existing and new data architectures.
During this masterclass, Rick van der Lans explains how you can work towards a data-on-demand Architecture and with which solutions and technologies this becomes a reality. He will discuss, among other things, what data minimization is, what influence it has on data architectures and how data virtualization enables you to reduce redundant data.
What you will learn
- How the design principle called data minimization is related to simpler data architectures
- What the two pillars of data minimization mean: data-on-demand and accessing original data
- What the real drawbacks are of creating too many copies of the data are, including higher data latency, complex data synchronization, more complex data security and privacy, and higher development and maintenance costs
- How new database, integration, and cloud technology can help to design simpler data architectures that contain less copied data
- What the effect is of applying data minimization to Data Warehouse and Data Lake architectures
- How managed-file-transfer can be replaced data-on-demand, and how the number of data flows between organizations can be reduced
- How data architectures should be designed from the perspective of data processing specifications and not data stores
- New technologies can simplify data Architectures
- Applying data minimization to current data Architectures
- Data track diagrams for designing data Architectures
- From data-by-delivery to data-on-demand