Data architecture

The 70s: Hierarchical and network databases

In the 1970s, computer systems were mainly dominated by centrally managed mainframes. Data was organised using hierarchical or network database models. These models offered the possibility of organising data in a database in different ways - whether in a hierarchical structure that represented a relationship from one element to another, or in a network that linked many elements together. 

The 80s: The client-server model

In the late 1980s and early 1990s, a new paradigm of data architecture emerged with the advent of the client-server model. This model meant a move away from centralised mainframe systems towards a distributed system in which responsibilities were divided between servers (providers of resources or services) and clients (users of these services). In the area of databases, this meant that the database software (DBMS) could be installed on a server, while users or applications could access the data from client computers. This approach revolutionised scalability and accessibility and simplified the management of growing volumes of data and an increasing number of users. 

The 2010s: Cloud and Data Lake Architectures

In the 2010s, the concept of cloud computing emerged as a new paradigm, providing scalable resources as a service over the internet. This development had significant impacts on data architecture and led to the creation of data lakes. Unlike traditional data warehouses, which use the ETL process (Extract, Transform, Load) to ingest data, data lakes employ an Extract-Load-Transform (ELT) process. Data extracted from various sources is first loaded into cost-effective BLOB storage, then transformed, and finally transferred to a data warehouse using expensive block storage.

The need to process large volumes of data in real time gave rise to the Lambda and Kappa architecture models. The Lambda architecture employs a hybrid approach, utilizing both batch and stream processing to gain accurate and up-to-date insights. All incoming data is captured and stored as an append-only log, creating an immutable historical record. This architecture is divided into three layers: the Batch Layer, the Speed Layer, and the Serving Layer. In the Kappa architecture, all data is ingested and processed as an unbounded stream of events. This architecture consists of three main components: stream ingestion, stream processing, and long-term storage. 

The 2020s: Data Fabric

The Data Fabric represents the fourth generation of data platform architecture. Its goal is to make data available anytime and anywhere. A Data Fabric consists of a network of data platforms such as Data Warehouses, Data Lakes, IoT/Edge devices, and transactional databases that interact with each other and are distributed across the enterprise's computing ecosystem. One node in the fabric can supply raw data to another, which then performs analyses. These analyses can be provided as REST APIs within the fabric, allowing them to be used by transactional systems for decision-making. Data assets can be published in various categories, enabling the creation of an enterprise-wide data marketplace.