Layered Architecture for Data Platforms: the place where data is stored

In the previous blogs about the Layered Architecture for Data Platforms we introduced the layered architecture for data platforms, dove deeper into the Data sources and Ingestion layer, and discussed the Processing layer. In this blog we look into the storage layer. In general, the storage layer is used to store the data in a data model so that the data can be analyzed or reported on.

When the data is ingested from the data sources and subsequently transformed in the processing layer, it is then stored in the storage layer. The purpose of the storage layer is to protect the data from disasters, malfunctions or user errors, to make the data available for developers, data scientists and end-users, and to archive data that needs to be kept for a long period of time.

There are many different technologies that can be used to store the data, each with their own advantages and drawbacks dependent on the type of storage you need. The most common storage technologies are:

• Relational Database Management Systems (RDBMS): These are relational databases that are often used for Online Transaction Processing (OLTP) systems like the ERP or CRM systems that many companies use. Most applications use a relational database to store the data that is entered in the system. For Online Analytical Processing (OLAP), relational databases can also be a good fit, however, other storage technologies are often used for these types of systems.
• Massive Parallel Processing (MPP) database: This is a type of relational database but with the main difference being that the data is split over several computers (with attached storage) that each process a part of the data. These kinds of databases are only used for OLAP solutions and are not suitable for OLTP, because these databases are designed to process queries that need huge amounts of data and are not good in processing many small requests. Mostly these MPP databases are used when the relational database is not fast enough.
• NoSQL databases: The NoSQL (Not Only SQL) databases are introduced to overcome some of the drawbacks of the relational databases related to the scalability and the flexibility of the data model. NoSQL database are non-tabular databases and store data differently than relational databases. Further in this blog we discuss the different types of NoSQL databases.
• Hadoop Distributed File System: This storage technology is a distributed file system that distributes (large) files over several computers with attached storage to make it faster to read and write. The idea of this storage solution is that low-cost servers can be used to limit the cost of storing huge amounts of data.
• In-memory databases: With in-memory databases the data is kept in the computers main memory. The data on the disk is only used as backup in case of power loss or malfunction, which makes in-memory databases very fast, but also quite expensive. Therefore, in-memory databases should only be used when the amount of data is small, performance is of high importance and the data is frequently accessed.
• Cloud storage: Cloud storage is comparable with the Hadoop Distributed File System. It is a storage solution that can store any kind of data such as files or tables. It is possible to choose between different protocols on how the data can be accessed, how fast it should be, the security options and the reliability (e.g. number of copies stored in different data centers or regions).

Most of the products from different vendors in the above categories are much alike, so for most use cases it really does not make much of a difference which product and vendor is chosen. This is not the case for the NoSQL databases. There are currently a few hundred NoSQL databases and they each have different properties that makes them useful for certain use cases. In general, these NoSQL databases can be categorized in 4 different groups (see figure 2):

• Document databases: stores data in a semi-structured file format like JSON.
• Key-value stores: stores keys and values. Data can only be retrieved by the key.
• Column families: stores data in tables, rows and dynamic columns
• Graph databases: stores data in nodes (people, places, things) and edges (relationships)

But also within each group there are many differences between the databases.

A data platform can use multiple different NoSQL databases or multiple storage technologies for different purposes or different types of data. However, before deciding on the storage technology you want to use, you need to consider the purpose of the storage layer. This can be:

• Landing Area: The landing area is the place where the ingested data is stored first before it is processed further.
• Staging Area: The staging area is an intermediate storage area for the ETL process.
• Data Warehouse: In the data warehouse, the data is integrated and stored in a data model so that it can be used for analysis or reporting.
• Data Mart: This is a special layer on top of the data warehouse that stores only a part of the data for a specific use case.
• Operational Data Store: Specific storage solution to store data that is needed to report on to support the current operations.
• Data Lake: This is a central place that stores all available data.
• Archive: This is where the incoming data is archived.

For each of these seven possible purposes, there is a specific storage technology that works best (as seen in Table 1).

A data platform often serves multiple purposes, which means that it can have multiple storage technologies for each different purpose. As you can see in Table 1, cloud storage is good for most purposes except when the data is frequently changed; the relational databases are good is most workloads, except when it involves unstructured data or when high performance is needed. Hadoop is a good choice when you need to store unstructured data or when you need a low-cost long term storage solution. In-memory databases are especially good when performance is very important and the amount of (structured) data is small. MPP databases are a good choice when you need to store and process huge amounts of structured data. NoSQL databases are not good for most purposes, but then can be a very good choice for certain specific use cases.

When considering which storage technologies to use, you can ask yourself the following questions:

Which type(s) of data do I need to store? Is it structured, semi-structured or unstructured data? Not only look at the current situation but also think about future requirements.

What are the performance requirements of the storage technology? Should the storage technology be fast enough to store the incoming data or should it also be fast enough to serve the data to the end-users?