Appliance configuration where the EDWH and Big Data functionality is tightly integrated and accessible via various API functions

The Application Enhancement compound pattern represents a solution environment where the Big Data platform is used to ingest large amounts of data in order to calculate certain statistics or execute a machine learning and then to feed results to downstream systems.

How can confidence be instilled in results whose computation involves applying a series of processing steps in a Big Data environment?

The Batch Data Processing compound pattern represents a solution environment capable of ingesting large amounts of structured data for the sole purpose of offloading existing enterprise systems from having to process this data.

For the introduction of a Big Data Platform in an existing application landscape that includes a data storage, transformation and analysis element, like a DWH there are four scenarious for integration possible. In this package four alternatives are described with their characteristics, advantages and disadvantages. For TDP one scenario must be selected for further implementation.

--

A Big Data blueprint as described in the literature. It is focused on Big Data but also useful for other data transforming activities

Beschrijving van de infrastructurele aspecten van een big data architectuur

Technology mechanisms represent well-defined IT artifacts that are established within an IT industry.

The simplest way to describe a pattern is that it provides a proven solution to a common problem individually documented in a consistent format and usually as part of a larger collection. The notion of a pattern is already a fundamental part of everyday life. Without acknowledging it each time, we naturally use proven solutions to solve common problems each day. Patterns in the IT world that revolve around the design of automated systems are referred to as design patterns.

The Big Data pipeline compound pattern generally comprises multiple stages whose objectives are to divide complex processing operations into down into modular steps for easier understanding and debugging and to be amenable to future data processing requirements.

Logical application composite for implementation of Big Data functionalities

The Big Data Processing Environment represents an environment capable of handling the range of distinct requirements of large-scale dataset processing.

Behandelen van big data als een tabelstructuur en werkt daarbij met een SQL achtige querytaal

Data transformation represents a solution environment where the Big Data platform is exclusively used for transforming large amounts of data obtained from a variety of sources.

The Big Data Warehouse represents a solution environment where a Big Data platform is used as a data warehouse capable of storing both structured and unstructured data online.

How can access to resources within a Big Data platform be managed efficiently and consistently?

How can a variety of datasets stored in a Big Data platform be governed efficiently and in a standardized manner?

How can large amounts of data be processed without investing in any Big Data processing infrastructure and only paying for the amount of time the processing resources are actually used?

How can large amounts of data be stored without investing in any Big Data storage infrastructure and only paying for the used storage space?

Big data sets can be voluminous so compressing has advantages for storage and/or processing

How can data stored in a Big Data solution environment be kept private so that only the intended client is able to read it?

How can a large dataset be made amenable to distributed data processing in a Big Data solution environment?

How can processed data be ported from a Big Data platform to systems that use proprietary, non-relational storage technologies?

How can large amounts of unstructured data be imported into a Big Data platform from a variety of different sources in a reliable manner?

Open source suite voor big data oplossingen

How can traditional BI tools access data stored in Big Data storage technologies without having to make separate connections to these technologies?

How can users seamlessly access enterprise IT systems and a Big Data platform’s resources without having to authenticate twice?

The Online Data Repository compound pattern represents a solution environment where the Big Data platform’s inexpensive storage is used to store data from internal and external data sources in its raw form available for consumption by any downstream application.

The Operational Data Store (ODS) compound pattern represents a solution environment such that the Big Data platform’s inexpensive NoSQL storage can be utilized as a traditional ODS where transactional data from operational systems across the enterprise is collected for operational BI and reporting.

Parallel integration of EDWH and Big Data functionality. Often between these functions an interconnect function is implemented

The Poly Sink compound pattern represents a part of a Big Data platform capable of egressing high-volume, high-velocity and high-variety data out to downstream enterprise systems.

The Poly Source compound pattern represents a part of a Big Data platform capable of ingesting high-volume and high-velocity data from a range of structured, unstructured and semi-structured data sources.

The Poly Storage compound pattern represents a part of a Big Data platform capable of storing high-volume, high-velocity and high-variety data.

Portal functionality to give users and administrator access to the other logical functionalities in a Big Data environment.

The Random Access Storage compound pattern represents a part of a Big Data platform capable storing high-volume and high-variety data and making it available for random access.

How can large amounts of processed data be ported from a Big Data platform directly to a relational database?

How can large amounts of data be imported into a Big Data platform from a relational database?

Management function for the various resources in a Big Data environment

Administration and registration of security concepts in a Big Data environment.

Serial implementation of Big Data and EDWH functionality where the Big Data functionality is consumed by the EDWH functionality

Storage function of big data sets.

How can processed data be exported in realtime from a Big Data platform to other systems?

How can high velocity data be imported reliably into a Big Data platform in realtime?

Implementation of EDWH and Big Data functionality extracted via a virtualisation function. This virtualization acts as an encapsulation layer and API for the consumer applications

For the introduction of a Big Data Platform in an existing application landscape that includes a data storage, transformation and analysis element, like a DWH there are four scenarious for integration possible. In this package four alternatives are described with their characteristics, advantages and disadvantages. For TDP one scenario must be selected for further implementation.

--

--

A Big Data blueprint as described in the literature. It is focused on Big Data but also useful for other data transforming activities

Beschrijving van de infrastructurele aspecten van een big data architectuur

Technology mechanisms represent well-defined IT artifacts that are established within an IT industry.

The simplest way to describe a pattern is that it provides a proven solution to a common problem individually documented in a consistent format and usually as part of a larger collection. The notion of a pattern is already a fundamental part of everyday life. Without acknowledging it each time, we naturally use proven solutions to solve common problems each day. Patterns in the IT world that revolve around the design of automated systems are referred to as design patterns.

Additional Big Data Patterns not linked in a compound pattern

In the appliance integration of a big data platform with DWH functionality the appliance acts like a black box in which all functionality is integrated in a (proprietary) solution. This solution is configured for optimal performance of transformation and analysis. Characteristics

• Appliance is developed, configured and often maintained by an external supplier
• It is introduced as a fully integrated solution therefore existing implementations of the DWH have to migrate to this solution
• Appliances are often introduced when a cloud solution is selected for the data platform

Layered or tiered architecture fortransformation of data from sources to utilisation. It includes three architectural columns that influence all layers

Plot of the described mechanisms on the Big Data Blueprint

Technology mechanisms represent well-defined IT artifacts that are established within an IT industry.

--

--

--

--

--

--

The hourglass model is a specific model for the transformation of data sources to a standardized model in a target datastore. It is the simplified implementation of a layered Big Data architecture. The hourglass model can be used to medel specific implementations of transformation of data in a pattern called the datapipe. In a number of other diagrams a detail view is given of these implementations in projects like Digital Transformation, TDP, MaxLimit and others.

--

The parallel integration is an extension of the DWH functiionality with the Big Data Platform. This extension makes it possible to use both functionalities side by side. Characteristics

• Easy (incremental) introduction of the Big Data functionality
• Integration of both functionalities requires attention for the introduction of the interconnect functionality because this can become a bottleneck in performance and configuration

Serial integration is implemented by introducing a big data platform for the transformation and extraction of unstructured and semi structured data as source for the EDWH functionality. Characteristics

• Introduction of the big data platform is relatively easy since it is an extra layer added to the DWH functionality
• Relative easy big data patterns are available because the source is always the datawarehouse
• Introducing big data solutions for other functionalities than DHW is not possible.