Continuous evolution is essential for every analytical system to provide increasing benefits to its owners and users. Numerous examples demonstrate the power of smart analytics in enabling intelligent decision-making. We firmly believe that even a single missed opportunity to move towards data-driven approaches represents a significant loss for companies in today’s world.
In the following material, we present our vision for the development of Enterprise Data Warehouse systems, along with a brief overview of our successful completed projects and the tools we have at our disposal.
Enterprise Level Reporting
This is the initial or, in other words, basic level of access to corporate information. We must take into consideration that this level is not rigid, but rather continually evolving throughout the lifecycle of an analytical system.
At this stage, users are granted the invaluable ability to utilize the essential basic features of an Enterprise Data Warehouse, including:
- consolidation of data from diverse source systems;
- data validation and cleansing;
- long-term access to the company’s data.
Resource-intensive analytical queries do not burden the enterprise Online Transaction Processing (OLTP) Systems.
At this stage, obtaining answers to questions such as “What’s going on in our company?” becomes a routine process of generating new types of reports.
Business users may access data via a wide range of interfaces:
- Specialized Business Intelligence (BI) tools;
- Scheduled reports or exports to the format of CSV / Excel files, which can be shared through mail, share point services, or FTP/SFTP;
- The protocol of interaction with the Enterprise Data Warehouse development team for creating ad-hoc requests (Report Request Service);
- Direct access of Power Users to the warehouse data model using SQL.
At this level, just five key components are typically sufficient to build a DWH, including:
-
Relational Database Management System (RDBMS): This serves as the core component of an Enterprise Data Warehouse solution, storing data and providing access to it;
-
Extract, Transfer and Load (ETL) tool: A collection of software components that are essential for the successful functioning of any solution;
-
Workload Automation tool: A highly desirable component of a robust Data Warehouse solution that provides a reliable and convenient way to monitor and control the execution of data loading processes;
-
Business Intelligence (BI) tool: Designed to retrieve, analyze, transform, and report data, this tool is critical for generating insights from the stored data;
-
Data Model: A variety of well-established paradigms exist for building data warehouses, and a thorough and comprehensive analysis should be conducted to determine which approach to use in each specific case, whether it’s Inmon or Kimball, Data Vault or Anchor Modeling. A significant advantage at this stage is the presence of a core data layer where information is stored in a normalized form, allowing for flexible and efficient retrieval of any new projections of data.
Our team possesses extensive experience in working with a comprehensive range of tools, which are listed below:
| Type of tasks | Free (conditionally paid) options | Paid Options |
| Persistent data storage system (DBMS) | MySQL, PostgreSQL, MariaDB, GreenPlum | Teradata, Oracle, Vertica, Microsoft, Netezza |
| ETL tools | Pentaho Data Integration, Talend | Informatica PowerCenter, SSIS, Oracle Data integrator, IBM Infosphere DataStage |
| BI Tools | BIRT, Knowage, RapidMiner, Microsoft Power BI Desktop, Tableau Public | Tableau, MicroStrategy Analytics, Qlik Sense, Microsoft Power BI, SAP Business Objects, IBM Cognos Analytics, Sisense, Looker |
GDPR
A project was undertaken to implement the General Data Protection Regulation (GDPR) requirements for organizing data operations in a bank. As part of this project, significant changes were made to the primary process for loading and processing data in the Data Warehouse. These changes enabled the fulfillment of critical GDPR requirements, such as Pseudonymisation, Right to Erasure, and Right of Access to personal data of clients, which were not initially incorporated into the solution’s architecture.
Client 360°
An all-encompassing implementation of the Data Warehouse was undertaken to satisfy the core reporting needs of the Bank. The solution’s foundation relied on the Teradata RDBMS, providing a comprehensive 360° view of individual clients. As part of the project, more than 500 attributes, encompassing clients, their agreements, and activities, were integrated into the Data Warehouse core. This was accomplished using a corporate data model based on the Financial Data Model (FDM), which was provided by Teradata Corp.
Migration
During the migration of a crucial data source for a large telecommunications operator, a tool was employed to automate script creation using templates. This approach drastically reduced development time while enhancing the reliability of the result and led to minimization of the human error risk.
OLAP Systems
The growth of analytical solution functionality is marked by the adoption of specialized Online Analytical Processing (OLAP) tools. These advanced tools facilitate the generation of reports from vast quantities of data (measured in terabytes) using interfaces familiar to business users, such as the PivotTable feature in Excel, with no need of involvement of IT staff.
The OLAP tools incorporate a comprehensive set of features that enable on-the-fly report generation across various periods and data subsets. Complex calculated key performance indicators (KPIs) can also be incorporated seamlessly. At the same time, the ability to access initial data sets at every stage of report generation is preserved. This ensures complete transparency in data handling, providing users with the ability to monitor and verify data use in real-time.
Our team possesses extensive experience in working with a comprehensive range of tools, which are listed below:
| Type of tasks | Free (conditionally paid) options | Paid Options |
| OLAP System | Druid, Mondrian OLAP server | Microsoft Analysis Services, SAS OLAP Server, Jedox OLAP Server, Oracle Database OLAP Option |
We’ve had the fortune of contributing to a number of remarkable projects, some of which are listed below:
Call-Center Cube
We undertook a project to develop an Analytical Cube for a telecommunications operator, which resulted in call-center managers being able to analyze employee productivity in-depth and with greater speed. This analytical capability facilitated the creation of a more equitable employee incentive scheme.
Call Data Record Cube
We have successfully developed an Analytical Cube that provides in-depth analysis of the telecommunications operator’s client activity with extensive historical data across a vast range of dimensions. The cube also incorporates a built-in transaction-level reporting capability that enables its users to view detailed information on the data used to generate specific analyses.
MDM Solution Systems
As an analytical solution matures, creating a golden record of key enterprise entities, such as products, customers, and points of sale, becomes crucial. To achieve this, the implementation of a Master Data Management (MDM) tool can be considered.
By employing this mechanism, a unified view of enterprise assets can be obtained, and issues such as duplication, contradictions, and omissions in data can be avoided, which typically arise when multiple accounting systems are used in the enterprise landscape.
The success of such a solution depends not only on selecting a suitable tool but also on the enterprise’s willingness to address potential data quality difficulties across the entire organization. This entails transforming the approach to data management to create a single centralized corporate data model and pursuing related organizational initiatives.
Our expertise:
| Type of tasks | Free (conditionally paid) options | Paid Options |
| MDM System | Taled Master Data Management | Informatica MDM, Oracle MDM, Ataccama One, InfoSphere MDM |
We’ve had the fortune of contributing to a number of remarkable projects, some of which are listed below:
MDM Remedy
We were tasked with optimizing the performance of an MDM solution for a cost containment services provider in the workers’ compensation industry of the United States. The original solution provider had failed to deliver a stable and productive production environment, and it fell on our team to ensure the reliability, transparency, and optimal performance of the entire solution we implemented.
Informatica Multi Domain MDM
We designed and delivered an MDM system for a clothing and accessories retailer, leveraging the Informatica Multi Domain MDM tool. Our solution was seamlessly integrated into the enterprise’s IT landscape, offering a transparent and easily manageable mechanism for controlling the company’s core entities through a unified mechanism.
Statistical Models
The next stage of the maturity of the analytical system goes beyond basic reporting and getting answers to the question “What’s happening?”. At this point, the system must satisfy more advanced requirements, allowing to answer to the question “Why is this happening?”:
- Access to all critical source systems has been established;
- A large volume of approved new indicators have been calculated;
- The enterprise has reached a consensus on the available information;
- The quality of the information available for analysis has reached an acceptable level;
- The system has earned a high level of user confidence, resulting in strategic requests becoming more frequent.
At this stage, the analytical system can transition to a mass creation of statistical analysis models. Using specialized tools, business users can access a new layer of information based on prepared data sets: the establishment of cause-and-effect relationships and answers to the “why” questions.
We’ve had the fortune of contributing to a number of remarkable projects, some of which are listed below:
Clients` Activity
A project was undertaken to develop a comprehensive flat data model for each customer of the telecommunications operator. The model includes over 300 pre-calculated Key Performance Indicators (KPIs) alongside relevant demographic information. This facilitates the creation of regression analysis models and enables Analysis of Variance (ANOVA) to be conducted.
Factor Analysis
We undertook a project on Factor Analysis for a health insurance enterprise. We managed to explore various new avenues to streamline the enterprise’s operations starting from greater flexibility in product configuration up to creation of a new tool to combat fraudulent activities.
HADOOP, Cloud, Building Machine Learning Models
As the number and complexity of tasks increase, basic tools for data storage and processing become insufficient:
- This is especially evident when dealing with semi-structured data from sources such as customer action logs and audio-video information. In such cases, a Schema-on-Read model is better suited, as it allows for flexible interaction with the original data without imposing strict structure requirements.
- Data from traditional sources can quickly accumulate and occupy significant storage space, driving up the cost of ownership. To address this, a more cost-effective model involves storing all raw data on relatively inexpensive media, and selectively moving only a portion of the information into a traditional data store.
As the volume and variety of data grow, it becomes necessary to explore complex patterns using advanced Data Science techniques such as Machine Learning. This extends beyond answering questions such as “What’s going on?” and “Why is this happening?” to include predicting “What can happen in the future?”
Thankfully, most cloud providers offer comprehensive platforms that provide access to storage systems and a wide range of advanced analytics methods and tools.
Just a short list of projects in which we were lucky enough to take part:
Our expertise encompasses such platforms as:
| Type of tasks | Free (conditionally paid) options | Paid Options |
| BigData | HDFS, Hbase, Hive, Spark, Zookeper | HortonWorks, Cloudera, MapR |
| Cloud | – | Microsoft Azure, Amazon Web Services, Google Cloud Platform |
| Machine Learning | H20.ai, KNIME, Spark, R, Python, TensorFlow | SAS, SPSS |
We’ve had the fortune of contributing to a number of remarkable projects, some of which are listed below:
Click Stream
We helped a large bank with collecting data on user activity, including Click Stream, on its website and mobile applications. This data will be used in subsequent stages of analysis, such as creating a model of customer behavior across interaction channels and measuring customer satisfaction with the website and mobile app.
Data Lake
We implemented the Data Lake concept in the telecommunication company to store a massive amount of Event Data Recorders (EDRs). A portion of the analytics was conducted directly within the Big Data systems, while the crucial data was extracted for additional processing in the core data store.
Migration to Google Cloud
We oversaw migration of the analytical solution of a beverage producer from an on-premises infrastructure to the cloud-based Google Cloud Platform.
Kafka, IoT, NoSQl, Search Engine: Active Data Warehouse
In the most advanced stage of development, the analytical system becomes an integral part of critical enterprise systems, automating operational decisions in real-time. The models are self-learning, and staff involvement is reduced.
Real-time data collection and processing systems in highly demanding environments, along with integration systems, are particularly important at this stage. Fast navigation and other features of Enterprise Search Engines have become an essential part of the entire solution.
In such projects we have experience of using such technologies as listed below:
| Type of tasks | Free (conditionally paid) options | Paid Options |
| Streaming Platform | Apache Kafka, RabbitMQ | IBM Websphere MQ, Microsoft MQ |
| Streaming Analytics | Apache Samza, Apache Storm | IBM InfoSphere Streams, Microsoft StreamInsight, Informatica Vibe Data Stream |
| NoSQL Database | Apache Cassandra, MongoDB, OrientDB | Neo4j, ArangoDB, IBM Domino |
We’ve had the fortune of contributing to a number of remarkable projects, some of which are listed below:
Campaign Management
We have implemented a marketing campaign management system in the telecommunication company that allows for real-time management and analysis of customer interactions, enabling quick responses to customer actions and movements.
Data Catalogue
We have developed a user-friendly interface to navigate data and metadata within the integrated Enterprise Data Warehouse solution. This solution includes Big Data objects, RDBMS objects, and exported reports, making it easier for users to access and analyze data.
Network Intelligence
We have designed an operational response system to address online events on the telecommunication company equipment, enabling the anticipation and quick response to breakdowns and loading increments on network elements.