Business Intelligence and Data Warehousing

What is Business Intelligence?

Business Intelligence is the set of processes, technologies, and tools that help an organization to transform raw data into meaningful and useful information for business analysis (Identifying business needs and determining solutions to business problems).

What is the need for Business Intelligence?

Below are some of the major benefits of Business Intelligence in any Organization.

·        Get deeper insights in business operations, Identifying new opportunities and implementing an effective strategy based on insights can provide businesses with a competitive market advantage and long-term stability

·        Business Intelligence Provides historical, current and predictive views of business operations.

·        Sales and marketing – Understanding the profitability of customer segments and answers to valuable questions like,

o   Which customers should an organization target?

o   Which are my most profitable campaigns per region?

o   What is the most profitable source of sales leads and how has that changed over time?

·        Improve Productivity and efficiency

·        Informed decision making

·        Improve Customer Service and satisfaction

·        Streamline budgeting and planning

·        Financial decisions based on results for important questions like

o   What is the full cost of new products?

o   How are forecasts trending against the annual plan?

o   What are the current trends in cash flow, accounts payable and accounts receivable and how do they compare with plan?

·        Overall business performance tracking based on

o   What are the most important risk factors impacting the company’s ability to meet annual profit goals?

o   Should we expand internationally and, if so, which geographic areas should we first target?

Common Functions of Business Intelligence are

·        Reporting

·        OLAP (Online Analytical Processing)

·        Data Mining

·        Process mining

·        Complex event processing

·        Business performance management

·        Text mining, predictive analytics and prescriptive analytics.

Stages of BI

Below are the five stages of Big data Business Intelligence in any organization.

·        Data Sourcing – Defining the data to be loaded into the system. Usually BI applications gathers data from a data warehouse (Data marts, OLTP or OLAP).

·        ETL (Extract Transform Load) – Extracting the source data and transforming per business rules and loading into the Data Warehouses.

·        Data Warehousing – Storing transformed data into various Data warehouses types and making it available for business analysis.

·        Data Analysis – Applying various techniques like data mining, text mining, Process mining to identify trends and patterns in business operations.

·        Decision Making – Based on the reports, dashboards and alerts from previous stage, making valuable business decisions and bench marking future growth.

Data Warehousing

What is Data Warehousing?

The process of extracting and transforming internal and external data into useful business information and loading it into a central database so that it can be explored by business users across the company is known as Data warehousing.

What is a Data Warehouse?

A data warehouse is a relational database that is designed for query and analysis. Enterprise Data warehouses store current and historical data and are used for creating trending reports for business management like annual and quarterly comparison reports.

Business Intelligence and Data warehousing architecture

Below is the typical Business Intelligence and Data warehousing platform architecture.

Data Warehouse(DW or DWH) Types

·        Data Marts – As shown in the above architecture, a data mart is a simple form of a data warehouse that is focused on a single functional area, like sales, finance or marketing.

·        Online Analytical Processing (OLAP) – OLAP databases store aggregated, historical data in multi-dimensional schemas. OLAP systems typically have data latency of a few hours, as opposed to data marts, where latency is expected to be closer to one day. Mainly used for Reporting and allows complex analytical and ad-hoc queries

·        Online Transaction Processing (OLTP) – OLTP systems support online transactions like INSERT, SELECT, UPDATE, DELETE within fraction of seconds. OLTP is mainly aimed at fast response, simplicity and efficiency but not for reporting purpose.

Below is the high level comparison chart between data warehouse types.

Function	OLTP	OLAP	Data Marts
Operation	INSERT, UPDATE, SELECT	Complex Queries	Report Generation
Latency	Within Seconds	Within Hours	Within Days
Analytical Requirements	Low	High	Medium
Age of Data	Current	Historical, current and projected	Historical and Current
Business Events	React	Predict	Anticipate

But to handle Big data, the above regular data marts are not capable and Hadoop (HDFS, Hive, HBase) plays the role of OLAP data ware house type in typical Big data Business Analysis using Hadoop. Below is the Hadoop Perspective of the Data warehousing architecture.