By: Ashish Bhat, Principal Consultant

Most companies today are faced with a common question – how can data that resides in various systems across the IT landscape be integrated? A mutual issue that organizations face today is that the data that they have is fragmented across different enterprise management systems, file systems, and mission-critical spreadsheets. Even if you find an organization with a central repository of data, it is difficult to find consistent master data, synchronized business rules, and capabilities to report the data for a business benefit to pull off data-driven decisions. Even though we keep hearing statements like “data is the new oil,” why is it that organizations struggle with the management of the data? The answer to that question lies in the following four statements:

1. How is data stored?
2. How is data defined?
3. How is data retrieved?
4. How is data viewed?

The industry has come to a full circle on how to handle data. In the 1970s, the distributed data management system (DDBMS) was being trialed and the way it allowed data access to write and edit did not allow it to handle the required levels of cross-referencing, validating, and other data management operations to uphold data integrity. Eventually, during the 1990s, we witnessed the emergence of mighty enterprise resource planning (ERP) like SAP. What ERP did was to centralize the recording and storing of the data, though duplication of data to data-warehouses was still required. The ERP system, coupled with data-warehouse systems such as SAP BW, allowed data storage and data retrieval without breaking the integrity. Of course, data companies developed applications that allowed more integration with their own products to start with.

With the changes in technology and the advent of internet and related technology, the amount of data organizations create, started growing exponentially. Not much thought was given to data created outside organizations till entities realized that they can no longer work in (data) silos. The need to integrate fragmented data from multiple systems and the world of internet was much more than ever before to succeed amidst the competition from a new breed of technology savvy companies. Today, companies are looking at integrating Big Data to help product developers analyze unstructured data, for instance, cultural trends and customer reviews, and respond quickly with the aim of allowing businesses to improve and personalize their customers’ experience.

So, how does Big Data sit along the enterprise resource applications and data warehouse structure? This again is a question related to data integration and analysis. Most companies are creating data lakes and marts to understand the impact of external data and trying to integrate that data to get a unique insight. Gartner, in their report, predicted that “organizations that provision an augmented data catalog to data consumers will realize three times faster ROI from their data and analytics investments.”
To reach a point where business can draw a unique insight from internal and external data, here are some conditions that need to be fulfilled:

1. Integrate the data through a unified interface service.
2. Understand the need of new data types, new data-quality levels, new volumes, new metadata, new user requirements, and new performance needs.
3. Understand the role of algorithms to achieve data discovery.
4. Understand the right technology mix and heterogeneous architecture best suited for business.
5. Invest in data-driven integration.
6. Invest in correct analytics tools.

In most cases, we get requests from our customers to fix the data issues, or they want us to provide guidance in deciding the selection of the analytical tool. The underlying aspiration in all such cases is to correct the data flow and fix the analytical issues, which stem from data fragmentation and inconsistency of data.

How is SAP helping its customers?

SAP acquired KXEN a leader in predictive analytics. SAP analytics is coupled with predictive technology to enhance the value of core SAP applications with the intention of managing operations, supply chains, and customer relationships. This is in turn helping customers make better decisions on petabytes of Big Data through predictive analytics and data mining.

At LTIMindtree, we help our customers to move to architectural models that help them manage their data. These customers use our expert analytical skills to visualize the data in order to provide unique business insights.

Conclusion:

As we move into a new era where data is the new currency, for companies with legacy systems and architectural design, which foster data silos, it is pertinent to adopt new tools and frameworks to bridge the gap and realize the potential of BIG DATA!