Querying Data: ETL versus Real Time and How Blue Light can Help

The focus on “real-time” data searching has long been a descriptive promise to provide the best availability of data for all magnitudes of analysts leveraging IBM i2 Analyst’s Notebook (ANB).  However, the reality is that very little data, especially in larger organizations, is of “real-time” stature, rather the data is more “near real-time” in nature simply due to the data transfer requirements for enterprise analysis systems. 

The majority of analysis systems in the market rely on the transfer of data from one or more source into another large, structured, relational database for a more streamline querying experience.  This process: Extract, Transform, and Load (ETL), is used to blend data from multiple sources and may require a heavy lift to clean and normalize the data.

The challenge with ETL and “real-time” arises with the latency of the transfer and the necessary cleaning and standardization of the data.  Therefore, the data instantly available in the originating sources is not immediately available to the analyst to be used within ANB.  In many situations, this lag may meet all or part of the workflow requirements.  On the other hand, a more streamlined connection to live data may be the better answer.

Blue Light recognized this need and developed the patent-pending federated search plug-in, Blue Fusion for Analyst’s Notebook.  Through data access on demand (DAOD) connectors fused into one querying user interface, the analyst now has the power to leverage “real-time” data at their disposal.  Furthermore, the query structure is familiar to that of ANB and allows complex questions to be asked with ease.  Pertinent results may then be consolidated and shared amongst a collaborative environment and further analyzed within the i2 platform.

These features provide significant advantages over the ETL process: analysts can now search multiple databases simultaneously while accessing the most “real-time” data available and then store relevant information in a central repository in collaboration with existing data thus providing a wide-spectrum view of an investigation or project.

Blue Fusion’s ability to access any data repository regardless of location, structure, or data format has saved i2 analysts up to 60% of their time.  Leveraging Blue Fusion provides organizations a significant increase in productivity, a decrease of capital and operating costs, and the ability to more efficiently secure and protect communities around the world.

Bruce Parkman

Chief Executive Officer, Blue Light LLC

Other news