#distributed-computing

Disadvantages of this method:

Flexibility: Targeting only relevant data for output means that any future requirements, that may need data that was not included in the original design, will need to be added to the ETL routines. Due to nature of tight dependency between the routines developed, this often leads to a need for fundamental re-design and development. As a result this increases the time and costs involved.

Hardware: Most third party tools utilize their own engine to implement the ETL process. Regardless of the size of the solution this can necessitate the investment in additional hardware to implement the tool’s ETL engine.

Skills Investment: The use of third party tools to implement ETL processes compels the learning of new scripting languages and processes.

Learning Curve: Implementing a third party tool that uses foreign processes and languages results in the learning curve that is implicit in all technologies new to an organization and can often lead to following blind alleys in their use due to lack of experience.

In the process, the tools have made ETL a long and tedious exercise, one that often results in lags of weeks or even months from when data is first collected until it reaches a point where it can be analyzed. Or have caused severe headache to businesses and stakeholders who pray that their nightly ETL jobs are complete without an error, so as to prevent a showdown with the application users.

Traditional data integration and ETL tools are becoming an inhibitor to timely availability of high-value data to the business and will not be able to scale effectively with ever-growing volumes of data. In addition to the costs and challenges of attempting to scale these environments as data volumes grow, the issue of data latency due to the intermediate systems required for these platforms becomes a bigger and bigger threat to the enterprise.

Regardless of whether your enterprise takes the Data Warehousing approach or the Data Lake approach, you can reduce the operational cost of your overall BI/DW solution by offloading common transformation pipelines to tools like NodeRed or Apache NiFi and using Microservices and/or Spark jobs to provide a scalable, fault-tolerant platform for processing large amounts of heterogeneous data.

Contact us today to learn more about how you can realize your Data Dreams!