Study of ETL optimization techniques in big data

Abstract

Extract Transform Load (ETL) helps to move the source data from transactional database to analytical database for analysis. Optimizing techniques to help improve the process of ETL could greatly help in real time analysis of data. ETL optimization could be achieved through several factors simplest being increasing the frequency of the process. Other ways to achieve optimization is through use of various architectures, programing models, intelligence in transformation and security. The paper does a detailed study about the various architectural features, programming models, ontologies, and security patterns in achieving near real time ETL. Architectures could help in identifying areas of improvement while temporarily processing the data after extraction before loading. Programming helps to identify new paradigm for parallel processing. Ontologies help in automation of transformation techniques in heterogeneous environment. Finally security helps to maintain confidentiality, integrity, availability and privacy. This study does a comparative analysis of the various available variants of architecture, programming, ontologies, and security. More so the study focuses on big data having various sources, and format such as structured, semi-structured, and un-structured.  

Key Words: ETL Big data Architecture Programming Ontologies Security