#hadoo

Objective

Apache Spark is an open-source, lightning fast big data framework which is designed to enhance the computational speed. Hadoop MapReduce, read and write from the disk, as a result, it slows down the computation. While Spark can run on top of Hadoop and provides a better computational speed solution. This tutorial gives a thorough comparison between Apache Spark vs. Hadoop MapReduce.

In this guide, we will cover what is the difference between Spark and Hadoop MapReduce, how Spark is 100x faster than MapReduce. This comprehensive guide will provide feature wise comparison between Apache Spark and Hadoop MapReduce.

Comparison between Apache Spark vs. Hadoop MapReduce

           Introduction

Apache Spark – It is an open source big data framework. It provides faster and more general purpose data processing engine. Spark is basically designed for fast computation. It also covers a wide range of workloads for example batch, interactive, iterative and streaming.

Hadoop MapReduce – It is also an open source framework for writing applications. It also processes structured and unstructured data that are stored in HDFS. Hadoop MapReduce is designed in a way to process a large volume of data on a cluster of commodity hardware. MapReduce can process data in batch mode.

          Speed

Apache Spark – Spark is lightning fast cluster computing tool. Apache Spark runs applications up to 100x faster in memory and 10x faster on disk than Hadoop. Because of reducing the number of read/write cycle to disk and storing intermediate data in-memory Spark makes it possible.

Hadoop MapReduce – MapReduce reads and writes from disk, as a result, it slows down the processing speed.

          Difficulty

Apache Spark – Spark is easy to program as it has tons of high-level operators with RDD – Resilient Distributed Dataset.

Hadoop MapReduce – In MapReduce, developers need to hand code each and every operation which makes it very difficult to work.

           Easy to Manage

Apache Spark – Spark is capable of performing batch, interactive and Machine Learning and Streaming all in the same cluster. As a result makes it a complete data analytics engine. Thus, no need to manage different component for each need. Installing Spark on a cluster will be enough to handle all the requirements.

Hadoop MapReduce – As MapReduce only provides the batch engine. Hence, we are dependent on different engines. For example- Storm, Giraph, Impala, etc. for other requirements. So, it is very difficult to manage many components.

          Real-time analysis

Apache Spark – It can process real-time data i.e. data coming from the real-time event streams at the rate of millions of events per second, e.g. Twitter data for instance or Facebook sharing/posting. Spark’s strength is the ability to process live streams efficiently.

Hadoop MapReduce – MapReduce fails when it comes to real-time data processing as it was designed to perform batch processing on voluminous amounts of data.

          latency

Apache Spark – Spark provides low-latency computing.

Hadoop MapReduce – MapReduce is a high latency computing framework.

          Interactive mode

Apache Spark – Spark can process data interactively.

Hadoop MapReduce – MapReduce doesn’t have an interactive mode.

          Streaming

Apache Spark – Spark can process real-time data through Spark Streaming.

Hadoop MapReduce – With MapReduce, you can only process data in batch mode.

          Ease of use

Apache Spark – Spark is easier to use. Since its abstraction (RDD) enables a user to process data using high-level operators. It also provides rich APIs in Java, Scala, Python, and R.

Hadoop MapReduce – MapReduce is complex. As a result, we need to handle low-level APIs to process the data, which requires lots of hand coding.

          Recovery

Apache Spark – RDDs allows recovery of partitions on failed nodes by re-computation of the DAG while also supporting a more similar recovery style to Hadoop by way of checkpointing, to reduce the dependencies of an RDD.

HDFS (Hadoop Distributed File System) çok sayıda küçük dosya ile pek verimli çalışmaz.  HDFS, büyük boyutlu dosyalar üzerinde işlemler gerçekleştirmek için tasarlanmıştır.  Bunun en önemli nedeni, dosyalar erişim için gerekli olan meta verinin Namenode üzerinde dinamik olarak saklanıyor olmasıdır. Çok fazla sayıda küçük dosya bu meta verinin efektif yönetimini zorlaştırır. Hatta bu amaçla konumlandırılan ikincil namenode her ne kadar failover durumları için düşünülse de, belirli periyodlarda asıl namenode hafızasını optimize etmek amaçlı “check pointler” oluşturmak için de kullanılır. HDSF son versiyonunda eklemelere (updates) izin verilmesine rağmen HDFS yapısı, “bir kere yazılıp defalarca oku “ (write one, read all ) prensibine göre çalışır. Hadoop mimarisi master-slave yaklaşımı ile düzenlenmiştir. Namenode master node, slave node ise verilerin saklandığı DataNode’ lardır. HDFS default block size 64 MB’ dır. Daha büyük block size seçilmesi arama kaynaklı gecikmeleri (seek overhead-disk) disk transfer zamanının %1’ i kadar azaltır. Blok boyutu 64 MB dan küçük olan dosyalar HDFS bloğunun tamamını doldurmazlar. Master node hafızası yönettiği slave node ve disk boyutuna göre seçilmelidir. Örneğin 100 makine toplamda 4 TB disk ile replikasyon faktörün 3 olduğu ve 64 MB blok boyutlu seçilen bir durum için :

100x 4 x 1024 x1024 / (3 x64) yaklaşık 2 Milyon MB için 2 GB hafıza seçilmelidir. (her bir milyon MB için 1 GB önerilmektedir)

Ada Wessanthara – sindu.me.mp3 Adare. Obe Dase – sindu.me.mp3 Allagena Kana – sindu.me.mp3 Allagena Katha (New Ver.) – sindu.me.mp3 Atha sita A Suwadak – sindu.me.mp3 Atheen Pa Salaba – sindu.me.mp3 Dana Mana Pinawana – sindu.me.mp3 Gangawe Alaye –…