Big Data Ecosystem: The 5 Vs, Hadoop HDFS & Apache Spark

Explanation

Traditional relational databases cannot scale to terabytes and petabytes of unstructured clickstream or IoT sensor telemetry. Hadoop revolutionized distributed storage by breaking large files into blocks (typically 128MB) replicated across commodity servers via HDFS. However, Hadoop MapReduce writes intermediate results to local disk after every job stage, creating I/O bottlenecks. Apache Spark solves this by caching data in cluster RAM through Resilient Distributed Datasets (RDDs), accelerating iterative machine learning and graph algorithms by up to 100x.

Key points

• Hadoop Distributed File System (HDFS): Master/Worker architecture (NameNode manages metadata, DataNodes store redundant data blocks).
• MapReduce Framework: Distributed programming model splitting computation into Map (filtering/sorting) and Reduce (aggregating) phases.
• Apache Spark: In-memory distributed computing engine replacing disk-bound MapReduce using Resilient Distributed Datasets (RDDs) and DataFrames.
• Stream Processing: Real-time event streaming architectures using Apache Kafka and Spark Streaming to process continuous sensor and financial transaction streams.

Example

Code examples

/*
MapReduce Word Count Paradigm:
Input Text: "apple banana apple"
1. Map Phase:
   ("apple", 1), ("banana", 1), ("apple", 1)
2. Shuffle & Sort:
   "apple"  -> [1, 1]
   "banana" -> [1]
3. Reduce Phase:
   "apple"  -> sum([1, 1]) = 2
   "banana" -> sum([1])    = 1
*/

Exam questions

• [8 Marks] Describe the 5 Vs of Big Data. Explain the master-slave architecture of Hadoop HDFS (NameNode vs DataNode).
• [8 Marks] Explain the MapReduce programming model using a complete Word Count example. Why is Apache Spark significantly faster than MapReduce?