Apache Spark Introduction
Apache Spark is an open-source cluster computing framework. It has been designed to perform generic big data processing with great performance, it is well suited to machine learning algorithms, in this posts We will cover basic aspects such as components, installation and python console.
Spark Core and Resilient Distributed Datasets
Spark Core contains the basic functionality of the project. It provides distributed task dispatching, scheduling, and basic I/O . Spark Core is the place where Resilient Distributed Datasets (RDD) API is allocated, this API hides complexity inherent to distribuited systems and expose funcionally through Api clients in Java, Python, Scala. For the point of view of applications, processed data is not distribuited accross several nodes, data seems to be on the same machine and it can be accessed in the same way as any local file is accessed.
This component provides query support trough this components, data can be accessed using SQL or domain-specific language to manipulate SchemaRDDs in Scala, Java, or Python. This component is very powerful because complex query with standard SQL and SchemaRDD can be mixed in order to execute complex data maniputation process.
This component ingests data and performs RDD transformations.
MLlib Machine Learning Library
This component implements many common machine learning and statistical algorithms to simplify large scale machine learning pipelines.
This component provides an API for expressing graph computation.
We will assume that You have downloaded and installed spark-1.2.1 with support for hadoop2.4 on /opt directory, let's check what is inside:
$cd /opt/spark-1.2.1-bin-hadoop2.4/ $ls bin conf data ec2 examples lib LICENSE NOTICE python README.md RELEASE sbin
You can use the command
$ bin/pyspark Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. Spark assembly has been built with Hive, including Datanucleus jars on classpath Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 1.2.1 /_/ Using Python version 2.7.3 (default, Mar 13 2014 23:03:55) SparkContext available as sc
Let's open /opt/spark-1.2.1-bin-hadoop2.4/README.md and try some basic operations:
>>> text = sc.textFile("README.md") >>> print text README.md MappedRDD at textFile at NativeMethodAccessorImpl.java:-2 >>> text.count() 98 >>> text.first() u'# Apache Spark' >>> lines_with_spark = text.filter(lambda x: 'Spark' in x) >>> lines_with_spark.count() 19 >>> text.filter(lambda x: 'Spark' in x).count() 19
Find the line with the most words:
>>>text.map(lambda line: len(line.split())).reduce(lambda a, b: a if (a > b) else b) 14
Find the line with the biggest number of odd words:
>>>text.map(lambda line: len(line.split()) if len(line.split()) % 2 != 0 else 0 ).reduce(lambda a, b: a if (a > b) else b) 13
We can create a python file named process_file.py with the code below:
from operator import add from pyspark import SparkContext logFile = "/opt/spark-1.2.1-bin-hadoop2.4/README.md" # Should be some file on your system sc = SparkContext("local", "Simple App") logData = sc.textFile(logFile).cache() # get set of list with odd number of words odd_words_count = logData.filter(lambda line: len(line.split()) % 2 != 0).count() #get number of unique words unique_words = logData.flatMap(lambda line: line.split(' ')).map(lambda x: (x, 1)).reduceByKey(add).collect() print "there are %s lines with odd number of words\n" % odd_words_count print "there are %s unique words\n" % len(unique_words) print "and the unique words are :" for w in unique_words: print "%s %s times" % w
We can execute this program with the lines below:
$bin/spark-submit \ --master local \ process_file.py