- Word count:
- Word count = the number of occurence of a word in a set of documents
- Example of
word count:
- We are given a number of files
(documents):
- Problem statement:
- Give the number of occurence of each word in all documents.
Answer:
- We are given a number of files
(documents):
- Input:
- A collection of
documents:
- A collection of
documents:
- MapReduce algorithm
for computing the
inverted index of
- The map function:
- Input of the
map function =
(DocumentID,
Content of the document)
- Output of the
map function =
for each word
x in
each document, output:
- (x, 1)
Graphically:
- Input of the
map function =
(DocumentID,
Content of the document)
- The shuffle mechanism
of the MapReduce framework will
then perform the following
movement (grouping) of
the outputs:
- The reduce( ) function:
- Input of the
reduce function = a
list of item of the form
(x,
1) for
word x in
the documents
- Output of the
reduce function:
( x, #items in input list )
Graphically:
- Input of the
reduce function = a
list of item of the form
(x,
1) for
word x in
the documents
- The map function:
- Dictatical note:
- I just want to show you an
implementation of the
MapReduce algorithm
- Specifically, I like to show you the Hadoop API
- A fuller treatise of
the MapReduce framework
is offered in the graduate course:
- CS581 --- High Performance Computing and Data Analysis(HPC + DA)
- I just want to show you an
implementation of the
MapReduce algorithm
- The mapper (class):
public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); // one = 1 as an object private Text word = new Text(); // Output help variable public void map(Object key, Text value, Context context) throws IOException, InterruptedException { /* ================================================ Tokenize input parameter (document) "value" ================================================ */ StringTokenizer itr = new StringTokenizer(value.toString()); while ( itr.hasMoreTokens() ) { word.set(itr.nextToken()); // Get next word context.write(word, one); // Output: (next word, 1) } } }Notice that:
- The key parameter is
used to identify the
input file
- The value parameter is the
content of the
input file
- You write a map( ) function that process one (1) data file (passed in the parameter value)
- The key parameter is
used to identify the
input file
- The reducer (class):
public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterablevalues , Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); // Convert int to IntWritable for output context.write(key, result); // Write output (word, count) } }Notice that:
- The key parameter is:
- the first field in the map( ) function's output.
In this case:
- key = a word in the documents
- The value parameter is:
- A list (bag) of values in the second field in the map( ) function's output
In this case:
- value = {1, 1, 1, ...}
- The key parameter is:
- The driver code to
set up the
MapReduce compute engine:
public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); // Set map function job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); // Set reduce function job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }Demo:
Go to aruba: ssh cheung@aruba
Start Hadoop (single node mode): cd /home/cheung/hadoop/bin start-hadoop
Goto the WordCount example directory: cd /home/cheung/hadoop/src/Examples/WordCount I have 2 input files: inp inp2 I have put them in Hadoop's test directory: hdfs dfs -ls test Found 2 items -rw-r--r-- 1 cheung supergroup 51 2016-06-07 17:54 test/inp -rw-r--r-- 1 cheung supergroup 26 2016-06-13 11:14 test/inp2 Compile: javac -cp `hadoop classpath` -d wordcount_classes WordCount.java Make jar file (computation is sent to the data in Hadoop): jar -cvf wordcount.jar -C wordcount_classes/ . Run program: hadoop jar wordcount.jar WordCount test test-out Get result: hdfs dfs -get test-out/part-r-00000 Remove the output: hdfs dfs -rm -r test-out