- Map-reduce:
- MapReduce = is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster (= share-nothing networked computers)
- Map-Reduce Computing Pattern:
- A MapReduce program consists of:
- A Map( ) procedure
- A Reduce( ) procedure
- The Map( ) procedure will:
- Filter/sort the
input into
separate queues
- There is one queue for one type/catagory/group of items
- Filter/sort the
input into
separate queues
- The Reduce( ) procedure will:
- Perform a computation (e.g. summary) on all items in each queue (group)
Schematically:
- A MapReduce program consists of:
- Note:
- The MapReduce program is
executed in parallel by
all (networked) processors
- The input data will be
distributed over
all processors
- The Map( ) result must be transmitted to its designated processor that takes care of the specific queue (group)
- The MapReduce program is
executed in parallel by
all (networked) processors
- Fact:
- There exists
software frameworks (implementations)
that
execute:
- A user-provided Map( ) function
- A user-provided Reduce( ) function
within the Map-Reduce environment:
- There exists
software frameworks (implementations)
that
execute:
- Framework for
execution of
Map-Reduce programs:
- The input has the following
structured as:
(key, value)
- The map( ) function
will perform
the following
transformation:
map( ) (key, value) ----------> (q1, w1), (q6, w2), (q1, w2) .....
- The MapReduce framework will
shuffle (= gather) the
output of the
map( ) function into
groups with
common values in the
first attribute
- The reduce( ) function
will perform the
following
transformation/computation:
reduce( ) (q1, w1), (q1, w2), (q1, ...) .... ------------> x1on each group produced by the shuffle function
- The input has the following
structured as:
- All that the
user need to do are:
- Write the
map( ) function
for a uni-processor
(that processes
a single stream of
input data)
- Write the reduce( ) function for a uni-processor (that processes a one group of shuffled data)
- Write the
map( ) function
for a uni-processor
(that processes
a single stream of
input data)
- The Map-Reduce
framework will:
- Run the
map( ) function
on multiple processors
- Run the
reduce( ) function
on multiple processors
- Group
the outputs of
all map( ) functions
according to the
first (= "key") value
in the output field
- Run the
reduce( ) functions on
one group of
the shuffled data
- Collect all the outputs of the reduce( ) function (as the final output)
- Run the
map( ) function
on multiple processors
- Hadoop:
- Hadoop = a distributed file storage and map-reduce computation framework
is an open source implementation of the MapReduce computation framework