- Inverted index:
- Inverted index =
an
index data structure that
stores a
mapping:
- of the content (e.g., each word of a file)
- to its locations in a file
- Inverted index =
an
index data structure that
stores a
mapping:
- Example of a simplified
inverted index:
- We are given a number of files
(documents):
- Problem statement:
- Create a data structure that can help us locate the words in all documents.
Solution: use an inverted index
- We are given a number of files
(documents):
- Input:
- A collection of
documents:
- A collection of
documents:
- MapReduce algorithm
for computing the
inverted index of
- The map function:
- Input of the map function = (DocumentID, Content of the document)
- Output of the map function = (word in document, DocumentID)
Graphically:
- The shuffle mechanism
of the MapReduce framework will
then perform the following
movement (grouping) of
the outputs:
- The reduce( ) function:
- Input of the reduce function = a list of item of the form (word, DocID of the document)
- Output of the reduce function = reformat the input into (word, list of DocID )
Graphically:
- The map function:
- Summary:
