---

Introduction to indexing

---

• Introduction to information retrieval
  • Suppose you have a large number of documents

    Examples:

    A library with a lot of eBooks A website with a lot of webpages The Internet with a huge amount of webpages

    ---

  • Problem:

    How can you find the locations of (some) (important) keywords in the documents ?

    Solution: search

    We can use KMP or Boyer-Moore to search the occurence of the keyword in each document

    ---

  • Problem with this solution:

    Multiple users can search for the same keyword(s) E.g., when Michael Jackson died, many different users searched for webpages containing "Michael Jackson"

    ---

  • $64,000 question:

    How can we avoid performing multiple identical searches on the input documentts ?

  ---

  ---

  ---

  ---

• Searching vs. indexing
  • Searching for patterns in a file:

    We can use the fast pattern matching algorithms (KMP, Boyer-Moore, etc) to search for patterns in text files.

    ---

  • Repeated searches for the same pattern:

    Different person (student) look up information on "Boyer-Moore" in a text book Different person (kid) look up information on "Batman" on the WWW Etc., etc.

    ---

  • Accomodating repeated searches:

    Use an index !!!

    Example:

    Index of a book:

  ---

  ---

  ---

  ---

  ---

  ---

• Indexing
  • Index

    Index = a dictionary containing entries of the form: Key, (location1, location2, ....) Example: Milk, (page 10, page 14, page 15) Recall: Dictionary = a map that stores a set of values (See: click here)

    ---

  • Fact:

    An index contains the result of search operations

    ---

  • Terminology: retrieval

    Retrieval = a lookup operation using some index

  ---

  ---

  ---

  ---

  ---

  ---

• Searching vs. Retrieving
  • Searching:

    We use searching algorithms when we rarely lookup patterns in an input document

  • Retrieving information:

    We index the input document and use retrieval algorithms when we frequently lookup patterns in an input document

  ---

  ---

  ---

  ---

  ---

  ---

• What an index look like
  • To give you a more concrete idea what an index (on data files) look like, here is an example:

    ---

  • The index is itself a file and is usually organized as a B-tree (B-trees are highly suited to be stored as a file --- see: click here)

    (A B-tree is a special case of an a,b-tree and you know everything about the a,b-trees -- you should be able to read the B-tree material on your own if you're interested.

    Instead of memory reference, they store record numbers to point to nodes stored in disk files)