---

Heavy Elements and Iceberg queries

---

• "Heavy" Elements
  • A heavy element is a value that occurs more frequent than othe values

  • The set of heavy elements will depend on the threshold

  ---

  ---

• Iceberg Queries
  • An iceberg query is a query that finds heavy elements

  • Canonical Iceberg Query:

    SELECT t1, t2, ..., tk, count(r) FROM R GROUP BY t1, t2, ..., tk HAVING count(r) >= T

    ---

  • Example:

    SNUM SNAME STATUS CITY ----- ---------- ---------- ---------- s1 Smith 20 London s2 Jones 10 Paris s3 Blake 30 London s4 Clark 20 Paris s5 Adams 30 Athens s6 Brown 20 Rome s7 Klein 5 Rome s8 Green 10 Athens s9 Johnson 15 Atlanta s10 Jackson 10 Rome s11 Bond 10 Atlanta s12 Newton 15 Paris s13 Lofton 15 Athens

    Number of suppliers in each city:

    select city, count(snum) from supplier group by city; CITY COUNT(SNUM) ---------- ----------- Paris 3 Atlanta 2 Athens 3 Rome 3 London 2

  ---

  ---

• Importance of Iceberg Queries
  • Many data mining queries are basically iceberg queries

  • Example: Market Analysis

    Store owners want to know the shopping behavior of their customers The Market Analysis query identifies user buying pattern by finding item pairs that are frequently bought together Examples: (Coffee, Milk), (Diaper, Beer), etc...

    ---

  • The first step in market analysis is:

    Find the items that are bought together most frequently

    This is ofcourse an iceberg query...

  ---

  ---

• Finding heavy elements efficiently
  • A research result clims that the most time consuming part in association rule data mining is identifying the heavy elements (Park et. al. 1995)

  • The paper "Computing Iceberg Query Efficiently" presents several techniques to find the heavy elements in a database.

  ---

  ---

• Definitions
  • Notations:

    R = data source (a materialized relation) R is of the form: {<t1, r1>, <t2, r2>, ...} V = ordered list of targets in R: V[r] is the rth most frequent item in R Freq(r) = the frequency of the item V[r] Area(r) = ∑i=1r Freq(r) n = number of items in V

    ---

  • Heavy Items

    Define: r1 = max{   r   |   Freq(r) ≥ T   } Schematically: Freq(1) ≥ Freq(2) ≥ ... ≥ Freq(r1) > Freq(r1+1) ≥ ... | | V V ≥ T < T The set of the Heavy Items is: H= { V[1], V[2], ..., V[r1], }

  ---

  ---

• False positives
  • Let F = the potential solution set of the heavy elements

  • False positive:

    F contains false positive if F   −   H   ≠   ∅ I.e.: some item in F is not heavy (and it is included as part of the solution)

    ---

  • Eliminating false positives:

    The Count(F) procedure:

    Input: R = input relation F = potential solution of H (|F} << |R|) Variable usage: count[e] = frequency of item e for ( each e ∈ F ) { count[e] = 0; // Initialize count } /* -------------------------------- Compute frequency of items in F -------------------------------- */ for ( each e ∈ R ) { if ( e &isin F ) count[e]++; // Count actual occurence } /* -------------------------------- Compute for false positives in F -------------------------------- */ for ( each e ∈ F ) { if ( count[e] < T ) delete e from F; // False positive }

    ---

  • NOTE:

    Notice that the memory size used is only |F| number of counter variables

  ---

  ---

• False negatives
  • Let F = the potential solution set of the heavy elements

  • False negative:

    F contains false negative if H   −   F   ≠   ∅ I.e.: some heavy item in H is left out from the solution (and are missing from the solution)

    ---

  • Regaining false negatives:

    It is very difficult to regain the omitted items The amount of overhead to recover the false negatives (omitted items) is usually just as expensive as solving the original probelm

  • Moral:

    You rather have too many items in F than too few It's easier to remove false positives...

  ---

  ---

• Presentation
  • We will fist discuss algorithms to find heavy elements in materialized databases

    In this environment, the algorithm can make multiple passes through the data.

  • Then we will look at algorithms to find heavy elements in data streams

    In this environment, the algorithm can only make one pass through the data.