---

Hybrid algorithms for finding heavy elements in database

---

• Hybrid Algorithms
  • The main focus of design of the algorithm is:

    Reduce the initial candidate set F

    Reason:

    The larger the candidate set, the more memory you need to use to store the values And, the more time you need to spend to eliminate false positives

    ---

  • The paper presents 3 hybrid algorithms that uses a combination and variations of the 2 basic techniques to reduce (cut down) the candidate solution set F:

    Defer-Count Multi-level Multi-stage

  ---

  ---

• Reducing the size of the initial candidate set
  • Scaled-Sampling

    We can reduce the size of the initial candidate set in Scaled-Sampling by: Using a smaller selection probability p

    ---

  • Coarse-Count

    When a light (not heavy) element is hashed in the same location as another heavy element, that light element will become a false positive The initial candidate set can be reduce by preventing light and heavy elements to be hashed together..

  ---

  ---

  ---

• The Defer-Count Algorithm
  • Fact:

    The Scaled-Sample algorithm (that uses Random Sampling) can identify candidates of the heavy element set

    We can exploit this result to:

    separate the identified heavy elements from the light elements to reduce the number of false positives.

    The is the gist (key) of the Defer-Count technique...

    ---

  • Defer-Count Algorithm:

    S = Scaled-Sample(R) using a suitable probab p; F = select f most frequent items in S; *** Results: 1. Items in F are most likely heavy, but not guaranteed 2. F can contain false positives 3. F can contain false negatives /* -------------------------------------------------------- Defer-Count: hash count without counting elements in F -------------------------------------------------------- */ /* ----------------------------- initialize counters ----------------------------- */ for ( i = 1; i <= m; i++ ) A[i] = 0; /* ------------------------------------ Hash-count items in R but not counting elements in F ------------------------------------ */ for ( each e ∈ R ∧ e ∉ F ) A[ h1( e ) ] ++; // Note: 2 items e1 and e2 that hashes // to the same value will be counted together... *** Note: we have NOT potentially heavy elements Light elements that WERE hashed together with these elements will now be removed from F /* ------------------------------ Form final solution set F ------------------------------ */ for ( each v ∈ R ) { if ( A[ h1( e ) ] ≥ T ) add e to F; } // At this point, F may have false positives. // But: F will not miss any heavy elements ! Properties of F: F has false positive F has no false negatives - every heavy element will be included in F ! F = Count(F); // Remove false positives ! // F will not have any false positives !!

    ---

    ---

  • Properties of the Defer-Count Algorithm

    S is small if selection property p is small enough Elements in F will most likely be heavy By excluding the elements in F from the counting procedure, fewer false positives will be generated by the hash counting procedure (The hash counting generates false positives when light elements and one heavy element are hashed into the same bucket. By removing the likely heavy elements from the hash counting step, fewer false positives will be generated) F is the exact solution of the set of all the heavy elements: F does not have false positives F does not have false negatives

    ---

  • Problems when using the Defer-Count Algorithm:

    Hard to pick a good selection probability p that produces a suitably small set of candidate heavy elements

  ---

  ---

  ---

  ---

• The Multi-Level Algorithm
  • The Multi-Level algorithm extends the Coarse-Count algorithm with a second level of hash counters to spread the heavy elements

    ---

  • The Multi-Level Algorithm:

    /* ------------------------------------------------------ Allocate more counters for potentially heavy elements ------------------------------------------------------ */ S = Scaled-Sample(R) with any probab p; for ( each e ∈ S ) { Mark A[e] as potentially heavy; Allocate m2 counter to A[e]; } *** Result: *** +------+ A: 1 | | m2 counters +------+ +---+---+----+---+ 2 | | * --> | 0 | 0 | .. | 0 | Potentially heavy elem +------+ +---+---+----+---+ | | +------+ +---+---+----+---+ k | | * --> | 0 | 0 | .. | 0 | Potentially heavy elem +------+ +---+---+----+---+ | | m2 counters +------+ ... +------+ m1 | | +------+ /* ----------------------------------------- Finer count for potentially heavy elems ----------------------------------------- */ for ( each e ∈ R ) { if ( A[e] is potentially heavy ) { A[e].counter[ h2(e) ]++; // Finer count for heavy elems // h2() is another hash function.. } else { A[e]++; // Light element } } /* ----------------------------------------- Find potentially heavy elements ----------------------------------------- */ F = ∅; for ( each e ∈ R ) { if ( A[e] is potentially heavy ) { if ( A[e].counter[ h2(e) ] ≥ T ) add e to F; } else { if ( A[e] ≥ T ) add e to F; } } *** Result *** F contains all heavy elements (F has no false negatives) F may contains false positives F = Count(F); // Remove false positives *** Result *** F = all heavy elements

    ---

  • Advantage of Multi-Level algorithm:

    The Multi-Level algorithm is not as sensitive to the selection probability p as the Defer-Count algorithm

  • Problems using Multi-Level algorithm:

    How to allocate memory to the secondary counter variables ? I.e., how large should m2 be ??? Proposed Solution: We can use the pre-scan S = Scaled-Sample(R) to compute m2 E.g.: m2 = Available Memory / |S|

  ---

  ---

  ---

  ---

• The Multi-Stage Algorithm
  • The Multi-Stage algorithm improves on the memory utilization in the Multi-Level algorithm by using of common counters for the heavy elements

    ---

  • The Multi-Level Algorithm:

    S = Scaled-Sample(R) with any probab p; for ( each e ∈ S ) { Mark A[e] as potentially heavy; Make A[e] point to the common pool of counters; } *** Result: *** +------+ A: 1 | | +------+ +---+---+----+---+ 2 | | * --+-> | 0 | 0 | .. | 0 | Common pool of +------+ | +---+---+----+---+ counter for | | | | 0 | 0 | .. | 0 | Potentially heavy elem +------+ | +---+---+----+---+ k | | * --+ | 0 | 0 | .. | 0 | +------+ +---+---+----+---+ | | m3 counters +------+ ... +------+ m1 | | +------+ /* ----------------------------------------- Count ----------------------------------------- */ for ( each e ∈ R ) { if ( A[e] is potentially heavy ) { A[e].counter[ h2(e) ]++; // Finer count for heavy elems } else { A[e]++; } } /* ----------------------------------------- Find potentially heavy elements ----------------------------------------- */ F = ∅; for ( each e ∈ R ) { if ( A[e] is potentially heavy ) { if ( A[e].counter[ h2(e) ] ≥ T ) add e to F; } else { if ( A[e] ≥ T ) add e to F; } } *** Result *** F contains all heavy elements, no false negatives ! F may contains false positives F = Count(F); // Remove false positives *** Result *** F = all heavy elements

    ---

  • Advantage of Multi-Level algorithm:

    The Multi-Level algorithm is not as sensitive to the selection probability p as the Defer-Count algorithm More efficient usage of the memory space for the counter variables

  • Problems using Multi-Level algorithm:

    How to allocate memory to the secondary counter variables ? I.e., how large should m3 be (compared to m1) ???