---

A Fast quantile summary algorithm

---

• Another way to use Greenwald and Khanna's combine/prune algorithm
  • Zhang and Wang found a new way to use Greenwald and Khanna's combine/prune algorithm discussed in the previous class notes.

  • They used the combine/prune algorithms to construct an ε-approximate quantile summary

    The resulting algorithm is super-fast...

  ---

  ---

• Notations used
  • Notations and definitions

    S = a multi-level quantile summary S = { s0, s1, ..., sL } si = quantile summary at level i b = ⌊ log(εN)/ε ⌋ b = the maximum size of a quantile summary at any level i

  ---

  ---

• 2 versions of the algorithm
  • First: number of items is given a-priori (will be useful to summarize large databases

  • Next: the algorithm will be generalized for data streams where the number of elements is not known a-priori

  ---

  ---

  ---

  ---

• The fixed size algorithm (N given)
  • Algorithm:

    for ( i = i; i ≤ N; i++ ) { insert ei into s0; // insert unsorted ! if ( |s0| < b ) contiune; // stop, get another item /* ----------------------------------- Level 0 is full... PACK IT UP !!! ----------------------------------- */ sort s0; sc = compress(s0, 1/b); // Greenwald+Khanna's prune(s0, ⌈b/2⌉ + 1); s0 = ∅; for ( k = 1; k <= L; k++ ) { if ( |sk| == 0 ) { /* -------------------------------------- Empty: put compressed summary in sk -------------------------------------- */ sk = sc; // Store it break; // Done } else { /* -------------------------------------- sk contained a compressed summary -------------------------------------- */ tmp = merge(sk, sc); // Greenwald+Khanna's combine(sk, sc); sc = compress(tmp, 1/b); // Greenwald+Khanna's prune(tmp, ⌈b/2⌉ + 1); // NOTE: sc is used in next iteration // - it is passed to the next level ! sk = ∅; // Re-initialize } } }

    ---

  • NOTE:

    The compress step is Greenwald and Khanna's prune algorithm            This step will introduce an error of 1/(2B) to the total error. Since B = ⌈b/2⌉, this step will increase the error by 1/b For Greenwald and Gahanna's material, see: click here

    ---

  • Graphically:
    • Initial state:

      ---

    • Initially, elements fill up s₀:

      ---

    • s₀ becomes full:
      1. Compress:

      2. Store or merge::

      ---

    • Elements continues to fill up s₀:

      ---

    • s₀ becomes full again:
      1. Compress:

      2. Store or merge:: merge !

      ---

    • Elements continues to fill up s₀:

    • And so on !!!

  ---

  ---

• Overview of the summary
  • To understand some of the theorems, it is good to know how the multi-level summary represent the entire input set

  • How the input are represented in the multi-level summary

    ---

  • Property:

    The summary sk: covers b × 2k items is an (k/b)-approximate quantile summary

  ---

  ---

• Lemma 1: number of levels L
  • Lemma 1:

    The number of levels L is < log(εN)

  • Proof:

    The number of items is equal to N After every b items, the summary s0 becomes full So s0 becomes full at most N/b times After every 2×b items, the summary s1 becomes full So s1 becomes full at most N/(2×b) times After every 4×b items, the summary s2 becomes full So s2 becomes full at most N/(4×b) times And so on... After every 2L×b items, the summary sL becomes full So sL becomes full at most N/(2L×b) times In an L-level summary, sL becomes full at most once !!!              Therefore: N/(2L×b) ≤ 1 <==> 2L ≤ N/b <==> L ≤ log(N/b) (b = ⌊ log(εN)/ε ⌋) = log( εN/log(εN) ) = log( εN ) - log( log(εN) ) < log( εN )

  ---

  ---

• Lemma 2: accuracy of the summary
  • Lemma 2:

    Each level of the quantile summary has an error that is less than εN

    Proof:

    The error in s0 is 0 (every element in the stream is represented) Hence: ε0 = 0             At level 1, the merge step will not increase the error margin But the compress step will increase error by 1/b according to click here Therefore: ε1 = 1/b              At level 2, the merge step will also not increase the error margin But the compress step will again increase error by 1/b according to click here Therefore: ε2 = 2/b              In general, the error margin in the quantile summary at level k is: εk = k/b              With: b = ⌊log(εN)/ε⌋ (see: click here), we have: εk = k/(⌊log(εN)/ε⌋)      ≤ k/(log(εN)/ε)      = ε × k/(log(εN)              The largest error occurs at level k = L and from Lemma 1, we know that: L < log(εN)              Therefore: εk ≤ ε × k/(log(εN)             k < log(εN)                < ε × log(εN)/(log(εN) εk < ε             Q.E.D.

  ---

  ---

  ---

  ---

• Using the multi-level quantile summary
  • The procedure to answer a ε-approximate quantile query is a bit complicated...

  • Procedure for answering a quantile query for element that ranks r:

    Sort the level 0 quantile "summary" s0 Merge all quantile summaries: S = s0; for ( i = 1; i ≤ L; i++ ) S = merge(S, si); (The merge operation will not increase the error !!!)              Then find the element v that has: rmin(v) ≥ r - εN rmax(v) ≤ r + εN              (Just like in Greenwald and Ghanna)