---

Adapting the quantile summary for data streams

---

• Handing data streams
  • The stream is partitioned into a undetermined number of disjoint substreams.

  • The size of the substreams are increasing as follows:

    substream 1:    1/ε elements              substream 2:    2/ε elements substream 3:    4/ε elements substream 4:    8/ε elements ...

    Graphically:

    ---

  • By partitioning the stream into substreams of known sizes, we can construct a multi-level summary using the previously described technqiue !

  ---

  ---

• Constructing the multi-level summary for a stream
  • Procedure:

    For each substream construct a &epsilon' = ε/2 multi-level summary: When a substream Pk completes, we compute: an ε/2-summary by merging all levels compress (prune) the result using: COMPRESS( ..., ε/2) (or prune( ..., 2/ε + 1)) into an ε-approximate quantile summary of size 2/ε + 1: Graphically:

    ---

  • Algorithm:

    S = overall quantile summary SC = quantile summary for the current substream N = size of the current substream S = ∅; SC = ∅; k = 0; N = 1/ε; while ( not EOS ) { e = next element in stream; n++; /* ----------------------------------------------------------- Add e to current quantile summary with error margin ε/2 ----------------------------------------------------------- */ SC = Update(e, SC, ε/2); if ( n == N /* SC is full */ ) { /* --------------------------------- Merge all levels of SC --------------------------------- */ sall = s0; for ( i = 1; i <= L; i++ ) { sall = Merge(sall, si); } /* ------------------------------------------------ Compress to an ε-approximate quantile summary ------------------------------------------------ */ Sk = compress( sall, ε/2); /* ------------------------------------------------ Store away ------------------------------------------------ */ S = S ∪ Sk; SC = ∅; // Re-initialize n = 0; N = 2×N; } }

  ---

  ---

• Using the stream quantile summary
  • To answer a quantile query of rank r using S:

    if ( SC != empty; ) { /* --------------------------------- Merge all levels of SC --------------------------------- */ sall = s0; for ( i = 1; i <= L; i++ ) { sall = Merge(sall, si); } /* ------------------------------------------------ Compress to an ε-approximate quantile summary ------------------------------------------------ */ Sk = compress( sall, ε/2 ); } /* -------------------------------------------------- Construct ε-approximate quantile summary -------------------------------------------------- */ S = S0 for ( i = 1; i ≤ k; i++ ) S = Merge(S, Si); Use S to answer quantile query;

  ---

  ---

• Performance Analysis
  • This method is super space efficient and super fast...

  • Theorem 3:

    The space requirement of the stream algorithm is O(   (log(εn))2/ε   )               I.e., it grows only with log(n)2 !!!

  • Theorem 4:

    The average update cost (run time complexity) of the stream algorithm is O(   log( log(εn)/ε )   ) I.e., the per item processing cost is extremely low (that's because most of the insertion are in s0 and it is handled by one operation)

    ---

  • Performance data: running time vs. input size

    Note: log scale !!!

    ---

  • Performance data: running time vs. accuracy

    Note: log scale !!!