---

Merge operation for Frequency Count

---

• Flashback on Frequency Counts
  • Recall that Manku's summary has the following form:

    {(e1, f1, Δ1), (e2, f2, Δ2), ..., (ek, fk, Δk)}              Property of the summary: f - εN   ≤   fi   ≤   f              f = actual frequency count of the item i fi = estimated frequency count of the item i N = number of items in the summary

  ---

  ---

• Frequency count summary for a window
  • When applying Arasu's algorithm to construct ε-approximate frequency count summary for a window, run Manku's algorithm with error parameter:

    2L 1 εi = ---- × -------- × ε (see: click here) 2i 4(L+1)

    (You will see later that quantile summaries requires higher accuracy)

  ---

  ---

• Merge (combine) algorithm for Frequency Count
  • Algorithm:

    Let B1, B2, ..., Bs be the best active blocks that covers the current window Let A1, A2, ..., As be their Frequency Count summaries A = ∅; // Freq. count summary for current window for ( k = 1; k ≤ s; k++ ) { for ( each (e,f,Δ) ∈ Ak ) do { if ( e ∈ A ) { A.e.f = A.e.f + Ak.e.f; // Add count to total } else { Add (e,f) to A; // Add new entry to window summary } } }

  ---

  ---

• Accuracy of the merge operation in respect to the covered data items
  • Lemma 1:

    Let B1, B2, ..., Bs be a collection of disjoint "bags" (or active blocks) The number of elements in B1, B2, ..., Bs is N1, N2, ..., Ns, respectively. Let A1, A2, ..., As be s sketches over B1, B2, ..., Bs, respectively, with error parameters of ε1, ε2, ..., εs. The the merge algorithm computes an ε-approximate frequency count summary for the union of the bags B1 ∪ B2 ∪ ... ∪ Bs with error parameter: ε1N1 + ε2N2 + ... + εsNs ε = ------------------------- N1 + N2 + ... + Ns

    Proof:

    Basically, we must prove that for any element e ∈ B1 ∪ B2 ∪ ... ∪ Bs: f − ε(N1 + N2 + ... + Ns)   ≤   fe   ≤   f                     f = actual frequency fe = estimated frequency (provided by the summary) The inequalities can be re-written as follows using the fact that: ε1N1 + ε2N2 + ... + εsNs ε = ------------------------- N1 + N2 + ... + Ns re-written as: f − (ε1N1 + ε2N2 + ... + εsNs)   ≤   fe   ≤   f            .... (1)                      Because summary Ai has error parameter εi, we have for any item ∈ Bi: fi − εiNi   ≤ fei   ≤ fi fi = actual frequency of item in Bi fei = frequency estimate of item in Ai                   Therefore, the actual frequency count of item ∈ B1 ∪ B2 ∪ ... ∪ Bs is: f = f1 + f2 + ... + fs              And the estimated frequency of item ∈ B1 ∪ B2 ∪ ... ∪ Bs is: fe = fe1 + fe2 + ... + fes              From Equation (1), it follows that: f − (ε1N1 + ε2N2 + ... + εsNs)   ≤   fe   ≤   f            Q.E.D.

  ---

  ---

• Warning on Lemma 1
  • Notice that Lemma 1 concerns with the accuracy of the merge operation concerning only the data items in the union of the bags (blocks)

    We are concerned with a sketch of the data items in the current window !!

  • We saw that the union of the active block is not equal to the entire current window:

  ---

  ---

• Accuracy of the merge operation in respect to the data items in the current window
  • Theorem 1:

    Let W be the maximum window size Let the current window be the bag of elements { em−N+1, em−N+2, ..., em } Then: The merge algorithm allows (εW)/N-approximate frequency counts to be computed over the elements { em−N+1, em−N+2, ..., em } Note: if N is always equal to W, then the merge algorithm will compute a ε-approximate frequency counts summary

    Proof:

    Let BW be the current window with maximum width W Suppose BW = { em−N+1, em−N+2, ..., em } (N ≤ W) The active blocks does not cover BW completely. Define: α = smallest integer such that α × [(εW)/4]   ≥  m − N              β = largest integer such that β × [(εW)/4]   ≤  m Graphically speaking: Define: Big = "bag" of ignored elements Big = { ei ∈ BW   |   m − N   <   i   ≤   α × [(εW)/4]   or   β × [(εW)/4]  <  i  ≤  n }             Bcon = "bag" of considered elements Bcon = { ei ∈ BW   |   α × [(εW)/4] + 1   ≤  i  ≤  β × [(εW)/4] } Claim: Bcon can be expressed as a union of k non-overlapping ACTIVE block where:              k   ≤   2(L + 1)          The paper did not provide a proof, but it is pretty obvious: All but (at most) 2 of the level 0 blocks are covered by level 1 blocks !!! Similarly, all but (at most) 2 of the level 1 blocks are covered by level 2 blocks... And so on... Hence: number of level 0 blocks in Bcon is ≤ 2 number of level 1 blocks in Bcon is ≤ 2 ... number of level L blocks in Bcon is ≤ 2 Therefore: k   ≤   2(L + 1)          Let: Let Bcon = B1 ∪ B2 ∪ ... ∪ Bk             with k ≤ 2(L+1) Let A1, A2, ..., Ak be the sketches (summaries) for B1, B2, ..., Bk, repsectively ( A1, A2, ..., Ak together makes up Arasu's approximate frequency count summary) Define: Aig = ∅            This is the "summary" for the ignored (uncovered) elements Big The error parameter for Aig is: εig = 1            We now have: BW = Bcon ∪ Big             From Lemma 1, we can compute an ε-approximate frequency count summary A for BW using A1, A2, ..., Ak       and            Aig !!!! The error parameter of this summary will be: ε1N1 + ε2N2 + ... + εkNk + εigNig εA = ------------------------------------ N1 + N2 + ... + Nk + Nig ε1N1 + ε2N2 + ... + εkNk + εigNig = ------------------------------------ (εiNi = εW/(4L+4) - see: click here) N k×[εW/(4L+4)] + εigNig = ------------------------ (k ≤ 2(L+1)) N εW/2 + εigNig ≤ --------------- (εig = 1) N εW/2 + Nig = --------------- N The number of elements in Big is at most 2×(εW/4) (two level 0 blocks): Therefore: εW/2 + Nig εA ≤ ------------ N εW/2 + εW/2 ≤ ------------- N εW = ------ Q.E.D. N

  ---

  ---

• Space requirement
  • The paper also shows that the amount of memory needed for the summary is:

    O( (1/ε) × log2(1/ε) )

    I will not discuss the proof of this..