Finding Maximum frequency and window with minimum window size restrictions

Problems with the base algorithm

Problems:

Generate a very large number of candidate windows

Solution:

This problem can be solved by specifying a minimum support (frequency) so that borders that do not meet the minimum frequency will not be included in the summary

Minimum window size = 1

This problem will be handled through a lemma
It turns out that there is a simple way to adapt the base algorithm to find the maximum window and maximum frequency with a minimum window size constraint

Satisfying the Minimum (support) Frequency requirement

Claim:

If freq(A, S_t[p_i, t]) < min_freq, then location p_i will not be the starting position of the maximum window (among those that have a frequency that is greater than min_freq)

Proof:

Given:

S: ........ p_i ........... p_t | | +----------------+ freq(A, S_t[p_i, t]) < min_freq

Suppose there is a continuation that will make the frequency in the window starting at p_i greater than or equal to min_freq:

S: ........ p_i ........... p_t .......... p_t' | | +--------------------------------+ freq(A, S_t[p_i, t']) ≥ min_freq

In that case, it must be because the frequency in the continuation is larger than min_freq:

S: ........ p_i ........... p_t .......... p_t' | | +-------------+ freq(A, S_t[p_t+1, t']) ≥ min_freq

Then:, p_i is not the maximum window !
(Because the window starting at p_t+1 has a larger frequency !)

Satisfying the Minimum Window Length (mwl) requirement

Note:
It turns out that the base algorithm can be used to find the maximum window (and maximum frequency) when the minimum window size is at least equal to mwl

Theorem 2

Let S = I₁, I₂, ...., I_L be the stream
Let mwl = minimum window length

Let S^-mwl = stream S without the last mwl items:

S: I₁ I₂ ..... I_L-mwl I_L-mwl+1 ..... I_L | | | | +--------------------+ +----------------+ S^-mwl last mwl items

If q is the start position of the maximum window of minimum length mwl in stream S (q = startmax^mwl(A, S), then:

Either: q = L - mwl + 1
Or: q is a border of stream S^-mwl without the minimum size restriction

Proof:

The start positions of a window of size mwl are:

S: I₁ I₂ ..... I_L-mwl I_L-mwl+1 ..... I_L <---------------------| | possible +----------------+ starting last mwl items positions

Assume that q is the start position of the maximum window of minimum length mwl in stream S

Case 1: q = L - mwl + 1

Case 2: q < L - mwl + 1

Then we must show that: q is a border position of stream S^-mwl:

I₁ I₂ .... I_q-1 I_q .... I_L-mwl ^ | q is a border

It is given that q is a maximum window in stream S:

I₁ I₂ .... I_q-1 I_q .... I_L-mwl .... I_L | | +---------------------+ max window of S (The fact that the size of this maximum window is at least equal to mwl, implies that: q ≤ L-mwl)

From Theorem 1 (click here) on the maximum window, we have:

I₁ I₂ .... I_p ..... I_q-1 I_q .... I_r .... I_L | | | | +-------------+ +---------+ ∀ p ≤ q-1, r ≥ q: freq(A, S[p, q-1]) < freq(A, S[q, r])

Now considering this property restricted to only items in the stream S^-mwl:

I₁ I₂ .... I_p ..... I_q-1 I_q .... I_r .... I_L-mwl [ ..... I_L ] | | | | +-------------+ +---------+ ∀ p ≤ q-1, r ≥ q: freq(A, S[p, q-1]) < freq(A, S[q, r])

According to Corollary 1 (click here), the location q is a border position in the stream S^-mwl

Q.E.D.

How to use Theorem 2 to find maximum window with size constrain

Theorem 2 tells us that:

The start position q of the maximum window of minimum length mwl in stream S (q = startmax^mwl(A, S), is:
This theorem give us the all the possible candidate start positions for the maximum window with minimum window size mwl
We still have to find which position is the actual start position of the maximum window
This is done with a search:
We can find the frequency of item set A at location q = L - mwl + 1 easily.
The main tricky task:
Answer:
To compute the border locations of S^-mwl, we use the input item sets from S, but leave out the last mwl items !!!
This can be done with a queue added before Calder's Algorithm

To find a maximum window with minimum window size mwl, we can do this:

Run the base algorithm of stream S^-mwl
(Keep the same summary information, except don't consider the last mwl items)
You can implement this easily by putting a queue between the arriving items and Calders' Algorithm:
The queue contains last mwl item sets from the input stream
The queue will only output an item to Calder' Algorithm if it overflows
When the base algorithm finishes, we compute the frequencies:
The one with the highest frequency is the maximum window

Calder's Algorithm with Minimum Window Size restriction:

S_t = ∅; ***** Notice the change: S ---> S^-mwl while ( not end of stream S^-mwl ) do { I = next item set; // S_t = current summary // S_t+1 = summary after processing the next item set S_t+1 = ∅; // New summary information if ( S_t == ∅ ) { if ( A ⊆ I ) { S_t+1 = [(t+1, 1)]; // One border } } else { if ( A ⊆ I ) { /* ----------------------------------------- An addition of A will either: 1. extend the last border region 2. create a new border It will not remove existing border ----------------------------------------- */ if ( a_r == t - p_r + 1 ) { // Situation: .... A A A A A S_t+1 = S_t; Update: a_r = a_r + 1; // extend last border } else { // Situation: .... A A A A O S_t+1 = S_t ⊕ [(t+1), 1]; // create new border } } else { /* ----------------------------------------- An addition of not-A may cause a border to disappear... ----------------------------------------- */ S_t+1 = S_t; // Original set of border... // Remove borders that do not satisfy the property (click here) i = r; while ( i > 1 ) { a_i a_i-1 + a_i if ( ------------- ≤ ------------- ) t - p_i + 1 t - p_i-1 + 1 { a_i-1 = a_i-1 + a_i; // Merge the counts in 2 consecutive segments delete entry (p_i, p_i) from S; i++; } else { break; } } } } } /* ----------------------- Maximum window ----------------------- */ Compute the frequency in: 1. S[L-mwl+1, L] 2. S[p, L] for p ∈ Border(S^-mwl) Max. window = [x..t] where frequency is maximized

Satisfying both the minimum window and the minimum frequency (support) requirements

Theorem 3:

Let S = I₁, I₂, ...., I_L be the stream
Let mwl = minimum window length

Let S^-mwl = stream S without the last mwl items:

S: I₁ I₂ ..... I_L-mwl I_L-mwl+1 ..... I_L | | | | +--------------------+ +----------------+ S^-mwl last mwl items

Suppose item A has a frequence greater than or equal to the minimum frequency:

If q is the start position of the maximum window of minimum length mwl in stream S (q = startmax^mwl(A, S), then:

Either: L - 2mwl + 2 ≤ q ≤ L - mwl + 1
Or:
1. q is a border of the stream S^-mwl, and
2. mfreq¹(A, S^-mwl) ≥ min_freq and
3. freq(A, S[q, q+mwl-1]) ≥ min_freq

Proof:

How to use Theorem 3:

The start position of the maximum window of minimum length mwl in stream S (q = startmax^mwl(A, S), can be:

L - 2mwl + 2 ≤ q ≤ L - mwl + 1

To accomodate this case:

We buffer the last 2 × mwl items
At the end of the stream, compute the frequency at each of these (2 × mwl) point
In addition to checking the best border, we must also check the frequency at each of these (2 × mwl) point to find the actual maximum window

The start position of the maximum window of minimum length mwl in stream S (q = startmax^mwl(A, S), can also be a position that satisfies all these properties:

q is a border of the stream S^-mwl, and
mfreq¹(A, S^-mwl) ≥ min_freq and
freq(A, S[q, q+mwl-1]) ≥ min_freq

How to use this result:

Property 1 tells us to run the base algorithm on input stream S^-mwl to find the border positions
(We have seen in the previous case on how this can be achieved with a queue put in front of Calder's base Algorithm)
Property 2 help us eliminate the unnecessary borders that will not produce sufficiently large frequencies (and reduce the size of the summary)
Property 3 also help us eliminate some unnecessary borders that will not produce sufficiently large frequencies