Finding the V-Optimal histogram (part 1) - Minimizing the squared error

Minimizing the squared error

We will first deal with the second problem (which is the easiest):
In this problem, the ranges of the bucket have been fixed

We must minimize the error of the histogram:

minimize:

SqError(histogram) = ∑_r=1..B ( ∑_{k=s_r..e_r} (h_r - f_k)² )

Or: minimize: SqError(histogram) = ∑_k=s₁..e₁ (h₁ - f_k)² + ∑_k=s₂..e₂ (h₂ - f_k)² + ... + ∑_{k=s_B..e_B} (h_B - f_k)²

Because the ranges of the buckets are fixed, the values:
are constant

Therefore, the sum

SqError(histogram) = ∑_k=s₁..e₁ (h₁ - f_k)² + ∑_k=s₂..e₂ (h₂ - f_k)² + ... + ∑_{k=s_B..e_B} (h_B - f_k)²

is minimized if each term is minimized.

Consider an aribitrary term in the expression of SqError(histogram) ( which is the error of some bucket b_r):

S_r = ∑_{k=s_r..e_r} (h_r - f_k)² or:
S_r = (h_r - f_{s_r})² + (h_r - f_{s_r+1})² + (h_r - f_{s_r+2})² + .... + (h_r - f_{e_r})²

Each of the f_... is constant value
The only variable in S_r is h_r
To find the minimum, we differentiate S_r with respect to h_r:

We solve S'_r = 0:

2 (h_r - f_{s_r}) + 2 (h_r - f_{s_r+1}) + 2 (h_r - f_{s_r+2}) + .... + 2 (h_r - f_{e_r}) = 0
<=> (h_r - f_{s_r}) + (h_r - f_{s_r+1}) + (h_r - f_{s_r+2}) + .... + (h_r - f_{e_r}) = 0
<=> p × h_r - (f_{s_r} + f_{s_r+1} + f_{s_r+2} + .... + f_{e_r}) = 0
<=> p × h_r = f_{s_r} + f_{s_r+1} + f_{s_r+2} + .... + f_{e_r}
<=> h_r = Average(f_{s_r} , f_{s_r+1} , f_{s_r+2} , .... , f_{e_r})

Conclusion:

The (minimum) error in a bucket
- The error in the bucket b_r is then: (see click here)

The need to compute S_r = (f_{s_r}² + f_{s_r+1}² + .... + f_{e_r}²) - (f_{s_r} + f_{s_r+1} + ... + f_{e_r})²/p efficiently

Context of the discussion:

We have not yet found the optimal partitioning (buckets) of the histogram boundaries
I.e., we do not yet know the optimal buckets.

All we did so far is

When we are given a bucket division of the histogram, we can:
1. Find the optimal value h_r for each bucket
2. Find the error S_r of each bucket

During the process of finding the optimal partitioning (buckets) of the histogram boundaries, we will need to compute the error value S_r for various bucket boundaries

In other words:

The process of finding the optimal partitioning (buckets) is a search process
During the search, we need to compute the error S_r very frequently (so see which division is better

So we have to compute S_r often, and we must find a way to compute it efficiently.

Example:

Suppose in one bucket partitioning the bucket b_r spans the following range:

To compute the error of the histogram, we must compute the error of each bucket separately:

for each bucket b_r compute:

Example: in the above figure:

S_r = (f₃² + f₄² + .... + f₈²) - (f₃ + f₄ + ... + f₈)²/6

Now suppose another bucket partitioning the bucket b_r spans the following range:
Problem:
Example: in the above figure:

Question:

A programming trick to compute S_r = (f_{s_r}² + f_{s_r+1}² + .... + f_{e_r}²) - (f_{s_r} + f_{s_r+1} + ... + f_{e_r})²/p efficiently

Programming Trick to compute S_r efficiently:

Use 2 arrays P[] and PP[]:

An array P[] with:
An array PP[] where:

Notice that:
And:
And one more:

You can find any 2 subrange sums easily as a difference between 2 array elements of P[]

Example:

f₁ + f₂ + ... + f₃₀ = P[30] - P[0]
f₅ + f₆ + ... + f₃₀ = P[30] - P[4]
f₂₅ + f₂₆ + ... + f₇₈ = P[78] - P[24]

Similarly that:

PP[1] - PP[0] = f₁²
PP[2] - PP[0] = f₁² + f₂²
PP[3] - PP[0] = f₁² + f₂² + f₃²
...
PP[k] - PP[0] = f₁² + f₂² + ... + f_k²

And:

PP[2] - PP[1] = f₂²
PP[3] - PP[1] = f₂² + f₃²
PP[4] - PP[1] = f₂² + f₃² + f₄²
...
PP[k] - PP[1] = f₂² + f₂² + ... + f_k²

And one more:

PP[3] - PP[2] = f₃²
PP[4] - PP[2] = f₃² + f₄²
PP[5] - PP[2] = f₃² + f₄² + f₅²
...
PP[k] - PP[2] = f₃² + f₂² + ... + f_k²

You can find any 2 subrange sums of the squares easily as a difference between 2 array elements of PP[]

Example:

f₁² + f₂ ² + ... + f₃₀² = PP[30] - PP[0]
f₅² + f₆ ² + ... + f₃₀² = PP[30] - PP[4]
f₂₅² + f₂₆ ² + ... + f₇₈² = PP[78] - PP[24]

Lemma 2: more buckets will always achieve smaller error

Lemma 2 in the paper click here shows that any solution using more buckets is better than one that uses less buckets.

Lemma 2:

For any k between (a,b):

Interpretation of this result:

Consider the frequency counts for the values [a..b] ( f_a, f_a+1, ..., f_b )
If we use a single-bucket histogram to represent the values f_a, f_a+1, ..., f_b , then the smallest possible error is S_[a,b]:
If we use a two-bucket histogram to represent the values f_a, f_a+1, ..., f_b , then we will split the interval [a..b] into two intervals [a..k] and [k+1..b]:
Lemma 2 says that:

For the proof, see the paper (not discussed in class)

Next...

Now that we know how to find the minimum error for a given bucket distribution, the next problem is:

Search for the best bucket ranges (where the number of bucket is equal to B)
In other words: