- Histogram is a commonly used
summary structure
to
capture the distribution of values' in a data set.
- Histograms have been used in:
- Estimating the output size of a query
- Approximating the answer of a query
- Data mining
- ...
- Histogram is a very commonly used summary structure that uses a very small number of information to represent a much larger set of values.
- Consider a
relation R
that has an
attribute X
- Notations and Definitions:
- Let V
= set of values that
X can take on
(i.e., its domain)
- f(v) = the frequency of value v (i.e., number of tuples in R where its attribute value for X is equal to v).
- Let V
= set of values that
X can take on
(i.e., its domain)
- Assumption:
- We assume
that the values in V
can be sorted,
- the ordering of the elements in V
is:
-
v1,
v2,
v3,
...
vN
- We assume
that the values in V
can be sorted,
- Frequency Vector:
- The frequency vector
F is
the vector:
-
(f1, f2,
f3, ... fN)
where f(vi) is the frequency count of the value vi
- The frequency vector
F is
the vector:
- We can draw the
frequency counts (frequency vector)
on the
Y-axis
(and use the
X-axis
to represent the
values v = {1, 2, ..., N}.
Example:
- Typically, a histogram
summarizes
a number of frequency counts
into one single count
(compression)
Example:
- The left figure depicts this fact:
- The value 2 occurs at 10 times
- The value 3 occurs at 20 times
- The value 4 occurs at 15 times
- The value 5 occurs at 25 times
- The value 6 occurs at 20 times
- The value 7 occurs at 10 times
- The value 8 occurs at 10 times
- The value 9 occurs at 30 times
- I.e.: Frequency vector F = {10, 20, 15, 25, 20, 10, 10, 30}
- A histogram at the right
summarizes
the data as follows:
- The values between [2,4) occurs at 15 times
- The values between [4,6) occurs at 20 times
- The values between [6,8) occurs at 15 times
- The values between [8,10) occurs at 20 times
- Advantage:
- the amount of data need to represent the information is reduced
- Disadvantage:
- Some accuracy
of the information is
lost
(cannot be recovered):
- e.g., we don't know the precise number of occurences of the value 2 (which is 10), and only have an estimate (15).
- Some accuracy
of the information is
lost
(cannot be recovered):
- The left figure depicts this fact:
- Given the input stream (sequence of values):
- v1, v2, v3, ...., vn
- Domain
= the range
that contains
all the values that can possibly appear in the input
- In a data stream, we do not know the domain in advance,
- Bucket:
- the domain
is divided into mutually disjoint
ranges;
- each range is called a bucket
Example:
- Domain = [10..220]
- Buckets: [10,60), [60, 90), etc.
- the domain
is divided into mutually disjoint
ranges;
- Notation:
- B = # buckets used in the construction of the histogram
Clearly, the larger the value of B, the better the histogram will approximate the original set of values (F).
But a larger value for B will also require more storage and processing...
- Error:
- Data extracted
from a histogram will
contain errors
- Example: 5 data points
between 10 and 60
are summarized (average)
into one data point:
After the histogram is constructed, the original set of values is no longer available
The histogram is used to "reconstruct" the original values.
Undoubtably, you will make errors in the reported values.
The obvious problem is:
- how do we minimize the error made.
- Data extracted
from a histogram will
contain errors
- Equi-Width histograms
- Partitions the domain (interval) into
equal length buckets (subranges)
- This type of histogram is not very interesting to
stream database analysis..
(no optimization possible)
- Example:
- Partitions the domain (interval) into
equal length buckets (subranges)
- V-Optimal Histogram
- Given the data frequencies
F =
{f1, f2,
f3, ... fN}.
- A V-Optimal Histogram
represents the
data frequencies in such a way that the
sum of the squared error is minimized
- (You will hear more about this type of histogram very soon)..
- Given the data frequencies
F =
{f1, f2,
f3, ... fN}.
- End-Biased Histogram
- This type of histogram maintains
-
exact counts
of items that occur with
frequency above a given threshold
and
- an approximate count of the other value (that are below that threshold)
This type of histogram is used to process iceberg queries (= that find items that stick out above other items.
The term iceberg query is due to the fact that 90% of an iceberg is under water and you only see 10% of the ice...)
-
exact counts
of items that occur with
frequency above a given threshold
and
- This type of histogram maintains