- We have seen that a
representative sample
of a join-result
must use a weighted sample
The weight w(t) of a tuple t is equal to the frequency of occurence in the other relation
- Therefore:
- a representative weighted
sample can only be constructed if we
have the
information on the
data distribution
in the input stream...
- The more information we have on the data data distribution, the better the weighted sample we can construct...
- a representative weighted
sample can only be constructed if we
have the
information on the
data distribution
in the input stream...
- The paper
(click here)
distinguishes the following cases:
- No information available
on R1 and
R2:
Naive-Sample
.
- No information on R1
and
Index/Statistics info. on R2:
Stream-Sample
- this is new research presented in the paper - No information on R1
and
only Statistics info. on R2:
Group-Sample
- this is new research presented in the paper - No information on R1
and
partial Statistics info. on R2:
Frequency-Partition-Sample
- this is new research presented in the paper
- No information available
on R1 and
R2:
Naive-Sample
.
- Two types of information
are useful to construct
sample:
- Index information:
-
When a relation R is
indexed,
a specific tuple t
can be found without having to
search in the relation
So index means: fast access to any tuple
- Statistics information:
-
Statistics on a relation
R tells us
the number of tuples have
a certain value in a attribute
The statistics information can be used as weights in a weighted sample
- Index information:
- If no information available
about the inputs
R1
and R2,
there is no way
to obtain a representative result
by sampling
the inputs
(R1
and R2,)
- A representative (smaller) sample
of the join operation, can only be
obtained by
sampling the output:
- The stream-sample algorithm
can be used in the following
situation
to construct a
sample of a join
result
(without computing the full join):
- No information
is needed on R1
- R2 is indexed (fast access) and has statistics information available.
- No information
is needed on R1
- This is the scenario where:
- R1 is a
data stream
- R2 is a materialized database with an index
- R1 is a
data stream
- The Stream-Sample Algorithm:
S1 = Weighted Sample of R1 (weights obtained from statistics info on R2) (This must be done completely first)
// The sample S1 is used like a materialized relation For ( each tuple t1 ∈ S1 ) do { t2 = pick one tuple randomly from R2 such that t2.A = t1.A (NOTE: we can use index info in this step to find tuples t2 with A value equal to t1.A) Output t1 * t2 (join) }
- The Group-Sample algorithm
can be used in the following
situation
to construct a
sample of a
join result
(without computing the
full join):
- No information
on R1
is necessary
(stream)
- R2 is not indexed but we do have statistics info. (i.e., we know how many tuples has A=1, A=2, etc...).
- No information
on R1
is necessary
(stream)
- This is the scenario
where:
- R1
is a general input data stream
and
- R2 is a materialized database without an index
- R1
is a general input data stream
and
- The Group-Sample algorithm
S1 = Weighted Sample of R1 (weights obtained from statistics info on R2) (The weight w(t) of tuple t is the frequency count of t.A in R2 !!!) (This must be done completely first)
For ( each tuple si &isin S1 ) do { Xi = si * R2 // Join tuple si with every tuple in R2 // NOTE: We must do this, because R2 is not indexed, // (we must read through all tuples in R2) Select one tuple from Xi Output the selected tuple }
- Needs information on most frequent items
- E.g.: Histogram
- skipped... because we must learn about
end-biased histograms