Slideshow:
The duplicate elimination operator (δ)
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Pass 1
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Pass 2
Pass 2:
(Graphically explained in next slide)
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Pass 2
Pass 2:
You only need a program variable to store 1 tuple for the output (i.e.: not an entire buffer)
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Example
- Pass 1
Input file:
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Example
- Pass 1
Pass 1: read chunks of M blocks and sort on entire tuple
Write the sorted chunks to disk
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Example
- Pass 2
Pass 2: read 1 block from every (sorted) chunk and find the next smallest tuple
If smallest tuple == last seen tuple then discard it (and repeat)
The 2-pass TPMMS based duplicate elimination
(δ) algorithm
- Example
- Pass 2
Pass 2: read 1 block from every (sorted) chunk and find the next smallest value
If smallest tuple > last seen tuple then output it and update the last seen tuple
Cost analysis of
the 2-pass TPMMS based duplicate elimination
(δ) algorithm
Buffer requirement of
the 2-pass TPMMS based duplicate elimination
(δ) algorithm
Buffer requirement of
the 2-pass TPMMS based duplicate elimination
(δ) algorithm
I.e.: if the size of relation R is B(R) blocks, we need M = Sqrt(B(R)) buffers to execute 2-pass TPMMS based δ algorithm
- Duplicate Elimination operator:
δ ( R ) - Example Duplicate Elimination:
R = { a, b, a, c, a, b } // red are duplicates δ ( R ) = {a, b, c}
- Pass 1:
(same as
TPMMS)
- Divide the
input file into
chunks of
M blocks each
- Sort each chuck
individually using the
M buffers
- Write the sorted chuncks to disk
Graphically:
- Divide the
input file into
chunks of
M blocks each
- Pass 2:
Use the M input buffer to read (at most) M sorted chunks
Repeat until all chunks are processed { Select t = tuple with the smallest sort key in non-empty chunks; if ( t has been seen before ) { discard t; } else { output t; } }Graphically:
- Example:
- Input:
- Pass 1:
sort chunks of
M blocks size
- Pass 2:
eliminate
duplicates
- Output the
key a once
and
as long as
a is
the smallest value,
we discard
the duplicates:
- When the smallest
key = b,
output the
key b
once
and
replace
the last seen value:
Now we will eliminate all duplicates of the value b
And so on
- Output the
key a once
and
as long as
a is
the smallest value,
we discard
the duplicates:
- Input:
- # disk I/O used: (assume that
R is
clustered)
- Pass 1:
Read R: B(R) Write sorted chunks: B(R)
- Pass 2:
Read sorted chunks: B(R) Write output: ≤ B(R) (not counted)
- Total: 3 B(R)
- Pass 1:
- Memory requirement:
- The relation R
can be sorted in at most
M chunks, because
in
Pass 2,
we can use at most
M buffers:
(Need 1 buffer to read 1 sorted chunk)
Note on computing buffer requirement:
- We are not
sorting as in
TPMMS
(that use 1 buffer for
output to
disk !)
- The
output tuples
may be
passed to
the next (relational algebra)
operator)
- The output tuples will then be stored in the (input) buffer of the next (relational algebra) operator
- We must not include the (input) buffer(s) of the next (relational algebra) operator in the buffer requirement for δ !!!
- We are not
sorting as in
TPMMS
(that use 1 buffer for
output to
disk !)
- Therefore:
M = # buffers available Max # chunks possible in Pass 2: M (1 buffer to read 1 sorted chunk) Max file size in Pass 2: M "chunks" (size of 1 chunk = M block) = M × M blocks
B(R) ≤ M × M blocks ≤ M2 Or: ------ M ≥ \/ B(R) buffers
- The relation R
can be sorted in at most
M chunks, because
in
Pass 2,
we can use at most
M buffers: