Slideshow:
Review:
Two-Pass algorithms bashed on hashing
- Pass 1
Pass 1:
Review:
Two-Pass algorithms bashed on hashing
- Pass 2
Pass 2:
Important observations on
the Two-pass algorithms based on
hashing
Observation 1:
Important observations on
the Two-pass algorithms based on
hashing
Observation 2:
(This memory buffer requirement was imposed by the One-pass algorithm run on each fragment)
Example::
Important observations on
the Two-pass algorithms based on
hashing
Observation 3:
Idea: create more smaller fragments ("sub-relations") so that each fragment ≤ (M−1) blocks !
How to create (more) smaller
sub-sub-relations ?
The 3-pass hash based data processing
algorithms
The 3-pass hash based data processing
algorithms
The 3-pass hash based data processing
algorithms
IO costs of
the 3-pass hash based data processing
algorithms
Max file size processed by
the 3-pass hash based data processing
algorithms
- Recall the
2-pass algorithms
based on hashing:
- Pass 1:
- Use 1 input buffer to
read an input relation
- Use the remaining M−1 buffers
as M−1 buckets
- Hash the input relation into M−1 buckets
Graphically:
- Use 1 input buffer to
read an input relation
- Pass 2:
- Process
each sub-relation (bucket)
(or pairs of
sub-relations)
using some
one-pass algorithm
Example: δ
- Process
each sub-relation (bucket)
(or pairs of
sub-relations)
using some
one-pass algorithm
- Pass 1:
- File size constraint:
- The maximum size of a
relation/file that can be
processed using a
2-pass hashing based algorithm is:
B(R) ≤ (M − 1)2
because:
- All the
non-duplicate tuples of
Ri must
fit in the search structure
using
M−1 buffers:
Therefore, in the worst case (when all tuples are unique):
B(Ri) ≤ M−1 blocks
- The best case scenario
of the hashing is:
- Every sub-relation has
the same size:
1 B(Ri) = --- B(R) M-1(Otherwise, some sub-relation will larger than B(R)/(M−1) !!!)
Therefore:
B(Ri) ≤ M−1 blocks and 1 B(Ri) = --- B(R) M-1
We have: 1 --- B(R) ≤ M−1 blocks M-1 <==> (M-1)2 ≥ B(R) B(R) ≤ (M-1)2 - Every sub-relation has
the same size:
- The maximum size of a
relation/file that can be
processed using a
2-pass hashing based algorithm is:
- Fact:
- There is no size restriction
imposed
on the input relation by
pass 1 of the algorithm:
Pass 1 can hash an arbitrarily large input relation into (maximum) M−1 buckets
- The restriction is on
the number of
buckets:
- Number of
hash buckets
≤
(M−1)
(Because we only have M buffers available !!!)
- Number of
hash buckets
≤
(M−1)
- There is no size restriction
imposed
on the input relation by
pass 1 of the algorithm:
- Consider the
pass 2 in any of the
hash-based algorithm:
- Pass 2
imposes
the memory requirement:
- The entire sub-relation (bucket) must fit within the available M−1 buffers
- Notice that:
- Pass 2 has
no restriction on
the number of
sub-relations
(because we process each (one or pair) sub-relation independently from one another)
- Pass 2 has
no restriction on
the number of
sub-relations
- It is very easy to
create (more)
smaller sub-relations
in hashing:
- We just hash each sub-relation again (using a different hash function - obviously)
Graphically:
- We can
add the
"reduce sub-relation size" step to
the 2-pass algorithm to
create a
3-pass hashing based algorithm:
- Pass 1:
hash the
input relation(s)
into M−1 sub-relations
(same as pass 1 of the 2-pass algorithm)
Each sub-relation will have (approximately) the size B(R)/(M−1) blocks
- Pass 2:
hash each
sub-relation
into M−1 sub-sub-relations
Each sub-sub-relation will have (approximately) the size B(R)/(M−1)2 blocks
- Pass 3:
run the one-pass algorithm for the
relational algebra operator on
each sub-sub-relation:
- Pass 1:
hash the
input relation(s)
into M−1 sub-relations
- Maximum file size that can be
handled by the
3-pass hashing based algorithm:
Each sub-sub-relation must fit in ≤ M-1 buffers B(R) Size of each sub-sub-relation = -------- (M-1)2 B(R) Therefore: ------ ≤ M-1 blocks (M-1)2 Or: B(R) ≤ (M-1)3 blocks
- Number of disk IOs used:
Pass 1: read R + write M-1 sub-relations = 2 B(R) Pass 2: read M-1 + write (M-1)2 sub-relations = 2 B(R) Pass 3: run one-pass algorithm = B(R) Total # disk IOs = 5 B(R)
- The
k-pass
relational algebra algorithms
based on
hashing:
Pass 1: Hash input relation(s) into M-1 buckets (Same as pass 1 of the 2-pass algorithm) Cost: 2 B(R) disk IOs Size of each sub-relation at end: B(R)/(M-1) blocks
Pass 2, 3, ..., k-1 ((k-2) passes): Use the "reduce sub-relation size" algorithm Cost per pass: 2 B(R) disk IOs Cost for k-2 pases: 2(k-1) B(R) disk IOs Size decrease per pass: 1/(M-1) times Size of sub-sub-..-relations at end: B(R)/(M-1)k-1 blocks
Pass k: run a one-pass relational alg. algorithm Cost: B(R) disk IOs (do not count output) Constraint on size: B(R)/(M-1)k-1 ≤ M-1 blocks
- Total # disk IOs performed by the
k-pass multiway merge algorithm:
# disk IOs = 2(k-1) B(R) + B(R) (if we don't count final output IO) = 2 k B(R) (if we include final output IO) Max file size ≤ (M-1)k blocks