Slideshow:
Observation on the Cartesian Product operation
(After hashing, we still need to combine tuples in one bucket with all other buckets....))
How to perform Cartesian Product on
large data sets
Recall:
buffer requirement and IO cost of
nested-loop algorithm
- Fact:
- The cartesian product does
not use
comparison to
compute the
result
- We must match
every tuple in
input relation R to
every tuple in
input relation S
R × S ------------- a x b y ---> (a,x) (a,y) (b,x) (b,y) (c,x) (c,y) cWe cannot limit the number of tuples that must be examined !!!
- We must match
every tuple in
input relation R to
every tuple in
input relation S
- Consequently:
- Pre-processing by
hashing
cannot
reduce the
input size
by partitioning
(because all tuples must be paired against every other tuple.
I.e.: we must examine tuples in other hash buckets !!!) ...
- Pre-processing by
hashing
cannot
reduce the
input size
by partitioning
- The cartesian product does
not use
comparison to
compute the
result
- To process
R × S for
large data sets,
we can/must use the
nested-loop cartesian product
algorithm:
(See: click here)
Let M = # available buffers; /* ------------------------------------------- Outer loop: read M-1 block of S ------------------------------------------- */ while ( S has more data blocks ) { read M-1 data blocks of S; Reset R; /* ------------------------------------------- Innerloop: read through R and compute × ------------------------------------------- */ while ( R has more data blocks ) { read 1 data block of R; for ( each tuple s ∈ M-1 blocks of S and each tuple t ∈ 1 blocks of R ) do { output (s, t); } } }
- The memory requirement
of the
nested-loop
cartesian product (×) algorithm is:
M ≥ 2 bufferThe cost of the nested-loop cartesian product (×) algorithm is:
B(S) Disk I/O cost = B(S) + ------- B(R) M-1See: click here