Slideshow:
Review of the
Two Pass Multiway Sort Algorithm
- Pass 1
Review of the
Two Pass Multiway Sort Algorithm
- Pass 2
Merge sort upto M−1 chunks of sorted fragments using 1 output buffer:
Maximum file size imposed by
Pass 2 of the TPMMS algorithm
Maximum file size imposed by
Pass 2 of the TPMMS algorithm
Important observation about
Pass 2 of the
TPMMS algorithm
Pass 2 limits the number of chunks:
Because we need to use 1 buffer to scan (read) 1 (sorted) chunk
Important observation about
Pass 2 of the
TPMMS algorithm
Pass 2 does not limits the size of the chunks:
Because we scan (read) the (sorted) chunks one block at a time !
How to increase the
size of the sorted chunks ?
Step 1: use M buffers to sort the input file into chunks of M blocks:
Write the sorted chunks (each chunk is M blocks) to disk !
How to increase the
size of the sorted chunks ?
Step 2a: use M− buffers to merge sort the first (M−1) chunks into a chunk of M(M−1) blocks:
Write the sorted chunk (the chunk is M(M−1) blocks) to disk !
How to increase the
size of the sorted chunks ?
Step 2b: use M− buffers to merge sort the next (M−1) chunks into a chunk of M(M−1) blocks:
Write the sorted chunk (the chunk is M(M−1) blocks) to disk ! -- And so on !!!
The 3-pass Multiway Sort Algorithm
- Pass 1
# Disk IO of Pass 1: 2 B(R)
The 3-pass Multiway Sort Algorithm
- Pass 2
# Disk IO of Pass 2: 2 B(R)
The 3-pass Multiway Sort Algorithm
- Pass 2
# Disk IO of Pass 3: 1 B(R) (don't count the last write - default is pipelining)
Maximum file size that can be sorted by the
3-pass multiway sort algorithm
Number of IOs used by the
3-pass multiway sort algorithm
Final comment....
Final comment: algorithm can be generalized to 4-pass, 5-pass, ..., n-pass
Example: the 4-pass multiway sort algorithm
Pass 1: sort input into chunks of M blocks
Final comment....
Final comment: algorithm can be generalized to 4-pass, 5-pass, ..., n-pass
Example: the 4-pass multiway sort algorithm
Pass 2: merge sort groups of (M−1) different chunks into a single chunk of (M−1)M blocks:
Final comment....
Final comment: algorithm can be generalized to 4-pass, 5-pass, ..., n-pass
Example: the 4-pass multiway sort algorithm
Pass 3: merge sort groups of (M−1) different chunks into a single chunk of (M−1)2M blocks:
Final comment....
Final comment: algorithm can be generalized to 4-pass, 5-pass, ..., n-pass
Example: the 4-pass multiway sort algorithm
Pass 4: merge upto (M−1) different chunks
Performance and Buffer requirement of the 4-pass multiway sort
algorithm
- Recall the
2-pass multiway sort
(TPMMS) algorithm:
- Pass 1:
- Divide the
input file into
chunks of
M blocks each
- Sort each chuck
individually using the
M buffers
- Write the sorted chunks to disk
Graphically:
- Divide the
input file into
chunks of
M blocks each
- Requirement:
- The number of chunks (K) ≤ M − 1
- Pass 2:
- Divide the
M buffers into:
- M − 1 input buffers
- 1 output buffer
- Use the M − 1 input buffers
to read
the K sorted chunks
(1 block at a time)
- Merge sort the
K sorted chunks
together into a
sorted file using
1 output buffer as follows:
- Find the record with the
smallest sort key among the
K buffers
- Move the
record with the
smallest sort key
to the output buffer
- When the output buffer
is full, then
write the
output buffer to
disk
- When some input buffer is empty, then read another block from the sorted chunk if there is more data
- Find the record with the
smallest sort key among the
K buffers
Graphically:
- Divide the
M buffers into:
- Pass 1:
- File size constraint:
- The maximum size of a
file that can be
sorted by the
TPMMS algorithm is:
B(R) ≤ M(M − 1)
Reason:
- In pass 2, we can
allocate
at most M−1 buffers to
read the
sorted chunks:
- Therefore, the number of chunks
that can be
generated by
pass 1 must be:
number of chunks in pass 1 ≤ M − 1
- Since each chunk
has the size of
M blocks, the
maximum file size is
bounded by:
number of chunks in pass 1 ≤ M − 1 chunks 1 chunk = M blocks ====> File size ≤ (M − 1) × M blocks
- The maximum size of a
file that can be
sorted by the
TPMMS algorithm is:
- Consider the
pass 2 in the
TPMMS algorithm:
- The restriction is:
- # (sorted) chunks
≤ M−1
because:
- # buffers used must be ≤ M
(The TPMMS algorithm needs to use 1 output buffer to write the sorted result to disk)
- # (sorted) chunks
≤ M−1
- Notice:
- There is no restriction on
the size of
a (sorted) chunk !!!
- There is no restriction on
the size of
a (sorted) chunk !!!
- Fact:
- We can use the M buffers to merge sort (any number) ≤ M−1 sorted chunks into one larger (sorted) chunk !!!
Detailed explanation:
- Suppose we have a
very large file:
- We first
use M buffers to
sort the
file into
chunks of
M blocks:
Suppose we get > (M−1) chunks.
- We
(re)-use the
M buffers
to
merge
the first (M−1) chucks
into a
chunk
of size
M(M−1) blocks:
- Then, we
(re)-use the
M buffers
to
merge
the 2nd (M−1) chucks
into a
chunk
of size
M(M−1) blocks:
- And on on !!!
- We will have large sorted chunks !!!
- We can add the
"increase chunk" step
to the
TPMMS to create a
3-pass Multiway merge algorithm:
- Pass 1:
(same as
Phase 1 in TPMMS)
- Divide the
input file into
chunks of
M blocks each
- Sort each chuck
individually using the
M buffers
- Write the sorted chunks to disk
Graphically:
Number of disk I/Os used in step:
- B(R) disk I/Os to read the entire input file plus
- B(R) disk I/Os to write the entire file (= all the sorted chunks)
- Divide the
input file into
chunks of
M blocks each
- Pass 2:
merge
groups
of
(M−1) sorted chunks
into
a M(M−1) block
sorted
"super" chunk and
write
sorted
"super" chunk to
disk:
Graphically:
Number of disk I/Os used in step:
- B(R) disk I/Os to read the entire file (= all the chunks in the file) plus
- B(R) disk I/Os to write the entire file (= all the "super" chunks in the file)
- Pass 3:
merge upto
M−1
(much larger) sorted chunks:
- Pass 1:
(same as
Phase 1 in TPMMS)
- Maximum file size that can be
handled
by the 3-pass multiway sort:
There are no memory restrictions on pass 1 and pass 2 !!! Pass 3 must merge: ≤ M−1 chunks; Each chunk in pass 3 has size ≤ (M−1)M blocks Maximum file size ≤ (M−1)2M blocks
- Number of disk IOs used:
Pass 1: read R + write sorted chunks = 2 B(R) Pass 2: read sorted chunks + write bigger chunks = 2 B(R) Pass 3: read bigger chunks + sort = 1 B(R) (Let's not count final output write to disk...) Total = 5 B(R)
- We can add the
"increase chunk" step
twice
to the
TPMMS to create a
4-pass Multiway merge algorithm:
- Pass 1:
(same as
Phase 1 in TPMMS)
- Divide the
input file into
chunks of
M blocks each
- Sort each chuck
individually using the
M buffers
- Write the sorted chunks to disk
Graphically:
- Divide the
input file into
chunks of
M blocks each
- Pass 2:
merge groups of
M−1
sorted chunks together and
write sorted result to
disk:
Graphically:
- Pass 3:
merge groups of
M−1
bigger sorted chunks together and
write sorted result to
disk:
Graphically:
- Pass 4:
merge upto
M−1
(extremely large) sorted chunks:
- Pass 1:
(same as
Phase 1 in TPMMS)
- Maximum file size that can be
handled
by the 4-pass multiway sort:
Pass 3 merges: ≤ M−1 chunks; Each chunk in pass 3 has size ≤ (M−1)2 M blocks Maximum file size ≤ (M−1)3 M blocks
- Number of disk IOs used:
Pass 1: read R + write sorted chunks = 2 B(R) Pass 2: read sorted chunks + write bigger chunks = 2 B(R) Pass 3: read bigger sorted chunks + write biggest chunks = 2 B(R) Pass 3: read bigger chunks + sort = 1 B(R) (Again: let's not count final output write to disk...) Total = 7 B(R)
- The
k-pass multiway
merge sort algorithm:
Pass 1: Same as pass 1 of TPMMS Cost: 2 B(R) disk IOs Size of each chunk at end: M blocks
Pass 2, 3, ..., k-1 ((k-2) passes): Use the "increase chunk size" algorithm Cost per pass: 2 B(R) disk IOs Cost for k-2 pases: 2(k-1) B(R) disk IOs Size increase per pass: (M-1) times Size of chunks at end: (M-1)k-2 M blocks
Pass k: Same as pass 2 of TPMMS Cost: B(R) disk IOs (do not count output) Size of sorted file ≤ (M-1)k-1 M blocks
- Total # disk IOs performed by the
k-pass multiway merge algorithm:
# disk IOs = 2(k-1) B(R) (if we don't count final output IO) Or: # disk IOs = 2 k B(R) (if we include final output IO)
Max file size ≤ (M-1)k-1 M blocks
- Note:
- The text book reports a
simplified (approximate)
result:
# disk IOs = 2 k B(R) blocks (includes the final output IO operations) Max file size ≤ Mk blocks
- The text book reports a
simplified (approximate)
result: