Slideshow:
Prelude to sort-based
2-pass algorithm:
sorting large files
The sort-based 2-pass algorithms are based on the TPMMS algorithm
So we will study the TPMMS algorithm first
Resource availability assumption
Question: how to sort an input file that is (much) larger than M blocks using M buffers ??
The Two-Pass Multi Way Sort Algorithm
- pass 1
Pass 1: read M blocks chunks of data and sort the data.
Write the sorted data back to disk !
The Two-Pass Multi Way Sort Algorithm
- pass 1
Pass 1: read (next) M blocks chunks of data and sort the data.
Write the sorted data back to disk !
The Two-Pass Multi Way Sort Algorithm
- pass 1
Pass 1: read (the final) M blocks chunks of data and sort the data.
Write the sorted data back to disk !
The Two-Pass Multi Way Sort Algorithm
- pass 2
Pass 2: read 1 block for each (sorted) chunk and merge sort the data
Move the next smallest key value among all (sorted) chunk to the output buffer
Write the output buffer to disk when it become full
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 1: read M blocks chunks of data and sort the data.
Write the sorted data back to disk !
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 1: read (next) M blocks chunks of data and sort the data.
Write the sorted data back to disk !
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 1: read (the final) M blocks chunks of data and sort the data.
Write the sorted data back to disk !
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 2: read 1 block for each (sorted) chunk and merge sort the data
Notice the constraint imposed by the M buffers: you can only read ≤ M−1 (sorted) chunks
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 2: read 1 block for each (sorted) chunk and merge sort the data
Find the smallest sort key value in the buffers: a
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 2: read 1 block for each (sorted) chunk and merge sort the data
Move a to the output buffer - and find the new smallest sort key value: b
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 2: read 1 block for each (sorted) chunk and merge sort the data
Move b to the output buffer - and find the new smallest sort key value... and so on.
The Two-Pass Multi Way Sort Algorithm
- Example
Pass 2: read 1 block for each (sorted) chunk and merge sort the data
When output buffer is full, write it to disk (and re-use the output buffer) !!
- 2-pass multiway merge sort:
- Traditional sort algorithms
(e.g., quick sort)
requires that the
file must
fit in the
main memory to be
sorted
- The Two-Pass Multiway Merge Sort (TPMMS) algorithm can be used to sort files that are larger than the main memory
- Traditional sort algorithms
(e.g., quick sort)
requires that the
file must
fit in the
main memory to be
sorted
- The TPMMS algorithm:
- Suppose:
- There are M buffers
available for storing
the file data
(1 buffer can hold 1 data block)
- There are M buffers
available for storing
the file data
- Pass 1:
- Divide the
input file into
chunks of
M blocks each
- Sort each chuck
individually using the
M buffers
- Write the sorted chunks to disk
Graphically:
- Use M buffer to
read the
first M blocks from
input file R and
sort the
blocks:
- Re-use
the M buffer to
read the
2nd M blocks from
input file R and
sort the
blocks:
- Repeat until entire input file
has been processed:
Number of disk IOs performed in pass 1:
- We read the
input file
one time and
- We write
fragments that
total equal to
the input file
Number of disk IOs in pass 1:
Disk IOs for pass 1 = 2 B(R)
Constraint: (this constraint is imposed by Pass 2 of the TPMMS algorithm -- see next topic)
- The
number of chunks (K)
≤ M − 1
Graphically:
- Divide the
input file into
chunks of
M blocks each
- Pass 2:
- Divide the
M buffers into:
- M − 1 input buffers (use 1 buffer to read a sorted chunk)
- 1 output buffer (to write the total sorted output)
- Because
we only have
M − 1
input buffers
to read
the
K sorted chunks,
we have the
constraint that:
-
# sorted chunks K
≤ M − 1
(as stated above)
-
# sorted chunks K
≤ M − 1
(as stated above)
- How to
use 1 output buffer to
merge sort the
K sorted chunks
into
one sorted output file:
- Find the record with the
smallest sort key among the
K buffers
- Move the
record with the
smallest sort key
to the output buffer
- If the output buffer is full, then write (= empty) the output buffer to disk
- If some
input buffer is
empty:
- Read the
next (sorted !) block from the
sorted chunk
if
there is more data
(If there is no more data in the chunk, then ignore the chunk in the merge operation)
- Read the
next (sorted !) block from the
sorted chunk
if
there is more data
- Repeat until data in all chunks are processed
- Find the record with the
smallest sort key among the
K buffers
Graphically:
- Divide the
M buffers into:
- Suppose:
- Example:
- Sort the following
input file:
- Pass 1:
- Step 1:
sort
first chunk of
M blocks:
- Step 2:
sort
second chunk of
M blocks:
- And so on:
- Step 1:
sort
first chunk of
M blocks:
- Pass 2:
- Use
1 buffer to
read
each chunk
and use 1 buffer
for output:
- Read
each chunk
1 block at a time:
- Move the
smallest element to
the
output buffer:
And so on:
When output buffer is full, empty it for re-use:
And so on...
- Use
1 buffer to
read
each chunk
and use 1 buffer
for output:
- Sort the following
input file: