CS584 - Stream Database

CS584 Sylabus & Progress

Material covered are displayed in blue.

Introduction.

Introduction and Overview to stream DB: click here

Relevant papers:

Continuous Queries over Data Streams. ACM SIGMOD Record archive Volume 30. click here
Models and Issues in Data Stream Management. SIGMOD Record, June 2003. click here
Stream DB Systems
- Aurora (Brown University)
  - Architecture: click here
  - Stream Query Algebra: click here
  - Aurora website: click here
  - Relevant paper: Abadi et al. Aurora: A New Model and Architecture for Data Stream Management. VLDB Journal, August 2003. click here
  - For your interest: M. Cherniack et. al., Scalable Distributed Stream Processing (Medusa: Distributed Aurora) CIDR 2003. click here
- STREAM (Stanford)
  - The Continuous Query Language (CQL): click here
  - CQL query plan and execution: click here
  - Official website: click here
  - Relevant paper: Arasu et. al. (Stanford group), STREAM: The Stanford Data Stream Management System click here
  Other stream database systems (not discussed in course)
- Telegraph (Berkeley): Official Website
  Krishnamurthy, et. al.: TelegraphCQ: An Architectural Status Report click here (This system uses an "Eddy")
  Chandrasekaran et al. (Berkeley group), TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. CIDR 2003. click here
- PSoup: a system for streaming queries over streaming data, Sirish Chandrasekaran, Michael J. Franklin. In VLDB Journal, August 2003. click here
  (The VLBD Conference version is "Streaming Queries over Streaming Data." Sirish Chandrasekaran, Michael J. Franklin. In Proc. International Conference on Very Large Data Bases (VLDB), HongKong, August 2002. click here )
Review of Relational Algebra
- CS377 material on Relational Algebra: click here
Memory Requirements in Stream Processing
- Characterizing Memory Requirements for Queries over Continuous Data Streams click here
- Relevant papers:
  - Characterizing Memory Requirements for Queries over Continuous Data Streams ACM SIGMOD Record 30(3): 109--120, Sept. 2001 click here
Sketching: building a summary of a data stream
- Random Sketching: click here
- Relevant papers:
  - "The space complexity of approximating the frequency moments", N. Alon, Y. Matias and M. Szegedy, ACM Symp. on Theory of Computing, pp 20-29, 1996 click here
Approximate answer to query using random sampling
- Introduction to unweighted sampling: click here
- Better unweighted Coin Flip sampling algorithms:
  - Concise Sample: click here
  - Counting Sample: click here
- Weighted Sampling for general join operation: click here
- Constructing representative sample for a general join operation click here
- Foreign key joins: click here
- Relevant papers:
  - Random Sampling with a Reservoir, J.S. Vitter, ACM Trans. on Math. Software, Vol 11, No. 1, 1985: pp 37-57 click here
  - On Random Sampling over Joins, S. Chaudhuri, R. Motwani and V. Narasayya, ACM SIGMOD 1999 click here
  - Join Synopes for Approximate Query Answering (Foreign key join), Acharya, Gibbons, Poosala and Ramaswamy click here
Approximation Techniques: Histograms
- Intro to Histograms: click here
- Intro to V-Optimal Histograms: click here
- Finding V-Optimal histogram (part 1) - Minimizing the error in one bucket click here
- Finding V-Optimal histogram (part 2) - searching for the best bucket partitions: click here
  - Relevant paper: Optimal Histograms with Quality Guarantees: click here
- A faster algorithm for finding approximate V-Optimal Histograms: click here
  - Relevant paper 1: "Data-Streams and Histograms": click here
  - Relevant paper 2: Approximation and Streaming Algorithms for Histogram Construction Problems: click here
- Recent Histogram research: (not discussed)
  - Fast, Small-Space Algorithms for Approximate Histogram Maintenance, Gilbert, et.al.: click here
  - Recent Paper 2, Guha: click here
  - Recent paper 3, Gibbons : click here
Mining "Heavy" (Frequent) Elements
- Introduction: click here
- Basic algorithms for finding heavy elements in database: click here
- Hybrid algorithms for finding heavy elements in database: click here
- Multi-Pass algorithms for finding heavy elements in database: click here
  - Relevant paper: Computing Iceberg Queries Efficiently, Fang, Shivakumar, Garcia-Molina, Motwani and Ullman click here
- Finding the majority elements: click here
- Finding heavy elements in data streams (Karp): click here
  - Relevant paper: A Simple Algorithm for Finding Frequent Elements in Streams and Bags, Richard Karp, Christos Papadimitriou and Scott Skenker click here
- ε-approximate frequency count (Manku): click here
  - Relevant paper 1: Approximate Frequency Counts over Data Streams, Manku & Motwani, click here
Mining Frequent Itemsets (and Association rules)
- Mining Association rules: click here
- Mining frequent itemsets in materialized databases (Apriori): click here **New**
- Generating association rules (Apriori): click here **New**
  - Paper: Click here
- Finding frequent itemsets in a data stream (Manku): click here
- A more efficient algorithm to mine heavy item sets in a stream: click here **New**
  - Paper: "An Algorithm for In-Core Frequent Itemset Mining on Streaming Data" - Jin & Agrawal click here
- Finding the maximum frequency and the maximum window of itemsets in a data streams without window size restriction - Part I (Calders): click here **New**
  Finding the maximum frequency and the maximum window of itemsets in a data streams with minimum window size restriction - Part II click here **New**
  - Paper: "Mining Frequent Itemsets in a Stream" - Toon Calders, Nele Dexters, and Bart Goethals Click here
Quantiles
- ε-approximate of quantiles in data streams: click here
- Efficiency issues in Greenwald and Khanna's Algorithm click here
  - Paper: Space Efficient Online Computation of Quantile Summaries, Greenwald & Khanna, click here
  - Power points of paper: click here
  - My written notes on the ε-approximate quantiles algorithm: click here (has been included in the course notes)
- Distributed computation of ε-approximate of quantiles on data streams: click here ***New***
  - Paper: Power-Conserving Computation of Order-Statistics over Sensor Networks Greenwald & Khanna, click here
- Fast algorithm for ε-approximate of quantiles - part 1: fixed size data click here ***New***
- Fast algorithm for ε-approximate of quantiles - part 2: data stream click here ***New***
  - Paper: "A fast algorithm for Approximate Quantiles in in High Speed Data streams, Zhang and Wang click here
Processing in a window
- ε-approximate quantile within a (large) window of a data stream (Greenwald & Ghanna): click here
  - Book Chapter : click here
- ε-approximate frequency count and quantile within a (large) window of a data stream (Arasu): click here
- Merge operation for frequency counts: click here
- Merge operation for quantiles: click here
  - paper: Approximate Counts and Quantiles over Sliding Windows, by A. Arasu and G. Manku, click here
  - Powerpoint slides for paper 1: click here
Clustering
- Clustering problem on materialized data: click here
- Clustering problem on data streams click here
  - Conference Paper: "Clustering Data Streams", Guha et. al. (note: Theorem 2.2 in this paper is so ambiguous that it is completely uncomprehensible) click here
  - Longer Paper: "Clustering Data Streams, Theory and Practice", Guha et. al. (note: very poor notations, barely readable) click here
Approximation Techniques: Wavelets
- Gilbert, Kotidis, Muthukrishnan & Strauss. Surfing Wavelets on Streams: One-pass Summaries for Approximate Aggregate Queries. VLDB 2001. click here
Approximation Techniques: Moving Window
- Babcock, Datar & Motwani. Sampling from a Moving Window Over Streaming Data (extended abstract). Proc. Symposium on Discrete Algorithms (SODA), 2002. - click here
  Powerpoint slides: click here
Query Processing using Moving Windows
- Kang, Naughton and Viglas. Evaluating Window Joins over Unbounded Streams, ICDE 2003. click here
- Golab & Ozsu. Processing Sliding Window Multi-Joins in Continuous Queries over Data Streams. Technical report, February 2003. click here
Approximation Techniques: Join Operation
- Das, Gehrke & Riedewald. Approximate Join Processing Over Data Streams. SIGMOD 2003. click here
Load Shedding
- Babcock, Datar and Motwani. Load Shedding Techniques for Data Stream System, 2003 Workshop on Management and Processing of Data Streams. click here