---

Introduction/Overview of Stream Database processing

---

• Characteristics of Traditional Database Systems

  data is contained within some form of persistent data sets data content are relatively permanent user queries can vary over a wide range of topics (relatively unstable)

  ---

  ---

• Stream Databases
  • is a recent phenomenon....

    applications generate a stream of information

  • Examples:

    stock tickers network traffic monitoring patient (vital signs) monitoring ...

    ---

  • Chracteristics of "Stream Databases":

    data managed is not contained within any form of persistent data set.. The data is constantly in flux - i.e., the data is transient... data content changes constantly user queries are relatively fixed In other words: a monitor query tracks a small number of well-defined events (e.g., in stock tickers: tracks the trend of the price so that users can obtain "buy and sell signals")

  ---

  ---

• Question: Can traditional Database systems be used to manage Stream Data ?
  • Answer: No

  • Traditional system are ill-equipped to handle stream data because:

    The stream data would first need to be stored in tables... And then the DBMS would process queries offline.... and mean while new data are coming in...

    ---

    ---

  • Example that illustrates the inefficiency of traditional DBMS for processing stream queries
    • An ISP (Internet Service Provider) monitors the load on the following 2 links (among many links):

      • The first link is a customer link
      • The second is a backbone link
      • The routers connected to these 2 links continuously send packet summaries to the query processor

      ---

    • Query:

      Compute the load on the backbone link average over 1 min period; Notify operator if average load exceeds a treshhold T

      SQL solution on a materialized database:

      SELECT notifyOperator( sum(length) ) FROM PT_backbone GROUP BY getMinute(timestamp) // per minute HAVING sum(length) > T

      ---

    • How to run the SQL query:

      Use database trigger (CS377 material) !

    • A database trigger "fires" (is run) when a new tuple is inserted (deleted or modified - in our case, insertion is the only reason for firing).

      ---

    • Result: terrible system performance:

      Network packets arrives rapidly In optical links, billions of packets can arrive in 1 second Traditional DBMS cannot handle billions of insertions and triggers firings in one second....

  ---

  ---

• A better processing model for stream data
  • The type of query processing model used in stream DB is called Continuous Query

  • A continuous query is issued once and...

    it is run continuously over the "database" (input data stream) - until it is stopped explicitly

  • Example:

    • One or more streams of input data.
    • Input data temporally buffered in query processor (computer)
    • Continuous query (program) operates on data in buffers

  ---

  ---

• Why is stream database different from traditional database ?
  • The way that a query on stream data is performed is fundamentally different from how a query is performed on traditional databases.

  • The following examples will help illustrate the differences

  ---

  ---

• Issue 1: accuracy of stream queries
  • Some times, we need to wait an indefinite time to obtain the exact outcome in stream query

    ---

  • Consider the network monitoring problem:

    Query:

    Find the fraction of the traffic on the backbone link that originate from the customer link.

    NOTE: unit is "packet" (not number of bytes in packet)

    SQL-like solution:

    (SELECT count(*) FROM PT_backbone AS B, PT_customer as C WHERE C.pkt_id = B.pkt_id) / (SELECT count(*) FROM PT_backbone)

    ---

    ---

  • Problem that arrises when query is processed:

    The query joins packets from streams PT_backbone and PT_customer on their keys (pkt_id) to identify that the source is Customer link However, to perform the join, you need both packets in the monitor: Problem: delay is unpredictable... We may need to buffer a very large amount of data to provide the exact answer to a stream query... $64,000 questions: How much do you want to pay for the exact answer ? Can you live with an answer that is not exact ?

    Conclusion:

    to obtain the exact solution, you may need an unbounded amount of buffer space (to store packets)...

  ---

  ---

• Issue 2: some queries are "blocking"
  • Some stream queries are blocking, i.e., you need to stop accepting new input and rewind the data (and perform a second pass through the existing data) to produce an exact solution

    ---

  • Consider the follwoing network monitoring problem:

    Query:

    Find the connection (Src, Dest) that sends the highest amount of data (number of packets) on the backbone link

    Here we will use the number of bytes as unit (no big deal...)

    SQL-like solution:

    SELECT src_addr, dest_addr, sum(length) FROM PT_backbone GROUP BY src_addr, dest_addr HAVING sum(length) = (SELECT max(sum(length)) FROM PT_backbone GROUP BY src_addr, dest_addr)

  • Problem that arises when this query is processed (in real time):

    In the above query, we must first find the max(sum(length)) value To find max(sum(length)), we must make one pass through the collected data and compute the total per (src,dest): Only after you have found the value max(sum(length)), you can find the flow ID using another pass through the computed totals The problem is... a stream query is processed in real time... You may not have to time to do all the operations ("drinking from a fire hose") Also, so stream queries may require multiple passes through the streaming data "Rewind" a flow is not easy to do... (Because ideally, a stream database system must avoid storing (too much) data)

  ---

  ---

  ---

• Join Queries in Continuous Query processing
  • Consider a relation R(X,Y) with 2 attributes

    X is the key of the relation

  • Consider the following simple Continuous Query that involves a join operation:

    SELECT * FROM R R1, R R2 // Self-join WHERE R1.X = R2.X

    ---

  • Interesting fact:

    The accuracy of the self-join query depends on the amount of tuples buffered by the stream query processor !!!!

    ---

  • Illustrative example:

    Processing of the continuous query is done as follows:

    ---

  • Consider the sample input stream:

    (A,B), (A,C), (A,D), ...

    If the continuous query processor buffers one tuple, the output will be: empty !!!

    If the continuous query processor buffers two tuples, the output will be: (A,B,A,C), (A,C,A,D), ... !!!

    If the continuous query processor buffers two tuple, the output will be: (A,B,A,C), (A,B,A,D), (A,C,A,D), ... !!!

  ---

  ---

• Research Problems in Stream DB
  • Basic requirements of stream DB processing:

    Query processing must be performed online - in other words: input data (tuples) comes in continuously and the data must be processed immediately - the output must reflect the input as soon and as close as possible The amount of storage space required to compute the answer of a continuous query must be bounded

    ---

    ---

  • Topics in Stream Database:

    System building Design and implement efficient stream database query processing systems Summarization. Summaries (or data synopses) is a concise (but not completely accurate) representation of the original data set at the expense of some accuracy. Summarization techniques include: sampling, histograms, and wavelets Summarization techniques are important in stream DB processing due to storage constraints (summarizes a large amount of data using small memory space). Summary information can also speed up query processing ! Online data structures. Online data structures are special data structures designed specifically to handle data streams. Develop efficient and effective data structures for stream database Query adaptation and optimization. The order of evaluation of the boolean conditions in a query will affect the speed of the query (e.g., performing a selection that removes a large number of tuples early will cut down the number of tuples early and improves processing speed). Continuous queries run over a long period of time... Input data characteristics can change over such a long period of time (e.g., a selection that filters out a large fraction of tuples hours ago may filter out a little fraction now due to change in input data...) So the processing steps of contiuous queries may need to be "shuffled" continuously... How do you design a query processing engine that allow "re-shuffling" of query steps. Query Language for expressing Continuous Queries Since most people are familiar with SQL, it is desirable to use SQL-like languages to express Continuous Queries (I'm not a programming language expert and this area is not the most interesting aspect in DB research for me... so I may be brief or omit it entirely :-)) Blocking operators A blocking operator is one that must obtain its entire input set before it can compute the output Typical blocking scenarios are: (1) queries that involves sorting and (2) queries that involves aggregation Note: a query that find the largest or the smallest value indirectly uses sorted and it is therefore blocking.... Obviously, input to the continuous query will take forever to finish arriving, so a blocking operator can't keep on blocking forever... :-) The semantic of such operator must be "re-defined" for a continuous query: what is the result of an aggregation or sort when more data may arrive later (that will surely alter the outcome).