1. Home
  2. NOVA School of Science and Technology
  3. Master in Computer Science and Engineering
  4. Stream Processing

Stream Processing

Objectives

Learn the fundamental concepts, languages, and systems for building applications that process data streams. This course discusses, presents and discusses generalist systems for real-time stream processing, and will focus on the study of systems for structured data flow-oriented models.

To knowledge

A. Know the main programming models for streaming data processing

B. Know the languages and assimilate the fundamental characteristics to solve problems in the stream processing domain.

C. Understand the advantages and disadvantages of stream processing platforms.

Application

A. Being able to choose the most appropriate models, languages and tools to solve a stream processing problem.

B. Set capable of developing and executing stream processing applications using current tools and
technologies.

General characterization

Code

11562

Credits

6.0

Responsible teacher

Nuno Manuel Ribeiro Preguiça, Sérgio Marco Duarte

Hours

Weekly - 4

Total - 50

Teaching language

Português

Prerequisites

Adequate programming skills in the Python programming language.

Bibliography

Opher Etzion and Peter Niblett. Event Processing in Action. Manning Publications, 2010.

Lukasz Golab and Tamer Özsu. Data Stream Management. Morgan and Claypool, 2010.

Bifet et al., (2018) Machine Learning for Data Streams, MIT Press

Several papers will be provided for further reading.

Teaching method

Lectures present the dominant models, languages and platforms for each thematic area of course syllabus, using paradigmatic examples to frame the discussion and the understanding of the issues.

Labs promote hands-on experience on the covered systems and languages, including application development for particular data streaming scenarios. Classes comprise demos, exercises and support for two programming projects to be developed during the semester. Examples of such application are the challenges of the main conference in this area (DEBS).

Grading comprises two quizzes (25% each) and two project assignments (25% each).

Evaluation method

Grading comprises two quizzes (25% each) and two project assignments (25% each).

Subject matter

Distributed Stream Processing Systems.

System models for stream processing: streams as sequences of mini-batches (e.g. Spark streaming); continuous processing (e.g. Apache Flink, Storm).
Programming models. System aspects: distribution, scalability and fault-tolerance.
Distributed time-series databases. Systems for IoT stream processing.

Data Stream Management Systems (DSMS).

Structured Data Models for Streams. Algebraic operators on stream and relations.
Continuous query languages (extensions to SQL and database management systems to deal with data streams).

Machine Learning for Streams.

Introduction to learning from data Dimensionality reduction for streams.
Learning under concept drift. Incremental learning.
Learning under imbalance and learning from graphs.