Meet the Experts: Data-driven Day with Tim Berglund


Dieses Event teilen

Datum und Uhrzeit



codecentric AG

Hochstraße 11

42697 Solingen


Karte anzeigen

Beschreibung des Events


(English version below)

Verarbeitung und Austausch von Daten stehen im Mittelpunkt eines jeden Systems. Beispielsweise müssen bei einer Microservice-Architektur die Services Daten miteinander austauschen. Um eine möglichst lose Kopplung zwischen den Services zu erreichen, wird zur Kommunikation häufig ein Event-getriebener Ansatz verfolgt.

Unabhängig davon gewinnt auch die Verarbeitung von Streaming-Daten durch die stetig wachsende Menge an durchgehend produzierten Daten zunehmend an Bedeutung. Diese Daten treten beispielsweise beim Internet of Things, bei Klickstreams oder im Werbegeschäft auf. Diese Informationen sollen dann zeitnah zur Verfügung gestellt und ausgewertet werden.

Beiden Feldern ist gemein, dass Daten hier durchgehend in Bewegung sind. Sie werden verteilt und von verschiedenen Systemen konsumiert und analysiert. Dabei geht es nicht nur um Geschwindigkeit, sondern auch um Veränderungen im Aufbau der Daten.

Meet The Experts: Data-driven Day bietet Ihnen einen Überblick über die Herausforderungen und möglichen Lösungsansätze und Technologien für datengetriebene Anwendungen und Use Cases.

Der Tag richtet sich an Entwickler und Architekten, die datengetriebene Anwendungen entwickeln, die Datenströme optimieren wollen und schneller Erkenntnisse aus Daten gewinnen müssen.


Data processing and exchange are at the core of every system, particularly in microservices architectures. In order to ensure a loose coupling between the services, event-driven approaches are often implemented for communication.

Apart from this, stream data processing is becoming increasingly important due to the steadily increasing volume of data produced, e.g. by the Internet of Things, clickstreams or in the advertising business. The information gained from these data then needs to be made available and analyzed promptly.

What both fields have in common is that data is constantly in motion. It is distributed and then consumed as well as analyzed by different systems. This process is not just about speed, but also about changes in data structure.

Meet The Experts: Data-driven Day provides an overview of the challenges, possible solutions and technologies for data-driven applications and use cases.


9:00 Uhr: Einlass & Kaffee

09:30 Uhr: Begrüßung (Lars Rückemann, Matthias Niehoff, codecentric)

09:45-10:45 Uhr: The Database Unbundled: Commit Logs in an Age of Microservices (Tim Berglund, Confluent)

— Kaffeepause —

11:00-12:00 Uhr: Data Stream Processing: Concepts and Implementations (Matthias Niehoff, codecentric)

– Pizza –

13:15-14:30 Uhr: Streaming Data with Apache Kafka (Tim Berglund, Confluent)

14:45-15:30 Uhr: Transparent End-to-End Security for Apache Kafka (Hendrik Saly, codecentric)

— Kaffeepause —

16:00-16:45 Uhr: Processing Streaming Data with KSQL (Tim Berglund, Confluent)

16:45 Uhr - Schluss: Snacks, Austausch & Networking

Hinweis: Alle Vorträge werden in englische Sprache gehalten.


The Database Unbundled: Commit Logs in an Age of Microservices
When you examine the write path of nearly any mutable data store, the first thing you find is a commit log: mutations enter the database, and they are stored as immutable events in a queue, only some hundreds of microseconds later to be organized into the various views that the data model demands. Those views can be quite handy–graphs, documents, triples, tables—but they are always derived interpretations of a stream of changes. Zoom out to systems in the modern enterprise, and you find a suite of microservices, often built on top of a relational database, each reading from some centralized schema, only some thousands of microseconds later to be organized into various views that the application data model demands. Those views can be quite handy, but they are always derived interpretations of a centralized database.

Wait a minute. It seems like we are repeating ourselves.

Microservice architectures provide a robust challenge to the traditional centralized database we have come to understand. In this talk, we’ll explore the notion of unbundling that database, and putting a distributed commit log at the center of our information architecture. As events impinge on our system, we store them in a durable, immutable log (happily provided by Apache Kafka), allowing each microservice to create a derived view of the data according to the needs of its clients. Event-based integration avoids the now-well-known problems of REST and database-based service integration, and allow the information architecture of the future to take advantage of the growing functionality of stream processing systems like Kafka, allowing us to create systems that can more easily adapt to the changing needs of the enterprise and provide the real-time results we are increasingly being asked to provide.

Data Stream Processing Concepts and Implementations

When it comes to stream processing there are plenty of options. In this talk I will give an overview on various concepts used in data stream processing. Most of them are used for solving problems in the field of time, focussing on processing time compared to event time. The techniques shown include the Dataflow API as it was introduced by Google and the concepts of stream and table duality. But I will also come up with other problems like data lookup and deployment of streaming applications and various strategies on solving these problems.

In the end I will give a brief outline on the implementation status of those strategies in the popular streaming frameworks Apache Spark Streaming, Apache Flink and Kafka Streams.

Streaming Data with Apache Kafka
When it comes time to choose a distributed messaging system, everyone knows the answer: Apache Kafka. But how about when you’re on the hook to choose a world-class, horizontally scalable stream data processing system? When you need not just publish and subscribe messaging, but also long-term storage, a flexible integration framework, and a means of deploying real-time stream processing applications at scale without having to integrate a number of different pieces of infrastructure yourself? The answer is still Apache Kafka.

In this talk, we’ll make a rapid-fire review of the breadth of Kafka as a streaming data platform. We’ll look at its internal architecture, including how it partitions messaging workloads in a fault-tolerant way. We’ll learn how it provides message durability. We’ll look at its approach to pub/sub messaging. We’ll even take a peek at how Kafka Connect provides code-free, scalable, fault-tolerant integration, and how the Streams API provides a complete framework for computation over all the streaming data in your cluster.

Transparent End-to-End Security for Apache Kafka
Apache Kafka comes with a lot of security features out of the box (at least since version 0.9). But one feature is missing if you deal with sensitive mission critical data: Encryption of the data itself. This talk will cover how to achieve transparent end-to-end encryption in Kafka and how this impact performance.
end-to-end security (including security "at rest") is a major requirement for enterprises, especially in the days where all data is stored in the cloud. For more details check out this blog post.

Processing Streaming Data with KSQL
Apache Kafka is a de facto standard streaming data processing platform, being widely deployed as a messaging system, and having a robust data integration framework (Kafka Connect) and stream processing API (Kafka Streams) to meet the needs that common attend real-time message processing. But there’s more!

Kafka now offers KSQL, a declarative, SQL-like stream processing language that lets you define powerful stream-processing applications easily. What once took some moderately sophisticated Java code can now be done at the command line with a familiar and eminently approachable syntax. Come to this talk for an overview of KSQL with live coding on live streaming data.

Mit Freunden teilen

Datum und Uhrzeit


codecentric AG

Hochstraße 11

42697 Solingen


Karte anzeigen

Dieses Event speichern

Event gespeichert