Conference Papers Year : 2022

An Elastic and Scalable Topic-Based Pub/Sub System Using Deep Reinforcement Learning

Thanos Giannakopoulos
  • Function : Author
  • PersonId : 1475951
Vana Kalogeraki
  • Function : Author
  • PersonId : 998192

Abstract

The ability to handle large volumes of event data and react to unexpected spikes, in real-time, remains an important challenge in stream processing systems, such as Apache Kafka, due to the amount of custom coding and technical expertise required to configure these systems. In this paper we investigate the use of reinforcement learning as a promising approach to address these issues. By feeding the machine learning technique with system performance metrics under a wide variety of configurations, we can effectively address any changes in the pub/sub system or overload situations while maintaining the desired performance goals. We implement our methodology on the Kafka pub/sub system without any changes in the application logic. Our experimental results illustrate the performance and benefits of our approach.
Fichier principal
Vignette du fichier
523617_1_En_11_Chapter.pdf (1) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04827161 , version 1 (09-12-2024)

Licence

Identifiers

Cite

Thanos Giannakopoulos, Vana Kalogeraki. An Elastic and Scalable Topic-Based Pub/Sub System Using Deep Reinforcement Learning. 17th IFIP International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2022, Lucca, Italy. pp.167-183, ⟨10.1007/978-3-031-16092-9_11⟩. ⟨hal-04827161⟩
14 View
0 Download

Altmetric

Share

More