CONFERENCE PROCEEDINGS
Advanced Search
Home > Proceedings > Volume 12970 > Article
Paper
21 December 2023 A two-phase submission algorithm using dynamic programming for Kafka consumer committing offset
Ruiguang Lv, Gang Wu, Zhi Li, Runlin Li, Mimi Zhang
Author Affiliations +
Proceedings Volume 12970, Fourth International Conference on Signal Processing and Computer Science (SPCS 2023); 129703U (2023) https://doi.org/10.1117/12.3012099
Event: Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), 2023, Guilin, China
Abstract
As an important middleware of distributed computing, message queue enables us to address issues like decoupling applications, asynchronous communication, fluctuating data traffic, and others. Kafka is a popular open-source message queue for collecting, processing, storing, and analyzing messages at scale, which is famous for its excellent performance, low latency, fault tolerance, and high throughput. In some scenarios like electronic trading, reliable consumption is the utmost concern. However, the native consumer client provided by Kafka cannot guarantee that messages are consumed successfully at least once. This paper proposes a two-phase submission algorithm using dynamic programming which can improve reliability of message consuming. This algorithm utilizes a two-phase submission to ensure that every message is consumed successfully at least once, and uses dynamic programming to improve submission efficiency. We conduct experiments to demonstrate that this algorithm is more efficient in contrast with the original submission strategy. In experiments, this algorithm must make sure that every message is consumed successfully at least once while various consuming failure rates are mocked.
(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.
Citation Download Citation
Ruiguang Lv, Gang Wu, Zhi Li, Runlin Li, and Mimi Zhang "A two-phase submission algorithm using dynamic programming for Kafka consumer committing offset", Proc. SPIE 12970, Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), 129703U (21 December 2023); https://doi.org/10.1117/12.3012099
ACCESS THE FULL ARTICLE
ORGANIZATIONAL
Sign in with credentials provided by your organization.
INSTITUTIONAL
Select your institution to access the SPIE Digital Library.
SELECT YOUR INSTITUTION
PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.
PERSONAL SIGN IN
No SPIE Account? Create one
PURCHASE THIS CONTENT
SUBSCRIBE TO DIGITAL LIBRARY
50 downloads per 1-year subscription
Members: $195
Non-members: $335 ADD TO CART
25 downloads per 1 - year subscription
Members: $145
Non-members: $250 ADD TO CART
PURCHASE SINGLE ARTICLE
Includes PDF, HTML & Video, when available
Members: $17.00
Non-members: $21.00 ADD TO CART
PROCEEDINGS
 8 PAGES
DOWNLOAD PAPER SAVE TO MY LIBRARY
GET CITATION
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Computer programming
Distributed computing
Data communications
Java
Reliability
Tolerancing
RELATED CONTENT
Subscribe to Digital Library
Receive Erratum Email Alert
Back to Top