Click here to Skip to main content
15,867,939 members
Articles / Programming Languages / C++
Tip/Trick

A Persistent Circular Queue Buffer

Rate me:
Please Sign up or sign in to vote.
2.30/5 (6 votes)
10 Aug 2022CPOL1 min read 4.5K   48   2   15
Using the filesystem as a repository for a circular buffer for achieving persistence of data
Sample code for a persistent circular queue buffer

Introduction

The inspiration for this piece of code was the following article which talks about a persistent message broker:
https://www.oreilly.com/library/view/zeromq/9781449334437/ch04s12.html

The article had the following improvement suggestions:

  • Use a single disk file for all data, rather than multiple files. Operating systems are usually better at handling a few large files than many smaller ones.

  • Organize that disk file as a circular buffer so that new requests can be written contiguously (with very occasional wraparound). One thread, writing full speed to a disk file, can work rapidly.

Hence came up with the first step of implementing the suggestions. Currently we are able to write data in circular fashion in the disk and also read from it consistently anytime even when the program restarts.

The original code was taken from this link but it didn't implement a circular queue properly: 

https://stackoverflow.com/questions/33793535/how-to-be-sure-data-is-on-disk-after-fwrite

After modifying the codes a little I was able to get it working.

Using the code

Using the code is pretty simple.

First push some items in the queue after specifying a filename which stores the data.

C++
//  Templated declaration  
    FixedSizeQueue<int> fq;
//  Specify the filename
    fq.open("sample_file.dat", 6);

//  Push some data

    int a1[] = {0, 1, 2};
    int a2[] = {3, 4};
    fq.push_values(a1, 3);
    fq.push_values(a2, 2);

Then fetch the saved data with a get_values calls

C++
//  fetch data

int values[100];
size_t numberofvalues = fq.get_values(values);

Do remember to include the header file : circularqueue.h

Points of Interest 

Using a different file to store the head, tail and size info so that next time the program restarts, the head, tail and size can be restored properly and the queue acts as a persistent circular queue.

History

Code submitted: 11th August, 2022

Code updated: 20th August, 2022

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Technical Lead Kotha Technologies
Bangladesh Bangladesh
If you are not in - you are out !
- Chapter 1

Comments and Discussions

 
AnswerNot an article Pin
Michael Haephrati18-Aug-22 3:28
professionalMichael Haephrati18-Aug-22 3:28 
GeneralRe: Not an article Pin
Mukit, Ataul18-Aug-22 23:10
Mukit, Ataul18-Aug-22 23:10 
AnswerRe: Not an article Pin
Michael Haephrati19-Aug-22 0:14
professionalMichael Haephrati19-Aug-22 0:14 
GeneralRe: Not an article Pin
Mukit, Ataul19-Aug-22 9:06
Mukit, Ataul19-Aug-22 9:06 
GeneralI Clicked This Link Because I Didn't Understand the Project Pin
honey the codewitch10-Aug-22 23:53
mvahoney the codewitch10-Aug-22 23:53 
GeneralRe: I Clicked This Link Because I Didn't Understand the Project Pin
Mukit, Ataul11-Aug-22 1:53
Mukit, Ataul11-Aug-22 1:53 
The inspiration for the circular queue implementation came from this article
which talks about a persistent message broker system using zeromq:
Disconnected Reliability (Titanic Pattern) - ZeroMQ [Book]

The message saving wasn't done efficiently and at the end of the article improvement suggestions were given which included implementing a circular buffer. Main suggestions were:
a) Use a single disk file for all data, rather than multiple files. Operating systems are usually better at handling a few large files than many smaller ones.
b) Organize that disk file as a circular buffer so that new requests can be written contiguously (with very occasional wraparound). One thread, writing full speed to a disk file, can work rapidly.
GeneralRe: I Clicked This Link Because I Didn't Understand the Project Pin
Randor 12-Aug-22 1:17
professional Randor 12-Aug-22 1:17 
GeneralRe: I Clicked This Link Because I Didn't Understand the Project Pin
Mircea Neacsu12-Aug-22 5:20
Mircea Neacsu12-Aug-22 5:20 
GeneralRe: I Clicked This Link Because I Didn't Understand the Project Pin
honey the codewitch12-Aug-22 14:09
mvahoney the codewitch12-Aug-22 14:09 
GeneralRe: I Clicked This Link Because I Didn't Understand the Project Pin
Mircea Neacsu12-Aug-22 15:03
Mircea Neacsu12-Aug-22 15:03 
GeneralRe: I Clicked This Link Because I Didn't Understand the Project Pin
Mukit, Ataul15-Aug-22 8:43
Mukit, Ataul15-Aug-22 8:43 
GeneralMy vote of 1 Pin
Rick York10-Aug-22 19:00
mveRick York10-Aug-22 19:00 
GeneralRe: My vote of 1 Pin
Mukit, Ataul10-Aug-22 21:06
Mukit, Ataul10-Aug-22 21:06 
GeneralRe: My vote of 1 Pin
Mukit, Ataul15-Aug-22 8:45
Mukit, Ataul15-Aug-22 8:45 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.