What tech/architecture is recommended to handle a high amount of real-time data From Server to the clients?
Please note - Gathering data from Client to server is not difficult. We have ingestion points to collect the data.
But for sending data from Server to Client. Looks like it needs some good study and evaluation.
The cave-man idea that immediately comes to me is using Sockets and keep streaming data back.
Is there any recent frameworks/services available that does this best?
Google Cloud/Firebase has the options, we do use. But anything better out there?
Think of it as even a Share Market Application. Where you have to update all the connected clients every few seconds with Real-Time data This is not domain I'm into but the requirement is technically same. It's all about sending data in real-time.
I don't know which frameworks are suitable for this. If you have to start from scratch, try TCP. If it doesn't scale to meet your needs, look at RTP[^]. If message loss or reordering is acceptable, you could even go with basic UDP. It shouldn't be hard to change the protocol if your software is well structured.
You might find my article Software Techniques for Lemmings (link in .sig below) useful. It describes things that should generally be avoided in serious servers.
What about a multicast scenario? Would that work? Do all clients want the same data? I honestly don't know anything about multicast, other than there's such a thing. It might work for you, but you'd have to look at routing issues and maybe a way to resend when a client misses some data or goes offline.
Various protocols support multicasting. If you start investigating RTP, you will find related protocols for multicasting (streaming) video. They usually allow dropped frames, so it's a question of whether it's OK for clients to not have all of the server's information.
Off the top of my head, I would implement this with three threads:
The first thread writes the data that will be streamed to clients into a large circular buffer.
The second thread services the clients by reading from the buffer to TCP-send each client its next batch of data.
If clients acknowledge reception so as not to lose data, the third thread uses TCP-poll and TCP-recv to handle acknowledgments on the client sockets.
The buffer's start pointer advances over data after it has been streamed to, or acknowledged by, all clients.
Edit: You might want to look at the Robust Services Core (see link in .sig), an open-source framework that I developed. It could probably be used to develop a server that supports this kind of application. What's in its nb and nw directories would probably be adequate.
Cool. I'm also thinking of a hybrid idea. I've done this before.
The hint that the new data is available happens through "Push".
But the decision to fetch or not & the actual data-pull happens separately.
The notification mechanism will not be required if it's a consistently high-speed requirement like Sharemarket.
If the solution involves stopping data production intermittently, then a Push-notification mechanism to trigger start/stop could be useful.
The hint that the new data is available happens through "Push".
I'm happily querying a webserver for a file by requesting its header along with the last datetime of my file. If my file is from the same date, it will only send a header back with a 403 (Not modified) (from cache).
If the clients need to be useable during the download, I'd go for the BITS-service that Windows uses to download its own updates. If the fetching of the data is more important than the clients responsibility, I'd search CodeProject for a download-manager and open 10 connections to the server from the client and have each download 1/10th of the file.
If speed is paramount then I'd recommend QNX, not Windows.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
I have an app that has 3 sets of features. The Agent, Dispatcher, and Admin.
Each's ones use the same Domain Models but the context of use is different.
And now, the Admin gets the Ticket Object that has data in it used by the Agent and the Dispatcher.
The app has a backend in Java and frontend in Angular. The communicate overt HTTP + JSON.
Should I separate them by domain, and create 3 separate microservices?
Would that be an overkill, as the app is not big?
Or should I keep the app as a monolith, and just reorganize the code in packages Admin Package, Dispatcher Package, etc... ?
I remember this from when I was a student, I'm sure my lecturer called it something like "state shadowing" (permutations of that name covered by the "shadow state" and "variable shadowing") - basically what I'm remembering is related to hardware programming where there's a time cost associated with querying the state of a connected hardware peripheral, so when setting the hardware state you also set a variable in-memory - that way whenever you need to know the hardware state you can just reference the variable.
Does anyone know what this kind of pattern would be called?
I'm looking for some inspiration on how to design something, so I'm just looking for any kind of related literature, but it's a bit hard to find anything when you don't know the name.
I would call it a cached result, but maybe some other term is in common usage among embedded system developers who work close to the hardware. Objects sometimes do the same thing to avoid calculating a result that clients frequently use.
We are starting a new project and are discussing about which architecture to use in our project.
We want to decide between Monolithic and Microservices architecture.
Just to give a overview of our project,
We have four functional modules, each with a specific functionality.
Each of these module is part of a process flow of the project.
After completion of each stage of the flow, that data will be used by the next module for further processing.
This will be a web application which can be used by anyone by registering.
Number of hits to this application may not be as much as with other applications, as it will be used only by people with specific expertise.
May be by around one million people.
Please suggest your thoughts on how to go about this project.
Last Visit: 31-Dec-99 18:00 Last Update: 22-Sep-23 15:35