|
There IS going to be a dialog for Customers and their options. I didn't mean they will use an editor to change settings. But that doesn't solve the issue of more than one person modifying it at the same time.
Since this is just a serialized file (XML) that contains the app's info (List of customers and their processing options), and I likely can only store that file in SharePoint, then I'm stuck with how to keep users from overwriting each other's changes.
Scenerio: User A opens the app, which reads the app's data from SharePoint as a serialized collection. The user decides to modify a customer's settings and clicks save, which writes it back to SharePoint. User B then changes the same. User A's customer settings are now out of sync with user B.
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
Kevin Marois wrote: The user decides to modify a customer's settings and clicks save, which writes it back to SharePoint. User B then changes the same. User A's customer settings are now out of sync with user B.
As described that is a invalid business scenario.
Businesses do not have two people working on changing a customer at the same time. A sales person might update an address and an order entry person might add an order but those are different data operations even if for the same customer. But there will never be two people attempting to change the customer address at the same time.
And if you are doing actual order entry then good luck because you are going to need to implement a full database yourself before your application can even work. Hopefully you are getting paid by the hour.
|
|
|
|
|
All I can say is that your design is wrong, and needs a rethink. If user A changes the file and rewrites it, then whatever user B reads in may be invalid. The only way I could see this as a possibility is if the update file is rewritten to each user's local disk. But again that will present its own problems.
|
|
|
|
|
You're going against the customers' architectural strategy. What you have "stuffed" into XML is the equivalent of a Sharepoint "list" or 2, that can be secured and shared twenty ways to Sunday, and accessed from WPF (if need be) via CSOM (SP client side object model)
"Before entering on an understanding, I have meditated for a long time, and have foreseen what might happen. It is not genius which reveals to me suddenly, secretly, what I have to say or to do in a circumstance unexpected by other people; it is reflection, it is meditation." - Napoleon I
|
|
|
|
|
XML is NOT a database and you're trying to treat it like one. This will go badly for you. Period.
There is no multi-user solution for an XML file, or any file for that matter.
|
|
|
|
|
Kevin Marois wrote: My client doesn't have a network.
As stated that is nonsensical with what you said in the rest of your post such as "then another user runs the app, makes a change, and saves BEFORE the first user is done". That second cannot happen unless there is a network.
Also Sharepoint is a teamwork collaboration tool. You can't use it without a network.
Perhaps you meant a shared file system.
But other than that the configuration file is either per user or there is only one for the entire company.
If per user then you store it on each computer. There is no problem.
If one per company then then since you know Sharepoint exists then use it. It has a REST API, and I would be really surprised if there was not a way to store a file. Your app can provide the management including a place for the current user to type in their Sharepoint credentials (or use some other method to authenticate.)
As for the overwrite problem...
There is something wrong with your design if you expect users to be constantly updating the configuration file all of the time. It should be something that is rarely updated and when it is updated one specific person will be tasked with doing that. It will likely be the case that only one person even knows how to do it. So it is a non-issue.
Otherwise it isn't a configuration file. It is in fact a persisted data store. And your design should have accounted for that in the first place.
|
|
|
|
|
When I say config file here, I mean a serialized copy of the app's data. Since there's no database, then I need to store the app's data somehow, and a serialized file seems like the way to go.
But all users need access to that file, as it has to load on startup. And, each user could alter that data and save it.
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
That description does not sound like a configuration file. Everything suggests you want a database but have decided you are not going to use one.
Note that all of this requires a shared data location otherwise it is pointless to even discuss this.
If you have customer data then you need to store it in one location.
If you have user data then you store it in an different location.
If there is shared data then you will need to hack a solution that is in fact a database.
Database servers handle the concurrency issue by being in process and by using locks either at the table level or the row level.
You can implement something like that by using data. The data is a marker that each app must look for before it attempts to write. If it is there then the user must refresh before they can update. Problem with that is if the user computer exits or the network goes down the lock file is still there so you will need to provide a way to force it. The granularity and layout is something you would need to design and implement.
|
|
|
|
|
Why not make the application acquire an exclusive lock on the file whenever it's being edited?
That will definitely prevent corrupted changes.
EDIT: On second thought, even this wouldn't work because a person might still write an older version of the file when committing changes.
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Richard Andrew x64 wrote: On second thought, even this wouldn't work because a person might still write an older version of the file when committing changes.
So, either lock the file, read what you need to make a decision, and write it back, in one go.
Or, if you require some time to ponder the changes: Read lock, take note of the last modification time, read, release. Do all your pondering, preparing your changes. When changes are ready to be written: Exclusive lock, check time of last modification. If later than the one you saw the first time, read the new data, release and repeat from the pondering step. If write timestamp is unchanged, write your new data and release.
This is a file system version of database 'optimistic locking'. With database transactions, you should always be prepared for a rollback if some other transaction modifies data you have based your calculations on. I am not saying that all database applications are prepared for rollback, only that they should be  . Databases with optimistic locking favors short transactions: The shorter the time span from data are read to the commit, the less risk for someone else making modifications inbetween. Your DIY optimistic file system locking is similar: Do all the preparations that you can do without reading the file, leaving the minimum work between reading and writing, to reduce the risk of a rollback. But you must be prepared for it, forcing you to read the updated data before attempting a redo transaction.
|
|
|
|
|
Scenario:
We have (as of today, will be more) 200 devices sending telemetry into an SQL server
with a frequency of 5 rows each pr. second.
Each row contains 6 decimal values to be processed into another dataset/table
containing min/max/avg of these values in 1 minute intervals.
I have never worked with DataLakes and similar tech, so I wonder:
Should i read up on storing the raw telemetry in a datalake, and set up post processing there,
or Just go for my existing SQL server and create a c# job post processing the data in the background myself?
TIA...
|
|
|
|
|
There are so many "it depends" that need to be looked at in here. We'll start with a simple question that should, hopefully, start us towards the answer.
Question: Do you need to do anything else with the data that you are inserting? Once you have created these six values, is the telemetry data ever used again?
|
|
|
|
|
The telemetry data are also needed later, yes.
|
|
|
|
|
Where are you hosting the data? Is it in a cloud provider such as Azure or AWS or are you managing it yourself?
|
|
|
|
|
The DB, and the code that receives tha data it hosted in Azure.
Data is recieved as MQTT messages by Azure IoTHub.
A C# worker subsribes to the incoming messages, and stores them normalized in an Azure SQLdb.
|
|
|
|
|
So, you have data coming in from 200 devices, 5 times a second. Let's think about the problem areas here:
- When you add data, at that volume, you're adding 1000 items a second. Not a problem in SQL Server.
- I'm assuming it's one aggregation per device every minute, so you're looking at a read of 60K rows every minute, with a write of 200 rows.
I would be tempted to invert the problem slightly. Rather than write to one source, write to two. So, write the incoming messages to SQL Server and, at the same time, write them to something like Redis Cache. Then, every minute, get your data from Redis and use that to calculate the values for the other table and write to that. This approach follows (somewhat), the CQRS so you're separating the reads and writes from each other.
The beauty is, this scales well.
|
|
|
|
|
That was actually a brilliant Idea, as I already have Redis running 
The number of devices will hopefully increase 10 to a 100-fold, but this will still scale.
So will it with additional transformations in the future.
Thanks!
The only drawback with this solution is that I have no excuse to sit down and play with Datalakes
and all its tools... 
|
|
|
|
|
What worries you most? Is it the data volume? A back of the napkin calculation shows something like 10-20k per sec. Almost any DB system could handle that.
Mircea
|
|
|
|
|
Volume, Yes.
As of today, the volume is not a problem at all.
But when we get thousands of devices online, I'm a little bit afraid that the
Job generating the aggregates will struggle/lead to locks.
Also, who knows what other Post-processing needs will occur in the future.
|
|
|
|
|
How many more devices you anticipate to connect: 20000, 200000? At 20000 level, I think you should still be safe. At 200000, not so sure.
Keep in mind however that inserting is the more probable bottleneck as it has to create all those indexes, allocate pages, etc. Querying should be much faster.
Mircea
|
|
|
|
|
Thanks!
|
|
|
|
|
In one of the sub-threads you said the following
"The number of devices will hopefully increase 10 to a 100-fold"
You should have a very hard idea of what this increase looks like before you start making decisions. You should know the following.
1. What is the timeline for the increase? Also will the increase be gradual or will it be abrupt?
2. What is the maximum? If you owned the market what size would you need to support?
Typically when I size I do the following
1. Document my assumptions.
2. Document my sources - where did I get the assumptions and how did I derive them.
3. I then increase the size from that by 3 to 10.
Don't attempt to create a system the will work for the next 100 years. One that works for 10 is fine.
After you have those numbers then you start looking for solutions that will handle those numbers. Keeping in mind of course that your architecture/design should support the larger values, but the implementation does not need to support that now. You just want to make sure the implementation does not preclude sizing it up.
|
|
|
|
|
Thanks!
Very good points.
The question was more about what would a datalake offer in terms of post processingand other benefits, since I am not familiar with the technology.
I have no problems implementing an efficient solution with Azure/sql/c# for the foreseeable future.
Just needed some insights in a tech I'm not familiar with.
|
|
|
|
|
i am building an application in which one of the part is calendar updates , push notification ,
i am wondering what is best way/architectural pattern/technologies are best suited for this scenario . I am a .net developer .if possible kindly guide me on same platform
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
