|
The back end is C# / MVC / Knockout
I've been trying to figure this out for a while with no luck. I can get it to partially work but the session is never passed back to the end user.
What I'm trying to figure out is how to add a single signon (non SAML/oAuth; that's already working) to authenticate a user to a remote site using a username / password combination. Most sites are HTML so from the server I can issue a login and it works. If I try to do this through the browser I get cross-site scripting errors (which I would expect). I know other people are doing this successfully. I need to do this device / browser independent. Outside of writing either a plugin for each browser, how is this accomplished?
|
|
|
|
|
I have these options =
A. Store more data into one single JSON document
Or
B. Distribute them as multiple documents
Storing data in one single doc, gives a neat solution, but I'm not sure how big is too big for a JSON document. What if it runs to 40-50 MB? Parsing docs of this size feels insane?
UPDATED:
This document works like an accumulator. For example, lets assume a requirement like this :
I need to store all the users taking part in discussions in CodeProject.
So, I need to capture the User ID & have a counter against each of them.
{User:1332,
PostCount:23, (incremented every time he posts a reply)
},
{User:1124,
PostCount:56,
},
{User:2323
PostCount:34
}
The document keeps growing as and when new users start posting in CP.
The document gets updated for the count, when old users make another post.
The document is read when the user metrics are needed to be shown on reports.
Now I have the option to maintain it- Yearly, Monthly or weekly OR just Keep it as a single document. Where all the users activities would be recorded in a single doc.
If you are dividing the document as Yearly, Monthly,weekly etc, The document size would be reasonable- Somewhere around <4-5MB (Monthly), but the no. of document would be more in the collection. You'll have to run an aggregate function to collect all the data & calculate the Count that's spread across documents for every user and produce the metrics. As I just said, Here the individual documents size would be reasonable as it can't go on increasing. It just bound by the start & end date.
And Single document means, Keeping everything in single file and keep updating it.
For calcs, Load the whole data in memory, run the aggregate function to parse through the document and produce the metrics.
(Single Document sounds like a Big-Fat approach. I think I'd rather keep them as monthly chunks and make it handy to handle the documents.)
Starting to think people post kid pics in their profiles because that was the last time they were cute - Jeremy Falcon.
modified 1-Dec-17 5:28am.
|
|
|
|
|
At that size it is no longer an exchange-format, but used as a data-store. In that case SQLite would probably be a better option.
Once wrote an application that used XML as a datastore; every edit means rewriting the entire file.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
Yep its a data-dump-store. Though a single document looks neat on the design, I guess performance wise its a bad thing. Mainly for the reason you mentioned, it has to load whole lot of data into memory everytime it has to update or read something. On the other hand, if I'm distributing it into multiple documents, it will need proper indexing/partitionin/sharding things done right.
Starting to think people post kid pics in their profiles because that was the last time they were cute - Jeremy Falcon.
|
|
|
|
|
Vunic wrote: On the other hand, if I'm distributing it into multiple documents, it will need proper indexing/partitionin/sharding things done right.
JSON, XML and CSV are great formats for exporting data; but they do not work well as datastores where frequent updates are expected. If your data contains images, they'll probably be stored in Base64, making the file even larger. If there is no updates expected to the data, and it is merely a static export - then one of the text-based exports is preferred. Reading is fast enough for text, but updating takes a severe and noticable penalty.
Vunic wrote: Though a single document looks neat on the design
Also a lot easier when making backups and sending stuff. I prefer to put all data in a single file; with a file-based database like SQLite, SQLCe and MSAccess you can easily even store binary data. Another nicety is that you can link to these databases from SQL Server and interact with them as if they were local SQL Server databases.
If the data is not related at all, then I'd recommend zipping your various documents, so it remains "one file" in the eye of the end-user. Since you initially chose JSON, I'd be expecting data that is somehow related.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I have updated the OP[^] Please check the sample scenario.
Starting to think people post kid pics in their profiles because that was the last time they were cute - Jeremy Falcon.
|
|
|
|
|
You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
Most internet traffic is automatically "zipped" (gzipped) by default; and will on average compress to 10% of its original (text) size. That addresses bandwidth.
Where else is is it "too big"?
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
If you look at my post, that's exactly what I did. Instead of benchmarking, I'll remind you that any text-based file needs to be rewritten to update, and it does not support random access.
Zipping such a text-file means rewriting and rezipping the entire stream, and writing that result to disc on every change. That's fine for a datadump in CSV that is imported, but it is not a ideal structure for data that tends to be modified.
Gerry Schmitz wrote: Where else is is it "too big"?
GMail, if the attachment is larger than 25Mb. So yes, would depend on context, wouldn't it? 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I wasn't talking to you.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
No, you were posting on a public forum which allows answering even if someone is not talking to you. Legally and technically no problem, and not intended to offend.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Except you usually miss the point / "context" (as in this case) and I have no interest in "debating it" when I wasn't addressing you.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry, you're normally good when answering but in this case, you really are coming across as a jerk. Rather than defending your position, you have made a blanket statement and are being hostile when presented with opinions that you don't like. Eddy presented situations where it's "too big" and you're ignoring him just because he's not the OP; oh, and your point was only a very minor part of what the OP might have wanted to do so you were making assumptions as to the intended purpose, and failed to address the larger issues, which Eddy did.
This space for rent
|
|
|
|
|
I was asking OP to elaborate: in what "context" did he think "his" data was too big. My experience re: JSON was in REST; I was alluding to bandwidth. i.e. the internet. And that browsers generally "zip" unless you turn it off.
Instead, I get addressed as if "I" was questioning said "offendee" about "his" benchmarking? And then he goes off on "email" attachments?
If one waits "a round or two", the context becomes more clear, and you don't come off as a "jerk" by derailing someone else because you're so eager to show "how smart your are".
It's (his) bait; you just can't see it but took it.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: If one waits "a round or two", the context becomes more clear, and you don't come off as a "jerk" by derailing someone else because you're so eager to show "how smart your are".
The hardest part is to get to the first round. I did not intend to be a jerk, but don't mind to be called that either
Gerry Schmitz wrote: It's (his) bait; you just can't see it but took it.
FWIW; there may come a day when we are forced to work on the same team.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
I was being rhetorical at this point because Hanlon has already said that I'm the "jerk" here.
And I wouldn't "force" anyone or myself to work with anyone they didn't want to. Or is that what you're used to?
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: I was being rhetorical at this point because Hanlon has already said that I'm the "jerk" here.
That's one opinion; enough people will say that I am a well trained jerk. Luckily it is not a competition.
Gerry Schmitz wrote: And I wouldn't "force" anyone or myself to work with anyone they didn't want to. Or is that what you're used to?
I prefer to work with people who are honest and who aren't afraid to say I might be wrong. Since you're a good programmer (with posts here to back that up) I'd say you fit the profile perfectly 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
To my users, I'm a necessary evil.
(Supposedly, those who "swear" a lot are also more "honest").
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: To my users, I'm a necessary evil.
My co workers may same they same thing. We do this because we're good with computers, if we were good with humans we would be doing something entirely different.
Just published an article on CP on the subject of patterns, and given the thread on patterns below, I would like your opinion. Would be worth a lot if it would change your initial reaction to "some patterns could be usefull" 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
If we were good with humans, we would be rich.
Never said patterns weren't usefull ... said most conversations (in my experience) aren't at a level where more than one person can "talk" "GO4" patterns. (and "most" think of GO4 if they think patterns; not someone else's).
All my "patterns" are based around "adapters" and builders and repositories: devices; threads; monitors; charts; recorders; serializers; 3rd party API; etc.
"Facades" and "decorators" just don't enter the conversation.
And my patterns evolve; some patters (industrial) I have modified over time based on "looking" at it in a different way.
New patterns and algorithms: quantum computing ...
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: Never said patterns weren't usefull ... said most conversations (in my experience) aren't at a level where more than one person can "talk" "GO4" patterns. (and "most" think of GO4 if they think patterns; not someone else's).
That must be some good marketing from the GoF.
Gerry Schmitz wrote: If we were good with humans, we would be rich.
..or in prison for attempting 
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Good marketing? Just a testimony to how slavish some organizations can be; no independent thought.
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
|
Your metrics are too vague to say how much "intelligence" you need but I would consider using Power BI to do a simulation of your requirements to see how much custom code you may (not) need.
I would design the best "operational" system for daily use; and create an "informational" db / extract to help with metrics if needed.
Using Power BI with JSON Data Sources and Files
Power BI Desktop | Microsoft Power BI
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
|
Gerry Schmitz wrote: You can't just look at a "number" and say it is "too big" (or small) without benchmarking.
I can.
But then I have been doing this for a very long time.
From the post as presented I would agree that keeping the data in the object is not a good idea.
|
|
|
|
|
Certainly ... but that is based on "previous" observations and "related" experiences; unless one is in the habit of guessing.
One generally doesn't ask questions on a subject one is already familiar with.
Since the question was asked, I assumed this was a "new" territory for OP. Benchmarking is one of the things I use in new territory.
A sport stats web site thought their 400 MB baseball "game files" were also "too big" for sending over the internet. They compressed down to 25MB when I could have made $ coming up with an alternate solution. Initially, even I thought 400 MB was "big".
"(I) am amazed to see myself here rather than there ... now rather than then".
― Blaise Pascal
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
