|
Hello everyone,
I'm currently working on a C# project that processes a large dataset. While my code works correctly, it's running slower than expected, especially with bigger files. I'm looking for advice on optimizing the performance.
Here’s a brief overview:
-I'm reading data from a CSV file.
-Processing involves several nested loops and string manipulations.
-Writing results to a new file.
I've tried using Parallel.For for the loops but it didn't improve much. I suspect the string operations might be the bottleneck. Does anyone have suggestions on more efficient data handling or alternative approaches for processing large datasets in C# ? Any tips on using memory more efficiently or reducing execution time would be greatly appreciated.
Thank you in advance for your help.
Best regards,
Steve
|
|
|
|
|
off to the C# forum you go 
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
Aside from posting your question in the correct forum - did you somehow miss the repeated "no programming questions" warnings at the top of the page? - the only thing we can suggest based on such a vague description is that you profile your code to find out where the actual bottleneck is.
Anything else is going to be a wild stab in the dark, with very little chance of success.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Profile.
CI/CD = Continuous Impediment/Continuous Despair
|
|
|
|
|
|
As has been mentioned, this is not the right place for programming queries - post your request here instead: C# Discussion Boards[^]
But ... Think about a few things first.
1) Remember that we can't see your screen, access your HDD, or read your mind - we only get exactly what you type to work with - we get no other context for your project.
Imagine this: you go for a drive in the country, but you have a problem with the car. You call the garage, say "it broke" and turn off your phone. How long will you be waiting before the garage arrives with the right bits and tools to fix the car given they don't know what make or model it is, who you are, what happened when it all went wrong, or even where you are?
That's what you've done here. So stop typing as little as possible and try explaining things to people who have no way to access your project!
Have a look at this: Asking questions is a skill[^] and think about what you need to know, and what you need to tell us in order to get help.
2) Parallelism isn't a "magic bullet" which automagically speeds up your code: it's is a complicated subject and it's very simple to slow an app down if you don't understand what you are doing.
3) The first part of speeding up an app is identifying where bottlenecks occur: if you are guessing (and "I suspect ..." is a good clue that you are) then you are very likely to chase off down a blind alley making tiny gains in performance but ignoring the "slow code". Profile your app and find out what is actually taking time!
4) We can't tell you "do this and your code will be faster" without understanding your code - and nobody wants to wade through a whole app looking for performance gains: we are all volunteers and most of us have paying job to do!
So show us code fragments that we actually need to see - without those, we are just whistling in the dark!
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
So, is it just me or does Microsoft Loop look like a gound up re-work of OneNote? Microsoft Loop: Collaborative App | Microsoft 365[^]
Many of the core facilities of Loop were already available (admittedly by somewhat dubious pathways) in OneNote (real-time collaboration, embedding of documents and web pages, live sketching/writing sharing etc) albeit in a much 'clunkier' UI.
However, with a little investment, OneNote could IMHO have become a fantastic real-time collaboration tool without the need to be permanently on-line (something that is only possible on a small percentage of the world's surface despite what MS, Governemnts, telcos etc tell you) -improved handwriting tools and better recognition, plus the ability to properly embed/edit PDFs would have got it very close to the Loop feature-set, adding a code window linked to git, VS (locally hosted or on-line) or other dev environments would have got it the rest of the way.
I did try to log in to the discussion forum provided (which had no comments when I tried), but I got an 'unknown error', all of which rather sums up the whole mess of Teams/OneNote/Loop/Sharepoint etc - tools which all seem to provide lots of advantages if you can get them to work, but only seem to work in certain very tightly specified environments that most smaller orgs etc do not, and cannot, run cost-effectively.
I have relied on OneNote almost since it first came out (Desktop not APP version) but recently, after extensive research for something with a better feature set re handwriting/PDFs etc) have switched to Nebo Nebo[^] , which does everything and more that OneNote did, across multiple platforms, is still being actively developed, and still allows for off-line working and later sync. THe only drawback for me really is the 'app'-like UI, but I can live with that...
I'd be interested in hearing what you think!
|
|
|
|
|
onenote desktop team in ms was maybe three people....also now that things move to app format... desktop = stop
Caveat Emptor.
"Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
|
|
|
|
|
Indeed so - that was my point (I think!). OneNote could have become much more, but was effectively abandoned (probably for good reason, at least from the MS company culture viewpoint). If you have to start over with Loop, then that's fine for new teams etc, but for us oldies with years of project notes etc already, it makes more sense to move to something that allows us to retain/migrate our existing workflow etc.
As almost always happens with MS (and indeed most large orgs) the attraction of the shiny and new is greater than the consideration for what has gone before and so long-term customers are, once again and at best, simply not considered.
|
|
|
|
|
onenote was interesting and it tossed out some new ideas. I've been looking for an electronic engineering notebook for decades, and I actually had hope for one note - nah. It just got too clunky. Worse, somehow Microsoft managed to shove their head up where the sun doesn't shine and start indexing folders everywhere. This irritates the living daylights out of me because I use SVN heavily. So the stupid little files show up yada yada yada.
Microsoft does do some interesting work, but they always seem to manage to snatch defeat from the jaws of victory.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
|
I am so installing Unix.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
|
Wordle 1,077 3/6*
🟨⬛🟨⬛⬛
⬛🟨🟩🟨🟨
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 1,077 X/6
🟨⬜🟩⬜⬜
⬜⬜🟩🟨🟨
🟩🟩🟩⬜🟩
🟩🟩🟩⬜🟩
🟩🟩🟩⬜🟩
🟩🟩🟩⬜🟩
|
|
|
|
|
Wordle 1,077 3/6
🟨⬜🟩⬜⬜
🟩⬜🟩🟨🟨
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 1,077 3/6*
🟨⬜🟨🟨⬜
⬜🟨🟩⬜🟩
🟩🟩🟩🟩🟩
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
🟩⬜🟩⬜⬜
🟩⬜🟩🟨⬜
🟩🟩🟩🟩🟩
In a closed society where everybody's guilty, the only crime is getting caught. In a world of thieves, the only final sin is stupidity. - Hunter S Thompson - RIP
|
|
|
|
|
Wordle 1,077 6/6*
🟨⬜⬜⬜⬜
🟩🟩🟩⬜⬜
🟩🟩🟩⬜⬜
🟩🟩🟩⬜⬜
🟩🟩🟩⬜⬜
🟩🟩🟩🟩🟩
Had to guess all possibilities before getting it right 
Happiness will never come to those who fail to appreciate what they already have. -Anon
And those who were seen dancing were thought to be insane by those who could not hear the music. -Frederick Nietzsche
|
|
|
|
|
Wordle 1,077 5/6
🟨⬛🟨⬛⬛
⬛🟨⬛🟨⬛
🟨⬛⬛⬛🟨
⬛🟨🟨⬛🟨
🟩🟩🟩🟩🟩
Ok, I have had my coffee, so you can all come out now!
|
|
|
|
|
I'm not sure if this is by design, or a bug.
I created a new WPF project & solution. I added 4 previously existing projects, which all have their own repos, to the solution. I then added a couple of new class library projects to the soltion.
When I open the Git window, it shows the 4 previously existsing repos, but there is no repo for my new app. How do I add my new solution, WPF project, and class library projects to Git using VS?
If I remove the 4 pre-exising projects, and close & reopen the solution, then there is an menu option to create a repo for it. But then I have broken refs and have to fix those & commit.
This can't be right. What am I doing wrong here?
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
You'll have to create a new repository for the new application.
File->New->Repository
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
That created a new repo ib Git, but nothing appears local. The remote has no files in it either.
[UPDATE]
You have to set the Local Repository path. It sets it My Docs, which created an empty repo there. Once I changed it to my app's directory, it worked fine.
Thanks
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
Seems that the Internet Archive (The Wayback Machine) are undergoing a DDoS attack. The only possible reason I can think of for that is there is some company or gov that has something to hide.
I often find myself going to the Wayback Machine to verify what a big corporate once said.
That sources of history are being attacked is worrying, to say the least. What can we as a group do to protect a verifiable source of truth?
Here's one suggestion: Help them to go distributed.
I'm an old veteran programmer (35 years), and am mostly out of it now, although I still do some DB and embedded stuff, but I'm not up on the recent stuff.
We need people to step up and help protect the history stored by the IA and the WBM, or bad actors will erase it, or prevent access to it. Can you say 'George Orwell' ?
(@ the guy who writes the subtitles to all the headlines to CodeProject news... you rock!)
|
|
|
|
|
Trellian wrote: the guy who writes the subtitles to all the headlines to CodeProject news... you rock!
That would be, the one and only Kent Sharkey[^]
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment
"Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst
"I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
