How to get values from LIST[STRING] into LIST[Object]

Question

0.00/5 (No votes)

See more:

Hi Friends,

the question sounds simple but I have to elaborate more what I want.

In my application I read a file and set all the lines in a LIST<STRING> _allFileLines object.
I now iterate with a for each over the collection to read all lines and parse the values to a float[] fieldsPerLine. I get 4 float values per array. That's OK.

After that I set the values in an object called FileItemValues
That object holds 13 float values. The 4 in the array and 9 calculated values based on the first 4.

Each FileItemValues is added to a LIST<FileItemValues>.

Since each file holds minimal 30,000 records, it takes around 30 seconds to process(iterating, calculating, adding to new object and determine the Maximum values for each of the 13 properties in the LIST<FileItemValues>

How can this be done quicker with LINQ where the steps are:
- From List<String> to List<FileItemValues> for only the first 4 values found in List<String>
- calculate the 9 other fields in the List<FileItemValues>
- Determine maximum values for each property in List<FileItemValues> set in a FileItemValues object.

Regards

ps:
current iterating from list<string> to float[]:

C#

foreach (var fileLine in _allFileLines)
            {
                float[] fieldsPerLine = fileLine.Trim().Split(new char[]{','}, StringSplitOptions.RemoveEmptyEntries).Select(float.Parse).ToArray();

At this moment the next part is slowing me down:

C#

foreach (var fileLine in resultProef)
            {
                FileItemValues currentLineValues = new FileItemValues();
                currentLineValues.MilliSecs = fileLine[0];
                currentLineValues.PosX = fileLine[1];
                currentLineValues.PosY = fileLine[2];
                currentLineValues.PosYerk = fileLine[3];
                currentLineValues.MutatieSnelheidX = _positions.Count > 0 ? (currentLineValues.PosX - _positions[_positions.Count - 1].PosX) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieSnelheidY = _positions.Count > 0 ? (currentLineValues.PosY - _positions[_positions.Count - 1].PosY) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieSnelheidRotatie = _positions.Count > 0 ? (currentLineValues.PosYerk - _positions[_positions.Count - 1].PosYerk) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieAcceleratieX = _positions.Count > 0 ? (currentLineValues.MutatieSnelheidX - _positions[_positions.Count - 1].MutatieSnelheidX) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieAcceleratieY = _positions.Count > 0 ? (currentLineValues.MutatieSnelheidY - _positions[_positions.Count - 1].MutatieSnelheidY) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieAcceleratieRotatie = _positions.Count > 0 ? (currentLineValues.MutatieSnelheidRotatie - _positions[_positions.Count - 1].MutatieSnelheidRotatie) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieJerkX = _positions.Count > 0 ? (currentLineValues.MutatieAcceleratieX - _positions[_positions.Count - 1].MutatieAcceleratieX) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieJerkY = _positions.Count > 0 ? (currentLineValues.MutatieAcceleratieY - _positions[_positions.Count - 1].MutatieAcceleratieY) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                currentLineValues.MutatieJerkRotatie = _positions.Count > 0 ? (currentLineValues.MutatieAcceleratieRotatie - _positions[_positions.Count - 1].MutatieAcceleratieRotatie) / (currentLineValues.MilliSecs - _positions[_positions.Count - 1].MilliSecs) : 0;
                _positions.Add(currentLineValues);
               
            }

Posted 30-Nov-15 23:06pm

Herman<T>.Instance

Updated 1-Dec-15 0:49am

v6

Add a Solution

Comments

F-ES Sitecore 1-Dec-15 5:45am

Doing things with linq is going to be slower, not quicker. Linq is a convenience technology, not a performance-enhancing one. If you are looping _allFileLines tens of thousands of times them one thing that might help is to define your separator array only once.

char[] sep = new char[]{','};
foreach (var fileLine in _allFileLines)
{
float[] fieldsPerLine = fileLine.Trim().Split(sep},

Herman<T>.Instance 1-Dec-15 6:00am

speeded that up with:
var resultTest = _allFileLines.Select(x => x.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries).Select(float.Parse).ToArray()).ToList();

It is iterating the LIST and calculating (currentline-prevline values) that slow me down tremendous. But after that the max values per float column should be easily found with linq

Richard Deeming 1-Dec-15 6:09am

I'd start by not reading all of the lines into an array (File.ReadAllLines); try using File.ReadLines instead, which lets you process each line as it's read.

Beyond that, you'll need to profile your code to see what's taking the most time, and concentrate your efforts there. If you can't see how to improve it, you'll need to post the relevant code here, rather than a vague description of the process. :)

Herman<T>.Instance 1-Dec-15 6:29am

Reading in to array is done in the blink of an eye. It is really the calculating that is slowing me down. Calculations hold values as difference from current and previousline

Richard Deeming 1-Dec-15 6:33am

But you haven't shown us that part of the code. We can't help you to fix something if we can't see it! :)

Herman<T>.Instance 1-Dec-15 6:48am

Your post have made me thinking.
What I did was determining max values in the iteration in stead of with LINQ after that. Validating rules are now after iterating and using LINQ. processing went down from 30 to 7 seconds for 25501 lines. Will add the code

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard Deeming · Accepted Answer · 2015-12-01T02:30:00

You might get some improvement by storing the last added item in a variable, rather than retrieving it from the list every time.

You should ensure that the _positions list has sufficient capacity before you start the loop.

You could also pre-calculate the denominator, since each property is divided by the same value.

C#

_positions.Capacity = _positions.Count + resultProef.Count;

FileItemValues previousLineValues = null;
foreach (var fileLine in resultProef)
{
    FileItemValues currentLineValues = new FileItemValues();
    currentLineValues.MilliSecs = fileLine[0];
    currentLineValues.PosX = fileLine[1];
    currentLineValues.PosY = fileLine[2];
    currentLineValues.PosYerk = fileLine[3];
    
    if (previousLineValues != null)
    {
        float denominator = currentLineValues.MilliSecs - previousLineValues.MilliSecs;
        
        currentLineValues.MutatieSnelheidX = (currentLineValues.PosX - previousLineValues.PosX) / denominator;
        currentLineValues.MutatieSnelheidY = (currentLineValues.PosY - previousLineValues.PosY) / denominator;
        currentLineValues.MutatieSnelheidRotatie = (currentLineValues.PosYerk - previousLineValues.PosYerk) / denominator;
        
        currentLineValues.MutatieAcceleratieX = (currentLineValues.MutatieSnelheidX - previousLineValues.MutatieSnelheidX) / denominator;
        currentLineValues.MutatieAcceleratieY = (currentLineValues.MutatieSnelheidY - previousLineValues.MutatieSnelheidY) / denominator;
        currentLineValues.MutatieAcceleratieRotatie = (currentLineValues.MutatieSnelheidRotatie - previousLineValues.MutatieSnelheidRotatie) / denominator;
        
        currentLineValues.MutatieJerkX = (currentLineValues.MutatieAcceleratieX - previousLineValues.MutatieAcceleratieX) / denominator;
        currentLineValues.MutatieJerkY = (currentLineValues.MutatieAcceleratieY - previousLineValues.MutatieAcceleratieY) / denominator;
        currentLineValues.MutatieJerkRotatie = (currentLineValues.MutatieAcceleratieRotatie - previousLineValues.MutatieAcceleratieRotatie) / denominator;
    }
    
    _positions.Add(currentLineValues);
    previousLineValues = currentLineValues;
}