|
Hi Guys. I have now searched the Web and can't seem to find the exact solution for my problem. I am trying to process a file of just over 2gb in size. File contains 879 987 lines of data.
I am using the FileHelpers library to read the file, search for specific values and then writing other data from the file out to a new file.
MY problem is that It takes a few hours to read the file and I have to abort the job. Is there a better way of doing this? I have resorted to writing a splitter to split the file into smaller chunks and then processing the smaller 20+ files which is much faster but also takes a bit of time to do.
Can anybody please advise of a better or faster way of reading the one big file and doing what I am doing below??
string filePath;
filePath = txtBoxSelectFile.Text;
FileHelperEngine<IvalueRev> engine = new FileHelperEngine<IvalueRev>();
engine.ErrorManager.ErrorMode = ErrorMode.SaveAndContinue;
IvalueRev[] res = engine.ReadFile(filePath);
if (engine.ErrorManager.ErrorCount > 0)
engine.ErrorManager.SaveErrors(@"C:\Errors.txt");
StreamWriter sw = new StreamWriter(filePath + ".txt");
foreach (IvalueRev rev in res)
{
if (rev.Flag == "2" || rev.Flag == "4" || rev.Flag == "5" || rev.Flag == "6" || rev.Flag == "7" || rev.Flag == "8" || rev.Flag == "9")
{
sw.WriteLine(rev.HID.ToString() + "\t" + rev.Flag.ToString());
}
}
sw.Close();
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
Hi,
the way I see it, your app would run much faster without the "FileHelperEngine".
Reading a 2GB file should take a couple of minutes, and no more.
The trouble with your "helper" class is it insists on processing the file as a whole, since it
returns an array of all the lvalueRev objects, whereas you only need them sequentially, as shown
by your foreach loop.
Hence I suggest you write some more code to read the file yourself, getting one lvalueRev out of it
at a time, dramatically reducing the memory load (and possible memory paging) the original is causing.
|
|
|
|
|
Thanks Luc. I will look into this. I wrote another simple line counter counting the lines in the same file and that runs for about 4minutes. I am just not sure how to "search the file for specific values at specified offsets but will look this up. Below the simple counter I wrote earlier. I guess it should work on the same principle, correct??
string filePath;
filePath = textBox1.Text;
int totalRows = 0;
Stream stream = new FileStream(filePath, FileMode.Open);
BufferedStream bstream = new BufferedStream(stream);
StreamReader reader;
String line;
reader = new StreamReader(bstream);
while((line = reader.ReadLine()) != null)
{
totalRows++;
}
stream.Close();
MessageBox.Show("Total NUmber of Records in the File: " + totalRows.ToString());
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
Hi,
the code shown is fine.
I do not know what your file looks like, nor what you mean exactly by "search the file for specific values at specified offsets".
You will get linear performance (twice as big takes twice as long, and no more) as long as you can solve your problem while reading the file sequentially without memorizing it all. So if you know in advance you need specific lines (say by line numbers), then sort the line numbers, read the file sequentially and check the current line number with the next one of interest.
If your offsets are byte offsets relative to the file's origin, then you could still loop while reading
lines as you did, and ask the stream for the current position, and act accordingly. This will work just
fine as long as you don't need to move backwards; again, you may need to sort the offsets first
(I am hoping you get all the offset values at the same time, otherwise you would not be able to sort them).
|
|
|
|
|
you can just pass the filepath straight to stream reader.
if you want to look for offsets then have a byte counter and read bytes in from file
i.e.
StreamReader sr = new StreamReader("C:\\File.txt");
int b = sr.BaseStream.ReadByte();
long Count = 1;//for read byte
while(b != -1)
{
if(Count == OffsetValue)
//Do something
Count++;
b = sr.BaseStream.ReadByte();
}
sr.Close();
|
|
|
|
|
It really all depends on what you need to do with the data. If you will be processing the data one line at a time, then using the StreamReader along with the ReadLine() method should do the trick: you read in a line and process it, read in the next line, etc. You definitely don't want to try to hold all the lines of the entire file in memory before processing them as it is unnecessary waste of time and resources and is probably what the FileHelper class you are currently using is doing. Using the StreamReader.ReadLine() to loop over the file line by line is simple and only takes this:
StreamReader reader = null;
try
{
reader = new StreamReader(filePath);
string holdLine = String.Empty;
while(!reader.EndOfStream)
{
holdLine = reader.ReadLine();
if(String.IsNullOrEmpty(holdLine))
continue;
<... process the line of data here ...>
}
}
catch(Exception ex)
{
<... handle the exception here ...>
}
finally
{
if(reader != null)
reader.Close();
}
Keep It Simple Stupid! (KISS)
|
|
|
|
|
Thanks Ben. What I am attempting to do is the following. I need to read the file searching for specific values, "2, 4,5,6,7,8,9" at offset 653. Once a match is made with one of the values above then I need to write data from offset 1134 for 11 characters to an output file.
Now, with the filehelpers engine this is easy but I am not sure how to do this manually as the filehelpers engine lets you specify the different offsets in a class. You can then just write the specific fields as output. As I do in the below piece of code.
foreach (IvalueRev rev in res)
{
if (rev.Flag == "2" || rev.Flag == "4" || rev.Flag == "5" ||
rev.Flag == "6" || rev.Flag == "7" || rev.Flag == "8" || rev.Flag == "9")
{
sw.WriteLine(rev.HID.ToString() + "\t" + rev.Flag.ToString());
From the above, I search the file for the values as stated. rev.Flag is the flag at pos 653. Once a match is made then I write rev.HID out to an output file.
Hope this explains what I am trying to do.
Excellence is doing ordinary things extraordinarily well.
|
|
|
|
|
Kwagga,
When you refer to values at offset 653 and 1134, are you referring to that offset in a particular line of text? In other words, do you need to read 11 characters starting from position 1134 when the character at position 653 in the line of text is 2, 4, 5, 6, 7, 8, or 9?
If so this would be how you do it (not sure if the offsets of 653 and 1134 are zero-based):
StreamWriter sw = ...;
StreamReader reader = null;
char flag = ' ';
try
{
reader = new StreamReader(filePath);
string holdLine = String.Empty;
while(!reader.EndOfStream)
{
holdLine = reader.ReadLine();
if(String.IsNullOrEmpty(holdLine))
continue;
flag = holdLine[653];
if(flag == '2' || flag == '4' || flag == '5' || flag == '6' ||
flag == '7' || flag == '8' || flag == '9')
sw.WriteLine(holdLine.Substring(1134, 11) + "\t" + flag.ToString());
}
}
catch(Exception ex)
{
<... handle the exception here ...>
}
finally
{
if(reader != null)
reader.Close();
if(sw != null)
sw.Close();
}
In order to account for what you are trying to do and not have to hold the contents of the file in memory, in the FileHelper class one alternate would be for it to have an event that could, for example, be triggered each time a row is read from file. Then, you could handle the event and process each row as it comes in, utilizing the objects that the FileHelper class provides because the event could supply the IValueRev object via a custom EventArg. It seems that there are some objects the FileHelper class creates that probably interpret the data and make it easier to work with, so it has value for the system you are working within. Perhaps the best solution is not to abondon the tool if it indeed has a valuable capability, but to expand it to be more flexible.
Keep It Simple Stupid! (KISS)
|
|
|
|
|
How should I set the DisplayMember and ValueMember properties for my ComboBox 'addressSourceComboBox' please? Here is my code:
private void PopulateCorrespondenceAddressComboBox()
{
ArrayList CorrespondenceSources = new ArrayList();
int corrCount = 1;
foreach (DebtorDataSet.DebtorRow debtorRow in DebtorDataSet.Debtor)
{
CorrespondenceSources.Add(new CorrAddress("D" + corrCount.ToString(), "Debtor " + corrCount.ToString() + " - " + debtorRow.Salutation.ToString() + " " + debtorRow.Forename.ToString() + " " + debtorRow.Surname.ToString()));
corrCount++;
}
addressSourceComboBox.DataSource = CorrespondenceSources;
}
private class CorrAddress
{
private string _source;
private string _text;
public CorrAddress(string source, string text)
{
_source = source;
_text = text;
}
public string ItemSource
{
get { return _source; }
}
public string ItemText
{
get { return _text; }
}
}
|
|
|
|
|
it's OK I figured it out:
addressSourceComboBox.DataSource = CorrespondenceSources;
addressSourceComboBox.ValueMember = "ItemSource";
addressSourceComboBox.DisplayMember = "ItemText";
|
|
|
|
|
Hi all
I have a DateTimePicker control in my windows application(c#)
am taking the selected value from the DateTimePicker to a textbox...now am getting the value
in the format of ' dd-mm-yyyy ' but i want to get the value like ' dd/mm/yyyy '...
i have given all these..
dateTimePicker1.CustomFormat = "dd/MM/yyyy";
dateTimePicker1.Format = DateTimePickerFormat.Custom;
txtdate.Text = dateTimePicker1.Value.ToShortDateString();
how can get the value like which i have mentioned...
any help is appreciatable....
thanks in advance...
|
|
|
|
|
txtdate.Text = dateTimePicker1.Value.ToShortDateString().Replace("-", "/");
That should do the trick.
Kinda ugly, but it should work.
Kristian Sixhoej
|
|
|
|
|
Thanks Kristian...
It wrked well....
|
|
|
|
|
Kristian Sixhoej wrote: Kinda ugly, but it should work.
This is really ugly. What's wrong with using ToString() with a proper format specifier?
modified 12-Sep-18 21:01pm.
|
|
|
|
|
You have half the answer yourself - you know the format and how to specify. What you may not know is that dateTime has an override of ToString which can take this format.
txtdate.Text = dateTimePicker1.Value.ToString("dd/MM/yyyy");
|
|
|
|
|
This should work
txtdate.Text = dateTimePicker1.Value.ToString("dd/MM/yyyy");
Hope this helps
|
|
|
|
|
hi....
actually am doing a simple application using
" WCF Service Application " & " ASP.NET Web Application "..........
my code is correct and everything is perfect..
but when i run, it displays the following message....
" An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue "
when i run in the morning the output is displayed..but after sometime , when i run it shows that message.....
whats the solution...??...
|
|
|
|
|
Looks like you are running out of resources.
Just a thought: Are you cleaning up the resources you are using? Do you have a lot of objects that implement the IDisposable interface that you are not disposing correctly?
|
|
|
|
|
So I read a file and that file has like 9k lines.
the lines look like this(well it might not fit here, but is on a line only):
A:"extatic" A:"extatici" A:"extaticul" A:"extaticului" A:"extaticii" A:"extaticilor" A:"extatică" A:"extatice" A:"extatica" A:"extaticei" A:"extaticele" A:"extaticelor"
And I use textreader in c# to find a char in that text for example "extaticele" here.
What I want is to extract if I found the char on that line the first char on that line. In this case "extatic" - but without "" of course and paste it in a text file.
I dont know exactly how to work with find space so I want your help...cause I thought I'd find the first space in the line then take the index and with the index 0 of the line to extract "A:"extatic" ".
Then remove the "A:" and the ".
I saw somewhere an example with myString.find("") , but I can't find the "find" I might need a namespace.
|
|
|
|
|
In .Net, reference types are also objects. That means that most of the commands that you are looking for are present in the string itself. You can play with stuff like:
String test = "abcd";
int index = test.IndexOf('b');
String test2 = test.Substring(index);
// etc...
There are plenty of methods on the string object. There are also static methods in the String class, like for example:
String.Compare(string1, string2...)
Last but not least, the namespace you are looking for is simply System.
Jean-Christophe Grégoire
|
|
|
|
|
I use that namespace, but there is no mystring.Find - I just want to find the index of the first space in a line
|
|
|
|
|
ipstefan wrote: I use that namespace, but there is no mystring.Find - I just want to find the index of the first space in a line
int index = myString.IndexOf(" ");
Kristian Sixhoej
|
|
|
|
|
Yep I thought at that option too earlier and it works...
I solved my problem with indexes and substrings.
Thanks
|
|
|
|
|
You're welcome
Kristian Sixhoej
|
|
|
|
|
hi ipstefan,
my sugession is read contents of file sequentially and check whether the character u r reading is ", it it is then take all next characters untill next " apprears so u can get each character between "". From these chars u can make a string by using + operator and later it can be pasted where u want.
Or as u mentioned myString.find() method, i hope find method is custom method not in framework. Instead, u can use following method of class String.
String myString = "abcd";
String cutString = myString.Substring(0,3);
the value of cutString will be 'abc' . Try to use this method. Ok..
Hope it will help u.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
