Click here to Skip to main content
15,922,015 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more:
I had a function that will read all the lines in the text file and output it in a datagridview, my problem is when the text file is about 600MB it will throws an exception. I also use reg expression to parse the data in different parts and for viewing purposes in the datagridview

C#
static readonly Regex ParseLineRegex = new Regex(@"^(?<ip>\S*) - - \[(?<date>.*)\] (?<method>\S+) (?<url>.+) (?<code>\d+) (?<tail>.+)", RegexOptions.Compiled);

DataColumn ip1 = new DataColumn("IP Address");
DataColumn date1 = new DataColumn("Date");
DataColumn method1 = new DataColumn("Method");
DataColumn url1 = new DataColumn("URL");
DataColumn code1 = new DataColumn("Code");
DataColumn tail1 = new DataColumn("Tail");

ip1.DataType = System.Type.GetType("System.String");
date1.DataType = System.Type.GetType("System.String");
method1.DataType = System.Type.GetType("System.String");
url1.DataType = System.Type.GetType("System.String");
code1.DataType = System.Type.GetType("System.String");
tail1.DataType = System.Type.GetType("System.String");

dt.Columns.Add(ip1);
dt.Columns.Add(date1);
dt.Columns.Add(method1);
dt.Columns.Add(url1);
dt.Columns.Add(code1);
dt.Columns.Add(tail1);

using (StreamReader reader = new StreamReader(txtSource))
{
    while ((line = reader.ReadLine()) != null)
    {
        var match = ParseLineRegex.Match(line);
        if (match.Success)
        {
            //ip, date, method, url, code and tail - Parsing
            var ip = match.Groups["ip"].Value;
            var date = match.Groups["date"].Value;
            var method = match.Groups["method"].Value;
            var url = match.Groups["url"].Value;
            var code = match.Groups["code"].Value;
            var tail = match.Groups["tail"].Value;

            #region Viewing Purposes
            dr[ip1] = ip;
            dr[date1] = date;
            dr[method1] = method;
            dr[url1] = url;
            dr[code1] = code;
            dr[tail1] = tail;

            dt.Rows.Add(dr.ItemArray);
            #endregion

        }
    }
}


PLEASE HELP.
Thank you.
Posted
Updated 20-Feb-13 17:33pm
v2
Comments
Sergey Alexandrovich Kryukov 21-Feb-13 1:13am    
You should radically review the code design. You need to give up reading the whole file and shift to file-based storage, search, etc. That's all. Particular detail needs knowledge of your goals, files structure and semantics of its parts, etc., something you did not share with us. Perhaps you would need to shift from big files to RDBMS. It all depends.
—SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900