Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: C# Threading
hi everyone,
In my application I need to collect the information of all the files in folder and send them to database.Since this is to be done for whole disk, this is a very time taking process. What I want to do is to create a seperate thread for each file in a folder and write each file simultaneously to db. I am not getting any idea of how to start it. If anyone could please help me in creating an array of threads for all files which will run simultaneously. Below is my code snippet where I need to do this.
 
private static void collectFileInfo(String dir)
        {
            try
            {
                String[] files = Directory.GetFiles(dir);
                DirectoryInfo dir1 = new DirectoryInfo(dir);
                string chkFlPath = Convert.ToString(dir1.FullName);
                Int32 ParentID = GetParentID(dir1.Name.ToString(), chkFlPath);
                if (files.Length > 0)
                {
           
                    for (Int32 i = 0; i < files.Length; i++)
                    {
                        FileInfo File1 = new FileInfo(files[i]);
                        double FileLength = File1.Length;// File size
                        string strExtn = System.IO.Path.GetExtension(File1.Name.ToString());
                        String fileName = File1.Name;
                        String filePath = File1.FullName;
                        string ParentName = dir.ToString();
                        String fileSize = Convert.ToString(Convert.ToInt32(File1.Length) / 1024);
                        String fileExtension = File1.Extension;
                        DateTime fileCreated = Convert.ToDateTime(File1.CreationTime);
                        DateTime FileModified = Convert.ToDateTime(File1.LastWriteTime);
                        string md5hash = GetMD5Hash(filePath.ToString());
                        string sh1hash = GetSHA1Hash(filePath.ToString());
                        DateTime lastAccessedDate = Convert.ToDateTime(GetLastAccessedDate(filePath.ToString()));
                        Thread t = new Thread(delegate()
                            {
                                WriteToTable(ParentID, fileName, filePath, ParentName, true, dir1.FullName, fileExtension, fileSize, fileCreated, FileModified, md5hash, sh1hash, lastAccessedDate);
                            }
                       );
                        t.Start();
                    }
                }
            }
            catch (Exception ex)
            {
            }
        } 
 
Thanks & Regards
Anurag
Posted 27-Oct-10 0:29am
Edited 27-Oct-10 1:30am
v2
Comments
@nuraGGupt@ at 27-Oct-10 6:34am
   
Hi Ankur, please help me, I can see that you edited something here,
 
Edited 3 mins ago
Ankur\m/12.4K
 

 

that means you read the post. Can you provide any solution....
@nuraGGupt@ at 27-Oct-10 6:59am
   
Thanks for giving me 4 points on my question, whoever did it :-)
Ankur\m/ at 27-Oct-10 9:02am
   
Yeah, I did a minor edit and I voted you 4. But I am sorry I can't help you much with this.
 

Ahh.. well, okay, let me read the question again. :)
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

For such large scale operations i used to use SQLite in an order that i was doing something like SQLiteDB.Query('update table');
and after all those update table i did Commit();
And it ended up way faster.
I cannot know what is the API used inside WriteToTable but if your DB can do it i suggest you build a sql script with all those updates and shove it inside the db.
 
As for threading i use:
private readonly object lockObject = new object();
volatile int jobCount;
private bool Stopped;
 
private void btnGo_Click(object sender, EventArgs e)
{
    btnGo.Enabled = false;
    ThreadPool.QueueUserWorkItem(new WaitCallback(Start));
}
 
private void Start(object obj)
{
    jobCount = 0;
    Stopped = false;
    for (int i = 0; i < Environment.ProcessorCount; i++)
    {//this for could be changed to fit your needs ofc
        ThreadPool.QueueUserWorkItem(new WaitCallback(execute),null);//you could pass a FileInfo here for example instead of null
        jobCount++;
    }
    lock (lockObject)
    {
        while (jobCount > 0)
        {
            Monitor.Wait(lockObject);
        }
    }
    ThreadStart finisher = new ThreadStart(Finish);
    finisher.Invoke();
}
 
private void execute(object obj)
{
    //FileInfo file = obj as FileInfo;
    while (!Stopped)
    {
        Thread.Yield();
        //execute code here
    }
    Decreace();
}
 
private void Decreace()
{
    lock (lockObject)
    {
        jobCount--;
        Monitor.Pulse(lockObject);
    }
}
 
private void Finish()
{
    Thread.Yield();
    btnGo.Enabled = true;
    //other code goes here, you can also skip the next command if you like but i prefer it to release ram after large operations even if it slows down my process alot. But in your case it would make your process seem alot faster though so its your decision.
    GC.Collect();
    Application.DoEvents();
}
 
Edit: forgot to declare and initialize Stopped variable :P
Edit2: Changed Code to assume not within a form (i.e finisher.Invoke())
  Permalink  
v3
Comments
@nuraGGupt@ at 4-Nov-10 1:07am
   
Hello qyte64,
Here Invoke method is not defined, and I am not getting that how to use it.Please specify.
qyte64 at 4-Nov-10 6:34am
   
It stands for Form1.Invoke() and since this code is inside the body of Form1 i omitted it.
Invoke basically executes a delegate from the Main Thread, i.e. a thread asks the main thread to execute a delegate.
qyte64 at 4-Nov-10 11:00am
   
I edited the Code and replaced the Invoke() with a ThreadStart finisher delegate that does the job.
@nuraGGupt@ at 11-Nov-10 5:08am
   
ok, but one more thing, what is this Thread.Yield(), I am not getting this method (Yield()),when I am putting a period(.) after "Thread" keyword. Please explain.
@nuraGGupt@ at 11-Nov-10 5:14am
   
I would like to clearify that I am using WPF in my application, not the traditional ASP.NET Forms Application, so that may be why I may not be getting these methods in my application, Application.DoEvents(), Thread.Yield() etc. Do you have any other way round for this?
Thanks
qyte64 at 11-Nov-10 5:27am
   
Just ignore them (comment them out i mean).
My example was meant for Windows Forms but really there should be no problem if you remove them.
@nuraGGupt@ at 11-Nov-10 5:37am
   
ok, thanks a lot.
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 1

Actually your code snippet doesn't look that bad. What is the actual problem with it?
To improve the speed you could outsource more of your logic to the thread and put more code outside the "For" loop.
By the way, it doesn't make sense to create as much threads as files. You should limit the number of threads to the number of CPU's/Cores in the system.
  Permalink  
v2
Comments
@nuraGGupt@ at 27-Oct-10 5:51am
   
At present, using this code, I am writing 5000(approx) file metadata to the table per minute, but my client wants it to make more faster. So, I am looking a way fit Boeing 747 engine to a chopper :-)

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 OriginalGriff 295
1 DamithSL 265
2 CPallini 235
3 Maciej Los 190
4 Sergey Alexandrovich Kryukov 174
0 OriginalGriff 5,415
1 DamithSL 4,422
2 Maciej Los 3,820
3 Kornfeld Eliyahu Peter 3,470
4 Sergey Alexandrovich Kryukov 2,911


Advertise | Privacy | Mobile
Web03 | 2.8.141216.1 | Last Updated 4 Nov 2010
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100