Click here to Skip to main content
15,891,253 members
Please Sign up or sign in to vote.
5.00/5 (1 vote)
See more:
Hi,all
I've a large amount of real time data need to be proceed as fast as possible.

This data is coming from multiple threads over network connections.

All network threads pass the data to a shared function to process it with some translation and interpretation, after that it saves the information into Concurrent Dictionary object by object.

The problem is I have an amount of objects that reaches 150K stored in this dictionary.

What happens is while fetching the object to update, it takes a long time rather than the accepted time.

public class MyObject
{  
  System.Timers.Timer LostTimer = new System.Timers.Timer();
  public int ID;
  public DateTime UpdateTime;

  public  MyObject()
  {
    LostTimer.Interval = 20000;
    LostTimer.Elapsed+=TimerElapsedHandler(LostTimer_Elapsed);
    LostTimer.Enabled = true;
  }
 
  void LostTimer_Elapsed(object sender,EventArgs e)
  {
    if(UpdateTime > DateTime.Now.AddSeconds(-20))
         Console.WriteLine(ID + " Lost...");
  }
  
}

public class MyClass
{
  public MyClass(){}

  private ConcurrentDictionary<int,MyObject> Objects = new ConcurrentDictionary<int,MyObject>();

  void NetworkThread1DataRecived(eventArgs e)
  {
    Translate(e.Data);
  }
  void Translate(string[] data)
  {
   Task.Factory.StartNew(()=>
   {
      Parallel.ForEach<string>(data, s (()=>
      {
         MyObject o = null;
         Objects.TryGet(int.Parse(s),out o)
         if(o == null)
         {       
             o = new MyObject();
             o.ID = int.Parse(s);
             o.UpdateTime = DateTime.Now;
         
             Objects.TryAdd(s,o);
         }
         else
         {
            o.UpdateTime = DateTime.Now;
         }
      });
   });
 }
}


Now when working with more than 30K of objects it gives me objects lost.
Please I need help urgently.

Thank you!
Posted
Updated 28-Nov-10 22:45pm
v6

1 solution

Hi Tamer,

I found you what's wrong:

UpdateTime > DateTime.Now.AddSeconds(-20000)

The timer.Interval is set to 20000ms that is 20 seconds, but then you're subtracting 20000 second. I think that should only be 20 secs.

BTW, what logic are you following that when the above snippet is true
that the object is "Lost". Can you please explain?

Cheers

Manfred
 
Share this answer
 
Comments
Tamer Fahmy 29-Nov-10 4:51am    
Thanks Manfred, I've updated it well, the logic is that i'm subtracting the object grace period from the current system time and compare it with the last update time for that object.
Manfred, Do you think that this type of thread safe array (Dictionary) can not handle this large amount of data and causes read./write access delays which causes object lost???
before i were using List<myobject> and Lock(Object){} to handle the multi thread access to this shared memory, but it fails after 10K of objects.
after changing it to dictionary (.Net build in thread safe list) it works well with 30K.
my target is 150K, can i reach it with this logic or you have better ideas.
thanks for your previous answers.
Manfred Rudolf Bihy 29-Nov-10 4:57am    
I think the concurrent dictionary can handle this amount. List failed earlier I think because object lookup is much slower than on a dictionary. Please elaborate on the grace periond thing I'm still not getting it.
On thing you shoult definitely try is making the timer interval larger than the grace period.
Set timer.Interval to 30000 and subtract 20 seconds in your if expression.
Tamer Fahmy 29-Nov-10 4:59am    
Okay, the grace period is the time which the user accept the object to be inviable from system monitors.
So it is the time of allowance for each object to disappear without firing lost event for this object.
Manfred Rudolf Bihy 29-Nov-10 5:07am    
The problem with a large number of objects is that the timer events take some time to process. So when you subract 20 seconds from DateTime.Now last UpdateTime may well have passed. You're implying that every event is handled exacty at the specified timer interval which is NOT true. Please try my advice to make the timer interval larger than the grace period.

My favorite approach to your problem would be to use only one watch dog thread that monitors all objects in the dictionary by iterating over them and then rest for some time before cheking again.
Tamer Fahmy 29-Nov-10 5:07am    
I Do it but with the opposite, I made the grace period larger than the lost check timer in time.
lost check timer interval was 20 sec. and grace period is 40sec.
Look Manfred, I've used a code profiler and it gives me too much processing time at Translate Function.
I think because all objects came in parallel threads and calling the same function to either create new object or update earlier object.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900