Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: C# Exception memory
Dear all,
 
I'm pretty sure I know what the problem is....could I please ask for your time and consideration as to the optimum course of action
 
I have an application which gets data from a USB source and stores that data in a list. For convenience during development, I made it a public static such that it could be referenced by several forms at once.
 
public static List <MyListType> MyList = new List<MyListType>();
 
Data is written into this list by only one of the forms (which also is the main thread and GUI). Other forms need access to properties of this list e.g.
 
MyList.Count
IndexOf(MyList) where an element value matches some criteria
MyList[val].Data where several element values match some criteria
 
I cannot know the ultimate size of the list when the data from USB is being acquired. It would not be unreasonable to leave this application going for several hours to acquire potentially 100MB or more of data in this list.
 
I have read up on using predicate delegate functions to improve the application performance of searching through this list - it get a bit slow at the moment when you have a large list - but that can be improved. What I can't think of at present is a good way of stopping the application crashing after (say) 10-30minutes of operation because its out of memory.
 
It isn't the only statically allocated list in the app, but it is the only one that is going to be bigger than 2k (and those others can easily be changed).
 
I was going to propose to make the list private and non-static and declare it in the form which writes to this list. If I understand correctly, my options to access it in other forms would be to do something like the following:
 
In the main form which writes the data:
 
private List<MyListType> MyList = new List<MyListType>();
public List<MyListType> MyList_Access
{
    get { return MyList;}
}
 
In a form that wants to access the list:
private List<MyListType> _privList = new List<MyListType>();
 
Then.... to actually get at the original list I have to 'find' the main form and cast its type...
 
_privList = mytype.MyList_Access;
 
Surely this will create a 'new' instance of the list - if so that would be the last thing I probably want to do.
 
Surely there must be another way....I'm sure I'm overlooking the obvious, and I just can't find the right solution in the wealth of knowledge that I've been trawling through.
 
Can someone point me in the right direction? I thank you for your time and trouble.
 
Kind Regards,
 
Aero
Posted 20-Jun-12 3:31am
Aero72593
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 1

A static collection is a typical source of a "memory leak by design". In .NET, you are more or less protected from the casual memory leaks, because you don't have to clean up memory after every operation you use it; the Garbage Collector (GB) takes care about it. Despite of the common believe, which is a mistake, it does not mean .NET absolutely protects you from memory leaks. It depends on what do you call a "memory leak", but if you can easily organize degradation related to consuming more and more memory in some program life cycle, when you don't give GC a chance to collect functionally unused memory, even if you only use managed memory, no unmanaged (which could be a separate source of memory leaks).
 
How can it be possible? Imagine you have some collection with the life time about the life time of your application. It often happens to the static collections (apparently all static objects live through the whole process life time), but it can be an instance field/property or even the a variable, if you have a function running beyond main life cycle, such as main UI application event loop. The problem appears when your application has some big life cycle repeated many times (which is always the case with UI applications) and when you use some collection which is used again and again beyond one cycle. However, this is not enough to create a leak. The leak happens when you add some objects the the collection which you don't remove by the next cycle, if you do it on a regular basis in at least some cycles. If you allow for such additions to the collection, you effectively block the objects from being garbage-collected.
 
You should understand, the GC can only collect objects which are rendered inaccessible by the currently executing code in your application domain. This is called reachability. Please see:
http://en.wikipedia.org/wiki/Garbage_collection_%28computer_science%29[^],
http://en.wikipedia.org/wiki/Garbage_collection_%28computer_science%29#Reachability_of_an_object[^].
 
In most cases, you should get rid of static collections. From your information, I cannot see why would you think your collection had to be static; in most cases, it is not required. Major concern about static collections is threading, not so much of the danger of the memory leaks. Besides, turning the static collection into an instance field/property does not provide 100% guarantee (but usually should be done and often can solve the problem). Rather, you should guarantee clean up of the collection in each cycle or "saturation", the situation when a collection is populated but stop growing after certain point. (One example is some information obtained from reflection; if you don't allow duplicates, you collect all the reflected data you have in the application domain and the collection get saturated.)
 
One alternative approach is using weak references instead of "regular" references, in case of reference objects stored in the collection. GC can still collect the objects accessible by weak references. Please see the class System.WeakReference:
http://msdn.microsoft.com/en-us/library/system.weakreference.aspx[^].
 
See also:
http://msdn.microsoft.com/en-us/library/ms404247.aspx[^].
 
With this approach, you should take care to prevent de-referencing of an object which has been garbage-collected.
 
Again, your information is not enough to give you the final recipe in your particular case, but I explained the principles you can utilize to analyze your application process and prevent memory leaks.
 
—SA
  Permalink  
v6
Comments
losmac at 20-Jun-12 16:48pm
   
GREAT answer, my 5!
Sergey Alexandrovich Kryukov at 20-Jun-12 16:55pm
   
Thank you, Maciej.
--SA
Aero72 at 20-Jun-12 18:54pm
   
Many thanks again, SA - lots of reading to do and code to re-configure. I will get back to you. The tip on WeakReference is potentially very interesting.
Sergey Alexandrovich Kryukov at 20-Jun-12 19:03pm
   
You are welcome. I would expect you also accept this answer formally (green button), even of your problem appears to be somewhat different. I gave an advice based on the information I can see -- thanks.
--SA
Aero72 at 21-Jun-12 15:11pm
   
No worries, SA - answer rated 5 and accepted. Got myself a trial of a memory profiler. I was blind - now I'm learning how to see.
Sergey Alexandrovich Kryukov at 29-Jun-12 20:59pm
   
Congratulations...
--SA
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

What sort of data are you storing in your list? I have used (in testing) extremely large lists without issue.
 
The size of any item in .NET is limited to 2GB. The maximum value for a lists capacity is 2147483647 so theoretically you could store that many bytes in the list and be right on the limit (you would need to set Capacity first or you would run into troubles sooner), or 1/4 that number of ints, or 1/4 (1/8 on 64bit) that number of class instances (IntPtr size as reference type).
 
In practice I would never go anywhere near these limits but several hundred thousand items should cause no issue unless each item was a large struct.
 
I don't think your OutOfMemoryExceptions are due to public static, although it's a good idea to change that anyway IMO.
  Permalink  
Comments
losmac at 20-Jun-12 16:51pm
   
Nice explanation! +5
Sergey Alexandrovich Kryukov at 20-Jun-12 16:57pm
   
This is a good advice, I voted 5.
As to the out of memory, it can be indirectly related to static (no matter public or not) collections, because it makes possibility of memory leak more likely. It depends on what the code actually does, of course. I explain it in my answer, please see.
--SA
Aero72 at 20-Jun-12 18:42pm
   
Thanks for your input - in answer to the questions you raise, the list type is a class which defines a float, two bytes, a UInt32 and a UInt64 - i.e. 18 bytes of data per instance in the list - so as you say I should be able to have several million items in this list without a problem.
 
As SA suspects there must be a leak. During one test run this pm, I calculated that the list should have occupied around 300MB, but overall memory usage for the program according to taskmgr (yes, I know I should look at using a memory profiler - that's tomorrow's job) was > 1.3GB!
Sergey Alexandrovich Kryukov at 20-Jun-12 18:57pm
   
Unfortunately, the task manager estimate is not accurate. Nevertheless, it's a certain sign of a leak, looks like. I would analyze the overall code design and/or tried to monitor adding and removing data via logging of diagnostic information, in a few strategic points. In most cases, it helps to find ends. A memory profiler is a pretty powerful option, of course.
--SA
DaveyM69 at 21-Jun-12 6:35am
   
If the List is a list of class instances then each element will just hold a reference (4 bytes 32bit, 8 bytes 64bit). The overall memory used by the app will increase by the the size of all instances combined, but the List object itself will approximately be IntPtr.Size * Count.
Aero72 at 21-Jun-12 12:04pm
   
Thanks, Davey....so if I understand you correctly, the list is effectively an array of pointers, and it still has to put the data somewhere...so on my x32 machine, the 18 bytes of data per class instance is effectively 22 bytes of required storage when you take into account it has to remember where it put the data in memory.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 Sergey Alexandrovich Kryukov 509
1 Nirav Prabtani 298
2 OriginalGriff 215
3 _Amy 165
4 Abhinav S 160
0 OriginalGriff 7,715
1 Sergey Alexandrovich Kryukov 6,911
2 Maciej Los 3,954
3 Peter Leow 3,698
4 CHill60 2,742


Advertise | Privacy | Mobile
Web03 | 2.8.140721.1 | Last Updated 20 Jun 2012
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100