|
Hi i want to fetch data from a website that which has 1000's of pages and put it in a excel file.
how should i go about doing it?
|
|
|
|
|
ivix4u wrote: how should i go about doing it?
Well, I would start off by defining my requirements and then moving onto producing a fairly detailed breakdown. At this stage, I'd feel ready to fire up Visual Studio and start off a new project. From there on, it's just code.
|
|
|
|
|
I might even throw in a few Visio pictures to help with the user review of the spec
Bob
Ashfield Consultants Ltd
|
|
|
|
|
|
|
Running the garbage collector would help if you have disposed memory that you are no longer using.
If all 25 files are being stored in the same datatable, the garbage collector won't shrink the size of that table. It would however free any intermediate storage you used while reading the data.
If your data is mostly strings using the standard ASCII characters (0-127), you might be able to cut your memory use in half by using the Unicode UTF-8 encoding or by using bytes rather than chars. UTF-8 uses one byte for ASCII characters, and only uses more than one byte for non-ASCII.
|
|
|
|
|
Alan Balkany wrote: Running the garbage collector would help if you have disposed memory that you are no longer using.
No, not true.
The garbage collector will automatically run when the system gets low on memory. You do not need to call it manually. Forcing a garbage collection is unnecessary - just trust the system to do it.
Disposal is merely running an objects dispose() methods to release unmanaged memory. It has nothing to do with managed memory or the garbage collector.
Your problem is likely due to the fact you are incorrectly holding onto a reference to something that you should have released, causing the object to remain in memory and survive garbage collection. Use the sysinternals tool "process explorer" to examine your app. Look at the .net memory information it provides. Check the heap sizes of all 3 generations. I would suspect you'll find a lot of objects are surviving to generation 2.
These two articles may help you in tracking down the leak:
http://blogs.msdn.com/ricom/archive/2004/12/10/279612.aspx[^]
http://www.codeproject.com/KB/dotnet/Memory_Leak_Detection.aspx[^]
Alternatively, like others have said, you may actually be just trying to hold too much in memory. Try doing it in stages, 1 file at a time, then clearing out memory and starting with the next file.
Simon
|
|
|
|
|
I doubt you would need all that data in memory at once; what are you trying to do?
|
|
|
|
|
Forcing a Garbage Collection is a bad idea and will not solve your problem anyway.
The problem is simple. How much data is in these records and why are you holding 25,000,000 records in a data table?? Just holding 25 million Integers takes a minimum of 100 MB of memory, so if there is more in the table, you ARE just running the machine out of memory. And that's not including the overhead of the DataTable itself.
The answer is simple. Don't try and hold millions of records in memory.
|
|
|
|
|
Are you on Vista? Check the process memory limit and increase if necessary.
xacc.ide - now with TabsToSpaces support IronScheme - 1.0 alpha 4a out now (29 May 2008) ((lambda (x) `((lambda (x) ,x) ',x)) '`((lambda (x) ,x) ',x))
|
|
|
|
|
|
Are you ultimately saving the DataTable into a database server? Do you have to make some correlation between the files or lines? If not, I would consider streaming the file contents into the database directly, into small chunks, like 10000 line at a time.
I have had a similar problem and I used the SqlBulkCopy class and a simple implementation of IDataReader that was specialized into reading my files.
Another alternative would be SSIS.
|
|
|
|
|
Hi,
I am working on a C# application. I have a windows form that has some richtext box, some images, buttons etc. I have created this application on my dektop PC. But when I run this on any other Laptop or tablet PC, the form doesn't get scaled properly.
Also, if I maximize it then also the contents in the form should be scaled but it doesn't. Kindly assist.
Cheers!!
Ethen
|
|
|
|
|
|
i have created the Crystal Report but when i try to run , it request me to enter logn creditentials
Error message read: This report reqiure further information
How do we solve these kind of error
|
|
|
|
|
you, uhm, enter login credentials presumably!
|
|
|
|
|
This is actually a PITA problem with Crystal Reports and a good question for novices of the tool.
Need software developed? Offering C# development all over the United States, ERL GLOBAL, Inc is the only call you will have to make.
Happiness in intelligent people is the rarest thing I know. -- Ernest Hemingway
Most of this sig is for Google, not ego.
|
|
|
|
|
Crystal is a PITA, call refresh database (I can't remember the name off the top of my head) before programatically using the report. If that doesn't work you can, in-code, assign the username and password before calling refresh ... maybe it is called verify database?
Need software developed? Offering C# development all over the United States, ERL GLOBAL, Inc is the only call you will have to make.
Happiness in intelligent people is the rarest thing I know. -- Ernest Hemingway
Most of this sig is for Google, not ego.
|
|
|
|
|
Generally using c#, how to read font files(.ttf) to know the font metrics present in the file for a string.........
plz help......
|
|
|
|
|
You have asked this 4 times now...
Just load the font and look at the metrics provided but the Font/FontFamily class...
xacc.ide - now with TabsToSpaces support IronScheme - 1.0 alpha 4a out now (29 May 2008) ((lambda (x) `((lambda (x) ,x) ',x)) '`((lambda (x) ,x) ',x))
|
|
|
|
|
You are going to get no help in the future doing what you are doing.
Blog link to be reinstated at a later date.
|
|
|
|
|
can anyone help me to understand the Difference between asynchronous and synchronous function with small code snippet.
Thanks
tbhattacharjee
|
|
|
|
|
asynchronous[^]
synchronous/synchronization[^]
No code snippet required really, an async function will normally return immediately and continue doing it's thing afterwards (on a seperate thread?) so the current thread is not blocked. A sync function will return at the end of the function, so the current thread is blocked while the function is running.
DaveBTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)Visual Basic is not used by normal people so we're not covering it here. (Uncyclopedia)
|
|
|
|
|
The main diffrence in Sync and Async function is "In Sync function: one has to wait till the function returnsthe control after exececution.In Async once one send the call to function one has not wait for the result during this one can do another task,when function execution gets completed it returns the value." For more see the link
http://msdn.microsoft.com/en-us/magazine/cc301332.aspx[^]
Cheers!!
Brij
|
|
|
|
|
well not much, but if is synchronous starts a work and after finished stars the next one, if is asynchronous can start many at same time.
If is asynchronous will not be stuck somehere and deals many calls at same time. (needed in chat applications)
nelsonpaixao@yahoo.com.br
trying to help & get help
|
|
|
|