|
blackjack2150 wrote:
And another related question. It helps in terms of performance to write as much data as possible at a time, doesn't it?
kind of depends what you mean by performance. but often yes.
1. The user starts the file save procedure and starts looking at email. The PC will look faster if your process is slightly less fast so smaller packets that don't block the disk queue might help.
2. The hard disk has issues. The program continually tries to write and fails. Because your packets are huge it fails more often as it's hard to find unbroken space that big and it has to attempt each write many times. (This would more likely be a problem on a network share for instance).
3. The hard disk is fast, is 0% fragmented and there's more ram and CPU power in the box than you could shake a stick at. This machine isn't going to slow down for anything so you might as well max the packet size and get the job done fast.
Do you need this file save to complete before the user continues? If not i would suggest sticking the save process in a background thread and maybe put a progress bar in the UI.
Russ
|
|
|
|
|
User interaction is not important for me. In fact is almost absent. I've converted my program to a console application, as I noticed that during the execution the UI got blocked. I know there are ways to avoid this, but I decided that it's not worth the effort.
If it helps I'll briefly describe what the program does:
I have a folder containing more than 7000 xml files which are news articles. I must read everyone of them and write to a file every distinct word in these xmls. It adds up to about 42000 words. I another file I must write the frequency of appearance of every word in every xml file, so there are 42000*7000 = a s...load of numbers to be written.
So, the user just starts the program, sits back and begins to read a book on programming. 
It takes about 5 minutes to read the files and store the words in memory, another 5 to write the 42000 words to the first file, and writing the last file....well God knows, as I never had the patience to wait that much.
This file should have: (7000 * 42000 * 2) / 1024 /1024 ~= 560 MB.  
The data structures I use are just basic arrays and matrixes. Does the word "matrixes" exist? Or should it be matrices?
So, what do you suggest?
|
|
|
|
|
Hi
well that does not sound like a I/O Performance problem. writing to streams is pretty fast.
i have an application that writes every day a 70GB Zip - File and it doesnt take more than 4 hours. and it gets slower the bigger the file gets. the first gb takes about 15 minutes.
is the whole data processing finished when your app writes the first file?
if not, i would say, the creation of the data used for your second file takes so long and not the write process of it..
greets
m@u
|
|
|
|
|
It seems to me that you can write the first file as you go, unless you want to write them alphabetically or something.
If you have a reasonable idea of the maximum number of digits for the frequency, you could write the second file as you go as well, just keep a Dictionary containing the word, the current count, and the offset to the position of the frequency.
blackjack2150 wrote: The data structures I use are just basic arrays and matrixes
I recommend a Dictionary.
blackjack2150 wrote: matrices
That is correct.
|
|
|
|
|
I mocked up a quick perf. test. You can run it yourself to see the results (in VS 2005).
Just make an empty console app. and past this code:
class Program<br />
{<br />
static string printme =<br />
"Fastest way to write a text file Fastest way to write a text file Fastest way to write a text file";<br />
<br />
static void Main(string[] args)<br />
{<br />
<br />
Stopwatch sw = new Stopwatch();<br />
sw.Start();<br />
<br />
<br />
test1();<br />
<br />
<br />
sw.Stop();<br />
Console.WriteLine(sw.Elapsed);<br />
<br />
<br />
sw.Reset();<br />
<br />
<br />
sw.Start();<br />
<br />
<br />
test2();<br />
<br />
<br />
sw.Stop();<br />
Console.WriteLine(sw.Elapsed);<br />
<br />
<br />
Console.Read();<br />
}<br />
<br />
<br />
static void test1()<br />
{<br />
TextWriter tw = new StreamWriter("test1.txt");<br />
for (int i = 0; i < 50000; i++)<br />
{<br />
tw.WriteLine(i + " " + printme);<br />
}<br />
tw.Close();<br />
}<br />
<br />
static void test2()<br />
{<br />
for (int i = 0; i < 50000; i++)<br />
{<br />
File.AppendAllText("test2.txt",i + " " + printme + "\n\r");<br />
}<br />
}<br />
}
I hope my logic is right, and if it is, you will hopefulyl see the same result as me. Streamwriter completes in a few seconds, and File.AppendAllText... takes forever.
There is also AppendText, but that returns a streamwriter and gives you the same result as test1.
Hope I helped.
public object BufferOverFlow<br />
{<br />
__get { return BufferOverFlow; }<br />
__set { BufferOverFlow = value; }<br />
}
|
|
|
|
|
blackjack2150 wrote: What is the fastest way to do it? Using the static methods of the "File" class or using streams?? I am aware that either way, it'll take a long, long tine, but ...time is precious.
I don't think that there is such a big difference. Most of the static methods in the File class are using streams anyway, so then the difference is just how you use the stream.
There might be a difference in writing a text stream, and encoding the text first and writing it as a binary stream, so that might be worth testing.
blackjack2150 wrote: It helps in terms of performance to write as much data as possible at a time, doesn't it?
That depends. The difference is hardly noticable unless the buffer size is very small or very big.
You should be aware of the large objects heap, though. Any object that is larger than 85 kb is allocated in the largs objects heap instead of the regular heap. The large objects heap grows when needed but never shrinks, so if you allocate a lot of large objects the program may allocate a lot of memory that is not returned to the system until the program ends.
---
single minded; short sighted; long gone;
|
|
|
|
|
Guffa wrote: There might be a difference in writing a text stream, and encoding the text first and writing it as a binary stream, so that might be worth testing.
Here is some code. Again, make a console app and paste this testcode. A Bytestream is obviously faster, but I can't seem to debug it properly (it generates a character that shouldn't be there at the beginning of each write, so the file is bigger).
<br />
static string printme =<br />
"Fastest way to write a text file Fastest way to write a text file Fastest way to write a text file";<br />
<br />
private static int linecount = 50000;<br />
<br />
static void Main(string[] args)<br />
{<br />
<br />
Stopwatch sw = new Stopwatch();<br />
sw.Start();<br />
<br />
<br />
test1();<br />
<br />
<br />
sw.Stop();<br />
Console.WriteLine(sw.Elapsed);<br />
<br />
<br />
sw.Reset();<br />
<br />
<br />
sw.Start();<br />
<br />
<br />
test2();<br />
<br />
<br />
sw.Stop();<br />
Console.WriteLine(sw.Elapsed);<br />
<br />
<br />
Console.Read();<br />
}<br />
<br />
<br />
static void test1()<br />
{<br />
TextWriter tw = new StreamWriter("test1.txt");<br />
for (int i = 0; i < linecount; i++)<br />
{<br />
tw.WriteLine(i + " " + printme);<br />
}<br />
tw.Close();<br />
}<br />
<br />
static void test2()<br />
{<br />
FileStream fs = File.Create("test2.txt");<br />
BinaryWriter bw = new BinaryWriter(fs);<br />
for (int i = 0; i < linecount; i++)<br />
{<br />
string toprint = i + " " + printme + "\r\n";<br />
bw.Write(toprint);<br />
}<br />
}<br />
}<br />
public object BufferOverFlow<br />
{<br />
__get { return BufferOverFlow; }<br />
__set { BufferOverFlow = value; }<br />
}
|
|
|
|
|
A BinaryWriter is not used to write text to a binary file, it's used to write data in binary format to a binary file. When you write a string using a BinaryWriter, it writes the length of the string, then the binary representation of the string. The string is not encoded either, so it will be in UTF-16, which is the internal format for strings.
You don't need a writer to write a binary file. A FileStream can do that.
static void test2() {
using(FileStream fs = File.Create("test2.txt")) {
for (int i = 0; i < linecount; i++) {
string toprint = i.ToString() + " " + printme + "\r\n";
byte[] buffer = Encoding.UTF8.GetBytes(toprint);
bw.Write(buffer, 0, buffer.Length);
}
}
}
---
single minded; short sighted; long gone;
|
|
|
|
|
I just discoverd that my quick test isn't very conclusive. It should be run in a closed environment, no influences from other programs, and probably run a 100 times before you can draw any real conclusions. 
I think using the filestream or textwriter is the fastest, can't see any difference.
Also I forgot about the using clause, it's very important to dispose of writers and streams. Thanks 
Visual Studio can't evaluate this, can you?
public object moo<br />
{<br />
__get { return moo; }<br />
__set { moo = value; }<br />
}
|
|
|
|
|
Hi,
I want to use radioButton in Each Parent Node of TreeView.
How can i do that. if any one knows please help me.
With Thanks,
sakthi.
|
|
|
|
|
|
hello
i want to use System DataGrid in pocket pc 2003 with checkbox feature
please help me
raj
code
|
|
|
|
|
I have the database already with the tables.I want the create query of each table.
Is it possible to get the query through c#.or is it possible in msaccess?
|
|
|
|
|
HI,
You can use Msaccess for generating query
go to Queries->Create query by using wizard
You can select the desirable tables for the queries
My small attempt...
|
|
|
|
|
thank u for u r reply.
Actually i have the tables already. your solution gives me the select query for the table but not the "create query". i want create query of the existing table.
is it posible to generate (create Query) query as we generate in sql server.
|
|
|
|
|
I think your question is not clear
do you want to create table using query?
What you mean by "Create query" ?
My small attempt...
|
|
|
|
|
I have already created the tables.but i need create query of the existing table.
|
|
|
|
|
i have the tables already.
i want the create table syntax of the existing table.How can i see?
say for example sample is the table that i have created already.
now i want the syntax like this
create table sample(.....);
Where can i go and find this systax in access.
If the search i could get the syntax like this only(Select .......)
But i want the create table syntax.
|
|
|
|
|
i think you need a function called createquerydef() on the access database object.
Russ
|
|
|
|
|
createquerydef()
how to get this function i didn't get u r idea.
can u give more explanation. is function available in MsAcess or available in .net
|
|
|
|
|
it's a function that is part of the access runtime library. I belive there is a Primary Interop Assembly available for MS Access so you should be able to use the object from c#.
have a look on MSDN for Access PIA and you should find a download.
Russ
|
|
|
|
|
he means he wants something like
"Create table mytable ID int Not null, somevalue varchar2(10), " etc.
V.
I found a living worth working for, but haven't found work worth living for.
|
|
|
|
|
Hi all,
Im kinda starter in C#.net, im trying to build a system, first thing's to establish a connection to an MSAccess DB, I want to make my connection public so all forms can use it by just calling it. I'm trying to work on creating a class for it but im having hardtime making it work...
How can I make a public connection in a class and call it in my forms?
Thanks in advance
-::maximus::-
|
|
|
|
|
Try to make a class like "DatabaseManager" to do all the databaseoperations
Please make all the methods as public so that all the classes can access them
You can use the following public methods
1.OpenConnection()
2.CloseConnection()
3.ExcecuteQuery(string query)
4 ExecuteQuery(string query,out Dataset dt)
Just call Excecute query function with a query,and you can managethe open and close connection from this function.
If you expect any output(like search) then you can use the overloaded function
which gives a dataset.
My small attempt...
|
|
|
|
|
Hi,
the idea with one class for the hole project is a good one.
You can create a oleDbDataAdpater with the Windows Forms generator.
Then you copy the source and put it into a new class, which is created by your own. Set the modifier to public and now you can use your class in the hole project.
regards
spotl
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
