|
Thank you for your answer.Yes,it does work!!
|
|
|
|
|
You're welcome.
|
|
|
|
|
Dear all,
I need a way to read a 500MB large TEXT file containing a url on each line and remove duplicates and then write data back on the text file. I know it sounds simple but here i need my algorithm to be as fast as possible. I don't want to design a special data structure for this purpose i just want to use C# pure language to do this.
Here i have some question in mind that i never found a good answer before:
1.When you use
string s = File.ReadAllText(filename); the exception 'out of memory' will thrown (Because the amount of data is too big). So we should read the file line by line. dose reading line by line affect the speed? isn't it better to read the whole file and then do the processing?
2.What is the fastest data structure in c# to access data on it. i mean between hashtable list String[] array...
3.Because it's the matter of speed, should i think of writing data back to a database after reading from file? is it faster than writing directly back to file? (like inserting data into MySQL server with an index on my url field with UNIQUE type, so by inserting the data it will take care of duplicates)
I know my question is a bit big..  but if you have any idea in any part i will be glad to know your precious opinions.
Thank you.
|
|
|
|
|
Hi,
some questions:
1. do you need to preserve the order of the URLs?
2. what is going to happen with the file with unique URLs afterwards?
3. can you show say 10 different URL to show how different they are?
4. what is generating the input file? can that code be slightly modified to gain a lot of speed afterwards?
|
|
|
|
|
1. Theres no need to preserve the order of urls, but if you want to sort the urls and then remove the adjacent identical records it's not doable here because i want to remove urls with same domain names also.
here is the sample of urls list:
http://iri.siam108site.com/dssff/
http://ieggfiv.300mb.info/12hf8485n/
http://forum.nto.pl/chlopcy-z-biskupic-mieli-sie-bic-tymczasem-jeden-zginal-t19232.html
http://www.osiol.com.pl/uwaga-tvn-2007-2008-tvrip-t26379.html
http://fdss.300mb.info/12hf8485n/
http://perpetuum-lab.com.hr/forum/showtag-physiology.html
http://hrbikwi.siam108site.com/1i0gejcj9j6
So http://fdss.300mb.info/12hf8485n/ and http://hrbikwi.siam108site.com/1i0gejcj9j6 should be removed from the list.
2. A web crawler will crawl urls to grab page contents
4. I wrote a program that is grabbing these urls from search engines and then it writes the urls into at the end of a text file (Without any kind of processing and checking on data)
Sorry for the late response it was 5 am when i asked the question :p
modified on Thursday, April 30, 2009 4:34 AM
|
|
|
|
|
OK,
1.
Generating a huge file, then search for duplicates, is a waste of time. If the code that generates such list can be modified, modifying it will always be a much better approach.
It will take less code, less effort to develop, and less CPU cylces to execute.
2.
One way to reduce the postprocessing effort (but not code) is by generating not one but many files, i.e. by applying some sorting. Say you calculate an 8-bit hash value on the URLs, and sort them accordingly in 256 different files. That is easy, and reduces your postprocessing to 256 much smaller jobs (average file size now is 2MB, and now you can do a ReadAllLines).
3.
The ultimate solution is to avoid duplicates right away.
I have written web crawlers before; obviously they need a way to avoid visiting the same URL twice, so they contain some logic for that.
Furthermore, you typically want to search breadth-first, i.e. read and process all of a page before switching to another page; that is more efficient than depth-first, since in the latter you have to either keep in memory or refetch the content of all the pages you are encountering while digging deeper.
The natural solution to these two issues is to organize the crawler around a list containing all the URLs you have encountered (and should visit). Every anchor, image source, style sheet encountered gets added, if not already in the list; when a page is done, move to the next in the list (use an index). When the index gets to the end of the list, you are done, and the list now holds all the unique URLs.
|
|
|
|
|
Thank you for your solution. I like the idea of 256 file and checksum. I will try it now.
|
|
|
|
|
To make it fairly simple, us a generic dictionary where each url key and an entry into the database. Read each line of the textfile one at a time and verify that the key doesn't exist if it doesn't add it to the dictionary. If the key does exist then ignore it. When you are all done use the foreach to go through each key writing it to a file as text.
Something like
Dictionary<string, string=""> Urls = new Dictionary<string, string="">();
while(TextReader.Read())
{
string key = TextReader.ReadLline();
if(!Urls.ContainsKey(key))
{
Urls.Add(key,key);
}
}
foreach(string s in Urls.Keys)
{
TextWriter.WriteLine(s);
}
Obviously you need to do all of the opening and closing of files, but this is probably the simplest way.
|
|
|
|
|
|
I think the easiest way to do this would be to provide your own Import/Export settings UI and handle the loading of data and saving to settings/saving to export file in your code. It should be fairly trivial.
|
|
|
|
|
I don't know if this will help you, but take a look at this CP Article[^]. I have found it useful and it contains some links to other articles/resources which may assist.
|
|
|
|
|
Hi,
I have a simple solution with a windows client and a website. The website exposes a web service that the windows client consumes. So far so good. Here's the weird part. When running my solution in VS, the call to the web service doesn't work when I hit CTRL + F5 unless I run it with a simple F5 first. So each time I open the solution in a new VS session, I have to run it in debug (F5) at least once before it works without debugging (CTRL + F5).
This makes no sense to me. Has anyone else run into an issue like this? Any ideas would be greatly appreciated!
Thanks!
Ian
|
|
|
|
|
It should respond with a (recompile if neccesary) during <CTRL>+<F5>. Have you checked/resetted your keymappings?
|
|
|
|
|
The key mappings are fine. Again, the application runs with a CTRL + F5 - it just doesn't work unless I've already run it with a simple F5. I wondering if it's a VS bug? I'm running it in a VM on a 64 bit test environment...
|
|
|
|
|
This is a very trivial question, but I have not had much chance in finding a solution. I have searched Google / blogs, but apparantly either there is no solution or I searched for the wrong keyword. Any help will be greatly appreciated.
I have an application (similar to a wizard) and I need to validate whether the user has the permission to write/read from a particular location.
I the following function (below), but I am always getting the value "true" eventhough from Windows Explorer when I try to navigate to the location, I'm always getting an error (eg. "You need permission to perform this action. [Retry] [Cancel]").
static bool CanReadAndWrite(string path)
{
var perm = new FileIOPermission(
FileIOPermissionAccess.Write &
FileIOPermissionAccess.Read &
FileIOPermissionAccess.PathDiscovery,
path);
try
{
perm.Demand();
return true;
}
catch
{
return false;
}
}
Many of the blogs/articles suggest that I should attemt to write to the location anyways, and if the write operation is successful, then I can proceed to write the main data. If there is an exception creating the stram then there is a perimission issue.
But this solution would NOT work for me. The application has two "pages".
- First wizard page: user selects the file path
- Second wizard page: specify options
- Third wizard page: write data
Each of the pages has Cancel button and the expectation is that user NOT be able to continue to the 2nd page if permission is denied. The above suggested solution would WORK fine if the user does not have permission. But if the user DOES have permission, then the validation check in Page # 1 would have created new file and/or overwrote an existing file with 0 bytes -- and if user cancels on Page #2 then there is a useless file on the file system.
So anyways, what should I do (clean implementation) to check for file permision?
I'm using Vista, .Net 3.5.
Thanks,
- Malhar
|
|
|
|
|
Create the temporary file to test the permissions and then immediately delete it. Whether the user continues the process or cancels it the file isn't needed at all after you've verified the permissions of the selected path.
|
|
|
|
|
Shouldn't you use an OR operator ('|') to request all those permissions?
I guess the AND result is plain zero.
|
|
|
|
|
Sorry for overlooking the detail when I pasted the code. But my original attempt included "|". I just tried now instantiating "perm" object like below:
var perm = new FileIOPermission(FileIOPermissionAccess.Write, directory);
But still no luck.
- Malhar
|
|
|
|
|
Hello,
I'm building my first web service in C#. I have met most of my needs for this service so far up to sending files. If this needs to go to another section. Please let me know which section to post in.
The basic steps of the service are:
1. receive an xml file through asp.net page
2. read xml and save to database.
3. read xml and retrieve file nodes and location for other files.
4. upload other files to directory on server.
This last step is where I need a tutorial or some suggestions.
As it stands I would just like to pull the files from client in some fashion after reading the xml file information. If anybody has a link to an article that would be great. I have looked at the Mtom solution but that is not viable for a web application.
Thanks to anybody that responds.
chuck
|
|
|
|
|
|
Have a little patience, you have only been waiting 4 hours. Instant answers cost money. 
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
|
I think I found something[^]  FOR FREE
And also another thing[^]  FOR 12,95 $/Month
To view the Experts-Exchange solutions for free use the techinique described in this web site[^].
modified on Friday, May 1, 2009 5:49 AM
|
|
|
|
|
I don't know why pictures aren't shown in listview.. i would like to add them like thumbnails..or small icons..so when i will click on them they will resize in picturebox.. Here is code for adding to listview which isn't working.
private void Form1_Load(object sender, EventArgs e)
{
DirectoryInfo dir = new DirectoryInfo(@"d:\\feri\\ičr\\galerija\\galerija\\galerija");
foreach (FileInfo file in dir.GetFiles())
{
try
{
this.imageList1.Images.Add(Image.FromFile(file.FullName));
}
catch
{
Console.WriteLine("This is not an image file");
}
}
this.listView1.View = View.LargeIcon;
this.imageList1.ImageSize = new Size(32, 32);
this.listView1.LargeImageList = this.imageList1;
for (int j = 0; j < this.imageList1.Images.Count; j++)
{
ListViewItem item = new ListViewItem();
item.ImageIndex = j;
this.listView1.Items.Add(item);
}
}
|
|
|
|
|
Speaking purely from memory (can't get to VS to test at the moment) resizing an ImageList clears it. Your code resizes the ImageList after loading the images.
Try moving the resize to be before loading the images.
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
