|
Hello everyone,
I'm working on a small hobby project. I'm making a web crawler for a particular website to extract some useful information from it. I've written information extraction algorithms but I'm completely new to HTTP response stuff. I'm using a HttpWebResponse class of .Net to get the source code of a webpage in form of StreamReader. Now, I'm wondering if I should process each line of stream or I should convert whole streamReader to a string and then process that. Let say, I'm looking for string that should be able to extract .mp3 links on webpage using regex then is it possible that a single link is on two differnt lines of StreamReader, look at the code below..
string url = "http://songs.pk";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
string line;
while ((line = sr.ReadLine()) != null)
{
Console.Read();
Console.WriteLine(line);
}
Now can I add some regex matching code here..will this code make sure that it will be able to extract all the .mp3 links from this website? Or should I convert the StreamReader to string and then use regex matching on that string?
I'm sorry, if I'm misunderstanding about StreamReader here, but I need some suggestions to parse the source code of a website. I've searched articles and Google but I'm unable to find something that could help me. Any articles, links or suggestions would be appreciated.
Thanks
Shivam Kalra
|
|
|
|
|
Your code is fine. You will read the HTML code line-by-line from the page and you can process the whole html page line by line. This is a pretty common scenario. You can use a regex to determine if the value from the line-field contains what you are searching for  .
|
|
|
|
|
I would read the whole stream (to get the full page content) and use your regexes on that. The streaming is very useful for large downloads, but a web page is easily small enough to work on in memory. StreamReader.ReadToEnd is the easiest way to do that.
|
|
|
|
|
You could also use the HTMLDocument, IHTMLElementCollection, and IHTMLElement to read each tag found in the response and grab the innerHTML if it's a tag you're interested in.
|
|
|
|
|
Is it possible to call a Web Service from C# without first adding a reference to the WS in your project? All the examples I see require a reference. An example would be great.
Thanks
Everything makes sense in someone's mind
|
|
|
|
|
|
|
Hello,
Sorry, if its the wrong place to post. I have been doing Embedded Systems (lower level) development on Linux/Windows, in Generic C++ and Visual C++ for the last 5 years or so. I am trying to switch my focus to the new stuff as far as Web applications and latest stuff out there. I would like to do Web Services (middle and Back end not so much web site development). I have started learning C# (winforms) which isn't much different from C++, I guess I can pick up WPF as well. Currently, I am working in robotics and automations which is database driven development, may be its one step closer to what I need.
Can you please give me good tips of how you might approach this? and what topics to really focus on so I might need(may be ASP.net, WCF, DataBases...etc) that can make me a lot more marketable for employers?
Thanks
|
|
|
|
|
Software2007 wrote: Sorry, if its the wrong place to post.
It is, and there's really no excuse for it. You can obviously read, which means you should have EASILY noticed the forum Titled "Web Development" in the menu you had to use to get to THIS forum.
To answer your question, the only real way to learn a new technology is to actually use that technology, and always remember - google is your friend. Create a VERY simple web service and then write a bunch of different clients to connect to it. I'd also put some time into learning about WCF.
Part of writing web services is correctly deploying them, and using them from a remote machine. This means you're going to need another box entirely, or a single machine with enough juevos to run a VM running IIS alongside your own native development deskop. I'm lucky enough to have a 2nd box so I don't have to mess with the sometimes messy bridging of netowrk adapters necessary to be able to communicate with the OS on the virtual desktop.
WPF isn't a back-end technology, and requires the same knowledge to hook an app up to a web service. However, Silverlight is a completely different animal, and it requires still more special knowledge to get it working with a web service.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass." - Dale Earnhardt, 1997
|
|
|
|
|
Thanks for the reply. I am still not too familiar with the architecture, so in a banking example, where clients log on to ask for balances and review account summary, what would be the wcf service, clients? Will the clients be any wpf/winform apps hooking up with the wcf service?
Is this a good example for me to emulate so I can pick up how these thing work?
Thanks
|
|
|
|
|
ok so in a classic web service situation you have an API that you can talk to from client apps (be they web site front ends, mobile apps, desktop apps, or whatever)
the API will be a REST API type thing (basically means you can call it via a URI like a web page) and typically you will get back either XML or JSON (depending on what problem domain you are dealing with)
so you would write essentially a web site that has no face and doesn't output any html as such ... and then you implement the clients who make requests (like custom mini browsers effectively) and get responses in whatever you decided to use (XML,JSON,etc) ... then your clients have to decide how to cache / display / save the data returned from the web service
this is a very simplistic overview but hopefully it isn't insulting or too stupid to be useful as a starting point
"mostly watching the human race is like watching dogs watch tv ... they see the pictures move but the meaning escapes them"
|
|
|
|
|
Simplistic, yes, but insulting it isn't. It's an excellent minimalist recommendation for implementation that makes perfect sense. Very efficient! 
Will Rogers never met me.
|
|
|
|
|
John Simmons / outlaw programmer wrote: Silverlight is a completely different animal, and it requires still more special knowledge to get it working with a web service.
Just for the record, both WPF and Silverlight are client-end technologies and are pretty much identical in how they work with web services, aside from some limitations in Silverlight with its smaller .NET runtime...
Mark Salsbery
Microsoft MVP - Visual C++
![Java | [Coffee]](https://codeproject.freetls.fastly.net/script/Forums/Images/coffee.gif)
|
|
|
|
|
Silverlight requires "special knowledge" if you're working on a secure site. It took me a couple of weeks to figure how how to put that jigsaw puzzle together...
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass." - Dale Earnhardt, 1997
|
|
|
|
|
I was very reluctant to start learning ASP .NET because I regarded coding in HTML as awful and disgusting. But I have switched from WinForms to WPF recently and I found it a right move. Since in WPF you code XML, it was easier for me to create my first ASP page which happened two weeks ago or so.
Greetings - Jacek
|
|
|
|
|
Great. I am not sure if I do need to learn ASP.net or any xml/html coding, since I do not want to do front end website design. I am mostly interested in the middle and backend of web development. Do you think I need to know ASP.net?
|
|
|
|
|
Software2007 wrote: Do you think I need to know ASP.net?
IMHO yes -- to fully understand how it works. But this is a question for proffesionals not me.
Greetings - Jacek
|
|
|
|
|
Hi,
I have a datatable in which i know one value of a column. How to get entire row of that particular data?
Thanks
|
|
|
|
|
|
Hi,
I need to split a large string and split that splitted string again. I have the following string:
root
{
stringB =
{
"key": "value",
"key": "value",
"StringC":
{
"key": "value",
"key": "value",
"key": "value",
"key": "value",
}
"StringD": "value",
"StringE":
{
"key": "value",
}
}
}
The result I actually want is something like this:
string[] AllStrings = StringB,StringC,StringD,StringE
Is there a way to do this? I used Regex.Split option.
Thanks in advance!
|
|
|
|
|
That doesn't look regular enough. And try the Regex forum.
|
|
|
|
|
You could probably do it with String.Split()[^].
The best things in life are not things.
|
|
|
|
|
I believe you are trying to parse a JSON string. If yes then this link might be of some help to you!
|
|
|
|
|
You can parse using JavaScriptSerializer.. This may help to you
string s = "{\"StringC\":"+
"{"+
"\"key1\": \"value\"," +
"\"key2\": \"value\"," +
"\"key3\": \"value\"," +
"\"key4\": \"value\"" +
"},"+
"\"StringD\": \"value\"," +
"\"StringE\":" +
"{"+
"\"key1\": \"value\"" +
"}"+
"}";
JavaScriptSerializer f =new JavaScriptSerializer();
JsonData jsd = f.Deserialize<JsonData>(s);
public class JsonData
{
public Hashtable StringC { get; set; }
public string StringD {get; set;}
public Hashtable StringE { get; set; }
}
|
|
|
|
|
Just so you know, your sample string isn't very consistent. I would think there would have to be a comma after the closing braces.
Beyond that, it seems to me you want to Split on the "{" character, and then the "," character on each of the first split result strings. I would probably write a method that accepted the delimiter character and the string to be split that returned an array of split items.
Regex is not what you want to use - sometimes, you just gotta write the code.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass." - Dale Earnhardt, 1997
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
