How to read full web page in C#

Question

1.00/5 (3 votes)

See more:

C#

using System;
using System.Collections.Generic;
using System.Linq;
using System.ComponentModel;
using System.Data;
using System.Text;
using System.Net;
using System.IO;

/* I WRITE THIS PROGRAM TO GET WEB CONTENT ON INTERNET , BUT IT NOT WORK CORRECTLY. PLEASE HELP ME */


namespace GetHtmlFromInternet
{
    class Program
    {
        public static WebClient wClient = new WebClient();
            public static StreamWriter sw;
            public static TextWriter textWriter;
        /*I USE THIS FUNCTION TO READ WEB PAGE ON INTERNET BUT IT NOT WORK CORRECTLY ON SOME LINKS*/
            public static String readFromLink()
            {
                /* IT WORK CORRECTLY IF WE SET url = "http://vn.yahoo.com/?p=us"; BUT NOT OK WITH URL BELOW*/
                String url = "http://raovat.com/?rv=detail&idrv=527443&idcate=57&tt=vppnhatthanh@yahoo.com.vn";

                /* PLEASE HELP ME MAKE FUNCTION READFROMLINK RUNABLE WITH THIS LINK*/
                System.Net.WebClient client = new System.Net.WebClient();
                byte[] data = client.DownloadData(url);
                String html = System.Text.Encoding.UTF8.GetString(data);
                return html;


            }
            public static bool WriteTextFile(String fileName, String t)
            {

                try
                {
                    textWriter = new StreamWriter(fileName);
                }
                catch (Exception)
                {
                    return false;
                }

                try
                {
                    textWriter.WriteLine(t);
                }
                catch (Exception)
                {
                    return false;
                }
                textWriter.Close();
                return true;
            }
        static void Main(string[] args)
        {
            String saveFile = "C:/saveFromURL.txt";
            String reSultString = readFromLink();
            WriteTextFile(saveFile, reSultString);
        }
    }
}

Posted 4-Apr-11 3:20am

gdctld

Updated 6-Oct-19 5:35am

Add a Solution

3 solutions

Solution 3

// RaviRanjakr , Kim Togo ! It still can not get full page content , please check that link with your browser and run my program with your code insert into the function readFromLink link like this :
public static String readFromLink()
{
String URL = "http://raovat.com/?rv=detail&idrv=527443&idcate=57&tt=vppnhatthanh@yahoo.com.vn";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
String data=null;
if (response.StatusCode == HttpStatusCode.OK)
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (response.CharacterSet == null)
readStream = new StreamReader(receiveStream);
else readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
data= readStream.ReadToEnd(); response.Close(); readStream.Close();
}
return data;
}

// Please help me again and again .I hope get your feedbacks until we can solve this problem!

Posted 5-Apr-11 7:33am

gdctld

Solution 2

I will do as RaviRanjakr has done. But add some more checking and encoding.

C#

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

if (response.StatusCode == HttpStatusCode.OK)
{
  Stream receiveStream = response.GetResponseStream();
  StreamReader readStream = null;

  if (response.CharacterSet == null)
    readStream = new StreamReader(receiveStream);
  else
    readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));

  string data = readStream.ReadToEnd();
  response.Close();
  readStream.Close();
}

Posted 4-Apr-11 21:19pm

Kim Togo

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

RaviRanjanKr · Accepted Answer · 2011-04-04T03:42:00

Solution 1

you can simple use the given code to read all webpage

C#

HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(URL);
myRequest.Method = "GET";
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(), System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
sr.Close();
myResponse.Close()

for more details you can go there[^] and there[^]

Posted 4-Apr-11 3:42am

RaviRanjanKr

Comments

Albin Abel 4-Apr-11 10:22am

Good answer. My 5

Sergey Alexandrovich Kryukov 4-Apr-11 22:10pm

Correct, my 5.
--SA

Kim Togo 5-Apr-11 3:54am

Instead of just assume that encoding always is UTF8, check with myResponse.CharacterSet what encoding is used.