Retrieving the HTML source code

Introduction

An app I was writing needed to store the full HTML of a web page. I looked all over the web and the MSDN library on how to get the complete HTML from a CHtmlView. I found out how to get the <BODY></BODY> data, but not how to get the <HTML></HTML> data. After lots of stumbling, I hit on the following very simple technique.

Examples of getting the outer HTML of the <BODY> tag abound. While exploring the IHTMLDocument2 interface, I noticed the get_ParentElement method. I realized that the parent of <BODY> is <HTML>.

This function took care of my problem:

bool CMyHtmlView::GetDocumentHTML(CString &str)
{
    IHTMLDocument2 *lpHtmlDocument = NULL;
    LPDISPATCH lpDispatch = NULL;

    lpDispatch = GetHtmlDocument();
    if(!lpDispatch)
        return false;

    lpDispatch->QueryInterface(IID_IHTMLDocument2, (void**)&lpHtmlDocument);
    ASSERT(lpHtmlDocument);
    lpDispatch->Release();

    IHTMLElement *lpBodyElm;
    IHTMLElement *lpParentElm;

    lpHtmlDocument->get_body(&lpBodyElm);
    ASSERT(lpBodyElm);
    lpHtmlDocument->Release();
    // get_body returns all between <BODY> and </BODY>. 
    // I need all between <HTML> and </HTML>.

    // the parent of BODY is HTML
    lpBodyElm->get_parentElement(&lpParentElm);
    ASSERT(lpParentElm);
    BSTR    bstr;
    lpParentElm->get_outerHTML(&bstr);
    str = bstr;

    lpParentElm->Release();
    lpBodyElm->Release();

    return true;
}

Points of Interest

There is bound to be a better way of doing this. If you know it, please share it with me.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Geno Carman

United States

Member

Article Top

Poor

Excellent

Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

You must Sign In to use this message board. (secure sign-in)

Search this forum
	Profile popups Noise Layout Per page

Thanks you!

Stoney Tian

21:56 26 Nov '10

Thanks you! My Homepage : http://blog.csdn.net/accesine960 china,beijing
Sign In·View Thread·Permalink

You saved my day too !

johan.julien

21:57 24 Feb '09

Thanks for the simple technique to get the html content of an IHTMLDocument2 !
Sign In·View Thread·Permalink

My Way of Doing It in C#

Sniper167

19:28 18 Jul '07

private void button1_Click(object sender, EventArgs e) { WebRequest webRequest = WebRequest.Create(textBox1.Text.ToString()); webRequest.Timeout = 100000; WebResponse webResponse = webRequest.GetResponse(); Stream s = webResponse.GetResponseStream(); StreamReader sr = new StreamReader(s, Encoding.ASCII); String doc = sr.ReadToEnd(); int byteNum = doc.Length; richTextBox1.Text = doc; } Sniper167
Sign In·View Thread·Permalink

Another way

vinhie47

16:15 3 Jul '07

In addition of using mshtml, I used another way that only use socket no MFC, no Win32 API like this: string request = "GET / HTTP/1.0"; request.append("Host: yoursite.com"); request.append(""); mySocket.send(request); string respone = mySocket.response(); My English isn't good, so you mayn't understand what I said
Sign In·View Thread·Permalink

how can i get a http-header

emmi

23:51 6 Feb '07

hi, is there a possibility to receive the http-header of the loaded url? what i want to do is to check if the loaded url is available or not and i want to do only one server connection. thanks for your help. best regards emmi
Sign In·View Thread·Permalink

Explaination of the code

Glen_CodeProj

18:40 28 Mar '06

hi, pl. send us the complete explaination of your code ro get the html content. rdgs , glen_codeproject (glenvdsilva@gmail.com) -- modified at 23:18 Wednesday 29th March, 2006
Sign In·View Thread·Permalink

Re: Explaination of the code

RancidCrabtree

19:36 28 Mar '06

Send what, to whom, to where?
Sign In·View Thread·Permalink

Re: Explaination of the code

Glen_CodeProj

17:27 29 Mar '06

email is : glenvdsilva@gmail.com rgds, glen_codeproject -- modified at 23:27 Wednesday 29th March, 2006
Sign In·View Thread·Permalink

Thanks you saved my day

Damir Valiulin

5:43 23 Sep '05

Thanks! I was struggling with this for past couple of days trying to get the HTML source from IHTMLDocument2 interface. I took over a project written in Managed C++ .NET (not my specialty), and my only solution that I found was to extract the source from IHTMLDocument2. However I could not figure out how to properly enumerate through the IHTMLElementCollection returned by 'all' method.

Your get_Parent method saved my day! Smile | :)

My method

HughJampton

2:24 10 May '05

This works in VC6: `CString buffer; CInternetSession isess; CHttpFile* f = (CHttpFile*) isess.OpenURL("http://test.com/test.htm"); while(f->ReadString(buffer)) { htmlstring += buffer; }` The full source code for the page is now in a CString. Hugh Jampton
Sign In·View Thread·Permalink

Re: My method

RancidCrabtree

12:29 12 May '05

Your method is much neater that mine. I appreciate your taking the time to show me.

I implemented yours, but discovered a memory leak. It is probably so obvious that you assumed I would do it, but the file needs to be closed, and the file pointer f needs to be destroyed.

bool MyApp::GetDocumentHTML(CString &str)
{
// From Hugh Jampton
    CString buffer;
    CInternetSession isess;
    CHttpFile* f = (CHttpFile*)isess.OpenURL(m_Browser.GetLocationURL());
    while(f->ReadString(buffer))
    {
	str += buffer;
    }
 
    f->Close();
    delete f;
    return true;
}

Re: My method

Sam NG

23:29 17 Apr '06

It is not quite the same. CHtmlView will handle HTTP (302) redirect, Javascript/VBScript redirect, onload, etc. Unless you are sure the page you want is exactly that URL (which usually not the case unless your site is static), using CHtmlView will be more close to WYSIWYG.
Sign In·View Thread·Permalink	1.50/5 (2 votes)

One-line way of doing it

jocool

3:37 13 Dec '04

Insteal of using the HTMLView or other tricks, why not simply download the whole HTML File ? Add this line in your code : URLDownloadToFile(0, "http://www.ANYTHING.com/index.html", "c:\\test.html", 0, 0); and you will have the HTML file on your HD. Jo JoCool
Sign In·View Thread·Permalink

Re: One-line way of doing it

RancidCrabtree

17:47 13 Dec '04

That's fine for getting the HTML onto disk, but I wanted access to it in my program. My article erroneously states that I `"needed to store the full HTML of a web page"`. In fact, my app modifies it and displays the modified copy. Writing to disk and then reading from it opens the door for a lot of needless problems.
Sign In·View Thread·Permalink	2.00/5 (1 vote)

The way I do it

Uwe Keim

19:51 30 Nov '04

This is the way I do it in a project of mine (for www.zeta-producer.com [^] to be exact Wink | ;-)

).

I don't know exactly, but I recall that I copied something from the CHtmlEditCtrl [^] class of MFC.

const CString CMyHtmlView::GetDocumentHtml() const
{
	MsHtml::IHTMLDocument2Ptr doc = GetHtmlDocument();
	IPersistStreamInitPtr stream = doc;
  
	// From AFXHTML.H.
	CStreamOnCString sstream;
  
	stream->Save( static_cast<IStream*>(&sstream), false );
  	
	CString result;
	VERIFY(sstream.CopyData( result ));
  
	return result;
}

Maybe this helps you.

--
Affordable Windows-based CMS: www.zeta-producer.com

Re: The way I do it

RancidCrabtree

7:38 1 Dec '04

Looks good, and I appreciate the feedback. I'm still using Visual Studio 6.0 which does not support `CStreamOnCString` class.
Sign In·View Thread·Permalink

Re: The way I do it

Uwe Keim

17:03 1 Dec '04

On VC 6, I simply copied the `CStreamOnCString` class from the MFC 7 library into my project to use it -- Affordable Windows-based CMS: www.zeta-producer.com
Sign In·View Thread·Permalink	2.75/5 (4 votes)