Click here to Skip to main content
Sign Up to vote bad
good
See more: C++BHO
The purpose is to eventually develop an extension for IE which is capable of detecting all search results URL's when searching for any topic on Google, and save these URL's in a text file.
 
The main problem I am facing is that all search results on Google are not visible, if you try inspecting the source code of a Google search result page, you'll find that their pages were built dynamically and therefore, there are no links available on the pages, just a bunch of dynamic java scripting.
 
How can I retrieve the true URL links using the extension?
Posted 1 Feb '13 - 0:07
Daroosh596


2 solutions

I managed to reach a solution to this problem.
 
My problem was that i thought i couldn't access the URL's on the Google search page. You won't be able to view them if you try viewing the PAGE SOURCE in IE however, the URL's are not stored in the PAGE SOURCE anyway since that's only the MAIN HTML SOURCE Page. The URL search list is actually built in dynamically and its available in the pages' DOM.
 
What you'll need to do is capture the URL's by catching them in one of the DISPID_DOM**** Events. The DOM events are fired only when the DOM is completely loaded, unlike the DISPID_DOCUMENTCOMPLETE event which is fired when the Source HTML is loaded.
 
The URL's are nested in the DOM in this way:
 
<div id="ires">
       <a class="1" href="URL">
      </a>
</div>
 
What you'll need to do is Search for the "div" Tag with an "id=ires" value using the get_id(&idStr) method. Then traverse every node within this tag till you find the "href" you are looking for.
  Permalink  
Comments
H.Brydon - 4 Feb '13 - 12:59
Great research. +5 for question and answer.
You can use Google API to get search results. And search word could be retrieved from URI.
  Permalink  
Comments
Daroosh - 1 Feb '13 - 8:20
Which Google API? Can you offer me some Source code to further elaborate your solution.
Kurkul - 1 Feb '13 - 8:29
Please look at https://developers.google.com/custom-search/docs/xml_results?hl=en#wsSampleQueries to get result in xml

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Your Filters
Interested
Ignored
     
0 Sergey Alexandrovich Kryukov 483
1 Arun Vasu 315
2 OriginalGriff 250
3 Maciej Los 218
4 Aarti Meswania 170
0 Sergey Alexandrovich Kryukov 9,670
1 OriginalGriff 7,409
2 CPallini 3,968
3 Rohan Leuva 3,352
4 Maciej Los 2,861


Advertise | Privacy | Mobile
Web02 | 2.6.130523.1 | Last Updated 4 Feb 2013
Copyright © CodeProject, 1999-2013
All Rights Reserved. Terms of Use
Layout: fixed | fluid