Click here to Skip to main content
14,391,651 members
Rate this:
Please Sign up or sign in to vote.
See more:
Hi.
I need to find all domains tied to country specific domain extension?
Because I need to find out all domains (url and total number) with *.tv, *.kz extensions( tv and kz are just example.) for my statics research paper. Most top domain providers are unwilling to share that info. If make Google search "site:*.tv" then I can list all *.tv domains( of course many duplicates because of subdomains). But still have challenge to scrape domain urls to text list. If it requires programming then I prefer C# which my main programming language
Thank you for your suggestions.
Posted
Updated 14-Dec-15 18:33pm
v2

1 solution

Rate this:
Please Sign up or sign in to vote.

Solution 1

The search string would be something like "site:.tv". You need to download HTTP response and scrape obtained HTML, and this needs some HTML parser. It could be, first of all, HTML Agility Pack, or something else. Please see my past answers:
How to get particular data from a url using c#[^],
get specific data from web page[^],
see also How to get the data from another site[^].

—SA
   
Comments
BillWoodruff 15-Dec-15 1:19am
   
+5
xmans 15-Dec-15 22:05pm
   
Can how can move to next Google search page ("site:.tv")? Because there are hundreds of pages. I mean how can I loop through all of this pages with minimum effort? Is there anyway to get result of Google search in one simple text list or file?
Sergey Alexandrovich Kryukov 15-Dec-15 22:10pm
   
Good question. But it's easy to answer.

Well, you can parse the first page and locate... give me a second, I'll try on some query...
You can locate the anchor "Next" or ">" and get it's URL. Use this URL in next HTTP request, and repeat the requests until the page without "Next" (or it's hidden or disabled, check it up). This way, you will visit all the pages.

Didn't I provide the complete answer to your question? Will you now accept it formally?

—SA
xmans 17-Dec-15 3:17am
   
So far nothing is constructive without any code or link to solution. I already knew what you said I just have trouble to figure out how to make it work.
Sergey Alexandrovich Kryukov 17-Dec-15 11:07am
   
If you knew it, you would not ask your follow-up questions. Sorry, but who would like to help you after your talking about "nothing is constructive" and the like? Please, you are free to do your own work.
—SA
xmans 30-Dec-15 3:58am
   
Can you post working code for google search for parsing and moving to next page?
Sergey Alexandrovich Kryukov 30-Dec-15 11:42am
   
The "Google search" is Google code, not mine. :-)
—SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)




CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100