Click here to Skip to main content
12,304,652 members (65,179 online)
Click here to Skip to main content

Tagged as

Stats

5.4K views
5 bookmarked
Posted

Why a robots.txt file is no longer sufficient

, 16 Jan 2013 CPOL
Over the past year Google decided to partially ignore the robots.txt file in an attempt to increase the effectiveness of its results.
No downloads associated with this content

By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author


You may also be interested in...

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.160530.1 | Last Updated 16 Jan 2013
Article Copyright 2013 by Zac Gery
Everything else Copyright © CodeProject, 1999-2016
Layout: fixed | fluid