Articles by Richard Penman (Technical Blogs: 24)

My solution using Webkit.

How to learn about web scraping.

How to use proxies.

Crawling with threads.

Using Google Translate to crawl a website.

I have three solutions for periodically scraping a website.

Why web2py?

Scraping JavaScript based web pages with Chickenfoot.

Webkit has now been ported to the Qt framework and can be used through its Python bindings.

I made an earlier post about using webkit to process the JavaScript in a webpage so you can access the resulting HTML and how to apply it to multiple webpages.

How to use XPaths robustly

HTML is a tree structure: at the root is a tag followed by the and tags and then more tags before the content itself. However when a webpage is downloaded all one gets is a series of characters. Working directly with that text is fine when using regular expressions, but often we want to traverse

I have been interested in automatic approaches to web scraping for a few years now.During university I created the SiteScraper library, which used training cases to automatically scrape webpages.This approach was particularly useful for scraping a website periodically because the model could automat

Some strategies to protect your data.

How to crawl websites without being blocked.

Using regular expressions for web scraping is sometimes criticized, but I believe they still have their place, particularly for one-off scrapes. Let's say I want to extract the title of a particular webpage - here is an implementation using BeautifulSoup, lxml, and regular expressions:import reimpor

In this post I will clarify what I do by walking through a simple web scraping job I worked on.

Sometimes I need to import large spreadsheets into MySQL.The easy way would be to assume all fields are varchar, but then the database would lose features such as ordering by a numeric field.The hard way would be to manually determine the type of each field to define the schema.That doesn't sound mu

Python and other scripting languages are sometimes dismissed because of their inefficiency compared to compiled languages like C. For example here are implementations of the fibonacci sequence in C and Python:int fib(int n){ if (n < 2) return n; else return fib(n - 1) + fib(n - 2);}int m

Automatically scraping website data based on example cases.

How to increase your Google App Engine quotas for free.

In a previous post I showed how to scrape a list of webpages. Here is an updated example that downloads the content in multiple threads.

There is a nice website screenshots.com that hosts historic screenshots for many websites.

Some websites require passing a CAPTCHA to access their content. As I have written before these can be parsed using the deathbycaptcha API, however for large websites with many CAPTCHA's this becomes prohibitively expensive. For example solving 1 million CAPTCHA's with this API would cost $1390.Fort