CodeProject Article Scraping

• Download CPAM - 365 KB

Introduction

This article describes a method for scraping data off of the CodeProject My Articles page. There is currently no CodeProject API for retrieving this data, so this is the only way to get the info. Unfortunately, the format of this page could change at any time, and may break this code, so it's up to you to stay on top of this issue. This should be quite easy since I've done all the hard work for you - all you have to do is maintain it.

The ArticleData Class

The ArticleData class contains the data for each article scraped off the web page. The most interesting aspect of this class is that it's derived from IComparable so that the generic list that contains the ArticleData objects can sort the list on any of the scraped values. There are several ways to sort a generic list, and I used the one that kept the referring code the cleanest. What I'm trying to say is that you should pick the way you want to do it. No method is more correct than any other, and is more a factor of programmer style and preference than anything else.

The Way I Did It

I chose to derive the AriticleData class from IComparable, and write the functions necessary to perform the sorting. This keeps the referencing code free of needless clutter, thus making the code easier to read. This is the way I like to do things. In my humble opinion, there is no point in bothering the programmer with needless minutia. Instead of posting the entire class in this article, I'll simply show you two of the sorting functions:

public class ArticleData : IComparable<ArticleData>
{
	// DATA MEMBERS

	// PROPERTIES

	#region Comparison delegates
	/// <summary>
	/// Title comparison for sort function
	/// </summary>
	public static Comparison<ArticleData> TitleCompare = delegate(ArticleData p1, ArticleData p2)
	{
		return (p1.SortAscending) ? p1.m_title.CompareTo(p2.m_title) : p2.m_title.CompareTo(p1.m_title);
	};
	/// <summary>
	/// Page views comparison for sort function
	/// </summary>
	public static Comparison<ArticleData> PageViewsCompare = delegate(ArticleData p1, ArticleData p2)
	{
		return (p1.SortAscending) ? p1.m_pageViews.CompareTo(p2.m_pageViews) : p2.m_pageViews.CompareTo(p1.m_pageViews);
	};


	// there are more comparison delegates here

	/// <summary>
	/// Default comparison (compares article ID) for sort function
	/// </summary>
	public int CompareTo(ArticleData other)
	{
		return ArticleID.CompareTo(other.ArticleID);
	}
	#endregion Comparison delegates

The ArticleUpdate class

This class is derived from the ArticleData class, and at first blush, it appears as if it's an exact duplicate of the ArticleData class, but that's not the case. To make the code truly useful, you need a way to identify changes since your last data scrape. For the purposes of this demo, that's what this class enables. I recognize that you might have different reasons for scraping the My Articles page, so you should be prepared to write your own class that performs the functionality your application requires. It's my guess that your implementation will be more extensive than my own.

The class has its own sort delegates. They're similar enough that I decided not to actually show them in this article because I think it would be redundant. The truly interesting methods n this class are:

ApplyChanges

This method is called from the scraper manager object (covered in the next section) when an article is scraped of the web page. If the article exists in the list of existing articles, we call this method to change the data to its existing values. If ANYTHING has changed for the article, this method returns true.

public bool ApplyChanges(ArticleUpdate item, DateTime timeOfUpdate, bool newArticle)
{
	bool changed = false;

	// make them all the same
	this.m_title			= m_latestTitle;
	this.m_link			= m_latestLink;
	this.m_lastUpdated		= m_latestLastUpdated;
	this.m_description		= m_latestDescription;
	this.m_pageViews		= m_latestPageViews;
	this.m_rating			= m_latestRating;
	this.m_votes			= m_latestVotes;
	this.m_popularity		= m_latestPopularity;
	this.m_bookmarks		= m_latestBookmarks;

	// set new info
	this.m_latestTitle		= item.m_latestTitle;
	this.m_latestLink		= item.m_latestLink;
	this.m_latestDescription	= item.m_latestDescription;
	this.m_latestPageViews		= item.m_latestPageViews;
	this.m_latestRating		= item.m_latestRating;
	this.m_latestVotes		= item.m_latestVotes;
	this.m_latestPopularity		= item.m_latestPopularity;
	this.m_latestBookmarks		= item.m_latestBookmarks;

	// make a note of the last update time stamp
	this.m_timeUpdated		= timeOfUpdate;
	this.m_newArticle		= newArticle;

	// see if anything changed since the last update
	changed = (this.m_title		!= m_latestTitle	||
		   this.m_link		!= m_latestLink		||
		   this.m_lastUpdated	!= m_latestLastUpdated	||
		   this.m_description	!= m_latestDescription	||
		   this.m_pageViews	!= m_latestPageViews	||
		   this.m_rating	!= m_latestRating	||
		   this.m_votes		!= m_latestVotes	||
		   this.m_popularity	!= m_latestPopularity	||
		   this.m_bookmarks	!= m_latestBookmarks	||
		   this.m_newArticle	== true);

	m_changed = changed;

	return changed;
}

PropertyChanged

The PropertyChanged method allows you to see if a specific property has changed. Simply provide the property name, and handle the return value (true if the property's value changed).

public bool PropertyChanged(string property)
{
	string originalProperty = property;
	property = property.ToLower();
	switch (property)
	{
		case "title"		: return (Title != LatestTitle);
		case "link"		: return (Link != LatestLink);
		case "description"	: return (Description != LatestDescription);
		case "pageviews"	: return (PageViews != LatestPageViews);
		case "rating"		: return (Rating != LatestRating);
		case "votes"		: return (Votes != LatestVotes);
		case "popularity"	: return (Popularity != LatestPopularity);
		case "bookmarks"	: return (Bookmarks != LatestBookmarks);
		case "lastupdated"	: return (LastUpdated != LatestLastUpdated);
	}
	// if we get here, the property is invalid
	throw new Exception(string.Format("Unknown article property - '{0}'", originalProperty));
}

HowChanged

This method accepts a property name, and returns a ChangeType enumerator indicating if the new value is equal to, greater than, or less than the last value that was scraped.

public ChangeType HowChanged(string property)
{
	ChangeType changeType = ChangeType.None;

	string originalProperty = property;
	property = property.ToLower();

	switch (property)
	{
		case "title": 
			break;

		case "link": 
			break;

		case "description": 
			break;

		case "pageviews": 
			{
				if (PageViews != LatestPageViews)
				{
					changeType = ChangeType.Up;
				}
			}
			break;

		case "rating": 
			{
				if (Rating > LatestRating)
				{
					changeType = ChangeType.Down;
				}
				else
				{
					if (Rating < LatestRating)
					{
						changeType = ChangeType.Up;
					}
				}
			}
			break;

		case "votes": 
			{
				if (Votes != LatestVotes)
				{
					changeType = ChangeType.Up;
				}
			}
			break;

		case "popularity": 
			{
				if (Popularity > LatestPopularity)
				{
					changeType = ChangeType.Down;
				}
				else
				{
					if (Popularity < LatestPopularity)
					{
						changeType = ChangeType.Up;
					}
				}
			}
			break;

		case "bookmarks": 
			{
				if (Bookmarks > LatestBookmarks)
				{
					changeType = ChangeType.Down;
				}
				else
				{
					if (Bookmarks < LatestBookmarks)
					{
						changeType = ChangeType.Up;
					}
				}
			}
			break;

		case "lastupdated": 
			break;

		default : throw new Exception(
				string.Format("Unknown article property - '{0}'", 
						originalProperty));
	}

	return changeType;
}

The ArticleScraper Class

To make things easy on myself, I put all of the scraping code into this class. The web page is requested, and then parsed to within an inch of its life. For purposes of this article, I placed no value in determining the category/sub-category under which the article is posted.

The RetrieveArticles method is responsible for making the page request,a nd managing the parsing chores, which are themselves broken up into manageable chunks. During testing of the scraping code, I went to the My Articles page in a web browser, and saved the source code to a file. This allowed me to test without having to repeatedly hammer CodeProject during initial development of the parsing code. I decided to leave the code in the class to allow other programmers the same luxury. Here are the important bits (the text file specified in the code is provided with this articles download file):

	if (this.ArticleSource == ArticleSource.CodeProject)
	{
		// this code actually hits the codeproject website
		string url = string.Format("{0}{1}{2}", 
					   "http://www.codeproject.com/script/",
					   "Articles/MemberArticles.aspx?amid=", 
					   this.UserID);
		Uri uri = new Uri(url);
		WebClient webClient = new WebClient();
		string response = "";
		try
		{
			// added proxy support for those that need it - many thanks to Pete 
			// O'Hanlon for pointing this out.
			webClient.Proxy = WebRequest.DefaultWebProxy;
			webClient.Proxy.Credentials = CredentialCache.DefaultCredentials;
			// get the web page
			response = webClient.DownloadString(uri);
		}
		catch (Exception ex)
		{
			throw ex;
		}
		pageSource = response;
	}
	else
	{
		// this code loads a sample page source from a local text file
		StringBuilder builder = new StringBuilder("");
		string filename = System.IO.Path.Combine(Application.StartupPath, 
							"MemberArticles.txt");
		StreamReader reader = null;
		try
		{
			reader = File.OpenText(filename);
			string input = null;
			while ((input = reader.ReadLine()) != null)
			{
				builder.Append(input);
			}
		}
		catch (Exception ex)
		{
			throw ex;
		}
		finally
		{
			reader.Close();
		}

		pageSource = builder.ToString();
	}

Note - The line in the code above that builds the url string is formatted to prevent the containing <pre> tag from potentially forcing this articles page to require horizontal scrolling.

After getting the web page, the pageSource variable should contain something. If it does, we hit the following code (and we're still in the RetrieveArticles method):

	int articleNumber = 0;
	bool found = true;

	while (found)
	{
		// establish our trigger points
		string articleStart = string.Format("<span id=\"ctl00_MC_AR_ctl{0}_MAS", 
					string.Format("{0:00}", articleNumber));
		// we use the beginning of the next article as the 
		// end of the current one
		string articleEnd   = string.Format("<span id=\"ctl00_MC_AR_ctl{0}_MAS", 
					string.Format("{0:00}", articleNumber + 1));

		// get the index of the start of the next article
		int startIndex = pageSource.IndexOf(articleStart);

		if (startIndex >= 0)
		{
			// delete everything that came before the starting index
			pageSource = pageSource.Substring(startIndex);
			startIndex = 0;

			// find the end of our articles data
			int endIndex = pageSource.IndexOf(articleEnd);

			// If we don't have an endIndex, then we've arrived 
			// at the final article in our list. 
			if (endIndex == -1)
			{
				endIndex = pageSource.IndexOf("<table");
				if (endIndex == -1)
				{
					endIndex = pageSource.Length - 1;
				}
			}

			// get the substring
			string data = pageSource.Substring(0, endIndex);

			// if we have data, process it
			if (data != "")
			{
				ProcessArticle(data, articleNumber);
			}
			else
			{
				found = false;
			}
			articleNumber++;
		}
		else
		{
			found = false;
		}
	} // while (found)

	CalculateAverages();

I guess I could have used LINQ to scrounge around in the XML, but when you get right down to it, we can't count on the HTML being valid, so it's simply more reliable to parse the text this way. I know, Chris, et al., work hard at making sure everything is just so, but they are merely human, and we know we can't count on humans to do it right every single time.

Processing an Article

By "process", I mean parsing out the HTML and digging the actual data out of the article's div. While fairly simple, it is admittedly tedious. We start out by getting the article's URL, which is a straightforward operation:

private string GetArticleLink(string data)
{
	string result = data;
	// find the beginning of the desired text
	int hrefIndex = result.IndexOf("href=\"") + 6;
	//find the end of the desired text
	int endIndex = result.IndexOf("\">", hrefIndex);
	// snag it
	result = result.Substring(hrefIndex, endIndex - hrefIndex).Trim();
	// return it
	return result;
}

Next, we clean the data, starting off by removing all of the HTML tags. A change was made to the source code to make the removal of HTML tags a little smarter. If the article title and/or description contain more than one pointy bracket, this method will be almost guaranteed to return only a portion of the actual text of the item in question. If you like, you can google for (and use) one of the many exhaustive HTML parsers available on the net. IMHO, it's not worth the effort considering this class' primary usage and consistently decent HTML we get from CodeProject.

private string RemoveHtmlTags(string data)
{
	int ltCount = CountChar(data, '<');
	int gtCount = CountChar(data, '>');

	// If the number of left and right pointy bracks are the same, we stand a 
	// reasonable chance that what we think are html tags really ARE html tags.
	if (ltCount == gtCount)
	{
		data = ForwardStrip(data);
	}
	else
	{
		// Otherwise, we have an errant pointy bracket, which we can almost 
		// always take care of depending on the order in which we search for 
		// tags.
		if (gtCount > ltCount)
		{
			data = BackwardStrip(ForwardStrip(data));
		}
		else
		{
			data = ForwardStrip(BackwardStrip(data));
		}
	}
	return data;
}


private int CountChar(string data, char value)
{
	int count = 0;
	for (int i = 0; i < data.Length; i++)
	{
		if (data[i] == value)
		{
			count++;
		}
	}
	return count;
}

private string ForwardStrip(string data)
{
	bool	found	= true;
	do
	{
		int tagStart = data.IndexOf("<");
		int tagEnd = data.IndexOf(">");
		if (tagEnd >= 0)
		{
			tagEnd += 1;
		}
		found = (tagStart >= 0 && tagEnd >= 0 && tagEnd-tagStart > 1);
		if (found)
		{
			string tag = data.Substring(tagStart, tagEnd - tagStart);
			data = data.Replace(tag, "");
		}
	} while (found);
	return data;
}

private string BackwardStrip(string data)
{
	bool	found	= true;
	do
	{
		int tagStart = data.LastIndexOf("<");
		int tagEnd = data.LastIndexOf(">");
		if (tagEnd >= 0)
		{
			tagEnd += 1;
		}
		found = (tagStart >= 0 && tagEnd >= 0 && tagEnd-tagStart > 1);
		if (found)
		{
			string tag = data.Substring(tagStart, tagEnd - tagStart);
			data = data.Replace(tag, "");
		}
	} while (found);
	return data;
}

Then, we remove all the extra stuff left behind:

private string CleanData(string data)
{
	// get rid of the HTML tags
	data = RemoveHtmlTags(data);

	// get rid of the crap that's left behind
	data = data.Replace("\t", "^").Replace(" ", "");
	data = data.Replace("\n","").Replace("\r", "");
	data = data.Replace(" / 5", "");
	while (data.IndexOf("  ") >= 0)
	{
		data = data.Replace("  ", " ");
	}
	while (data.IndexOf("^ ^") >= 0)
	{
		data = data.Replace("^ ^", "^");
	}
	while (data.IndexOf("^^") >= 0)
	{
		data = data.Replace("^^", "^");
	}
	data = data.Substring(1);
	data = data.Substring(0, data.Length - 1);
	return data;
}

After this, we're left with a pure list of data that describes the article, delimited with caret characters. All that's left is to create an ArticleUpdate item and store it in our generic list.

private void ProcessArticle(string data, int articleNumber)
{
	string link	= GetArticleLink(data);
	data = CleanData(data);
	string[] parts = data.Split('^');
	string title = parts[0];
	string description = parts[7];
	string lastUpdated = GetDataField("Last Update", parts);
	string pageViews = GetDataField("Page Views", parts).Replace(",", "");
	string rating = GetDataField("Rating", parts);
	string votes = GetDataField("Votes", parts).Replace(",", "");
	string popularity = GetDataField("Popularity", parts);
	string bookmarks = GetDataField("Bookmark Count", parts);

	// create the AticleData item and add it to the list
	DateTime lastUpdatedDate;
	ArticleUpdate article = new ArticleUpdate();
	article.LatestLink = string.Format("http://www.codeproject.com{0}", link);
	article.LatestTitle = title;
	article.LatestDescription = description;
	if (DateTime.TryParse(lastUpdated, out lastUpdatedDate))
	{
		article.LatestLastUpdated = lastUpdatedDate;
	}
	else
	{
		article.LatestLastUpdated = new DateTime(1990, 1, 1);
	}
	article.LatestPageViews		= Convert.ToInt32(pageViews);
	article.LatestRating		= Convert.ToDecimal(rating);
	article.LatestVotes		= Convert.ToInt32(votes);
	article.LatestPopularity	= Convert.ToDecimal(popularity);
	article.LatestBookmarks		= Convert.ToInt32(bookmarks);

	AddOrChangeArticle(article);
}

private void AddOrChangeArticle(ArticleUpdate article)
{
	bool found = false;
	DateTime now = DateTime.Now;

	// apply changes
	for (int i = 0; i < m_articles.Count; i++)
	{
		ArticleUpdate item = m_articles[i];
		if (item.LatestTitle.ToLower() == article.LatestTitle.ToLower())
		{
			found = true;
			item.ApplyChanges(article, now, false);
			break;
		}
	}

	// if the article was not found, it must be new (or the title has changed), 
	// so we'll add it
	if (!found)
	{
		article.ApplyChanges(article, now, true);
		m_articles.Add(article);
	}

	// remove all articles that weren't updated this time around - we need to 
	// traverse the list in reverse order so we don't lose track of our index
	for (int i = m_articles.Count - 1; i == 0; i--)
	{
		ArticleUpdate item = m_articles[i];
		if (item.TimeUpdated != now)
		{
			m_articles.RemoveAt(i);
		}
	}
}

The Sample Application

The sample application is admittedly a rudimentary affair, and is honestly intended to show only one possible way to use the scraping code. I decided to use a WebBowser control, but about halfway through the app, I began to regret that decision. However, I was afraid I'd become bored with the whole thing, and determined to soldier on.>/p>

You'll see that I didn't go to heroic lengths to pretty things up. For instance, I used PNG files for the graphics instead of GIF files. This means the transparency in the PNG files isn't handled correctly on systems running IE6 or earlier.

The application allows you to select the data on which to sort, and in what direction (ascending or descending). The default is the date last updated in descending order so that the newest articles appear first.

The WebBrowser control displays the articles in a table, and uses icons to indicate changed data and certain statistical information regarding articles. The article titles are hyperlinks to the actual article's page, and that page is displayed within the WebBrowser control. To go back to the article display, you have to click the Sort button because I didn't implement any of the forward/back functionality you find in a normal web browser.

The icons used are as follows:

- Indicates a new article. All articles will display as new when you initially start the application.

- Indicates the article with the best rating.

- Indicates the article with the worst rating.

- Indicates the article with the most votes.

- Indicates the article with the most page views.

- Indicates the most popular article.

- Indicates the article with the most bookmarks.

- Indicates that the associated field increased in value.

- Indicates that the associated field decreased in value.

Other controls on the form include the following.

Show New Info Only

This checkbox allows you to filter the list of articles so that only new articles, and articles that have new data are displayed.

Show Icons

This checkbox allows you to turn the display of icons on and off.

Automatic Refresh

This checkbox allows you to turn the automatic refrsh on and off. Once every hour, a BackgroundWorker object is used to refresh the article data.

Button - Refresh From CodeProject

This button allows you to manually refresh the article data (and this button is available even if auto-refresh is turned on).

Lastly, you can specify the user ID of the user for whom you would like to retrieve data. After specifying the new ID, hit the Refresh button.

Closing

This code is only intended to be used to retrieve your own articles - the scraper class accepts the user ID as a parameter, and that ID is currently set to my own. Make sure you change that before you start looking for your own articles.

I've tried to make this as maintainable as possible without forcing the programmer to do conceptual back-flips, but there's no way I can acommodate everyone's reading comprehension levels, so what I guess I'm saying is - you're pretty much on your own. I can't guarantee that I will be able to maintain this article in a timely fashion, but that shouldn't matter. We're all programmers here, and the stuff I've presented isn't rocket science. Besides, you have plenty of examples in the provided classes to modify and/or extend their functionality. Have fun.

Remember also, that the png files and css file need to be in the same folder as the executable, or it won't find them.

History

10/14/2008: Addressed the following:    

• Added support for retrieving the web page via a proxy (thanks Pete O'Hanlon!).
   
• Added code to throw any exception encountered during the web page retrieval process (thanks again Pete O'Hanlon!).
   
• Added a slightly more thorough HTML parse to handle errant < and > in the title or description of the article (thanks ChandraRam!).
   
• Embedded the icons as resources in the exe file. They will be copied to the app folder the first time the exe is run.
   
• Added a new statistic item at the top of the form - "Articles Displayed".
   
• Enclosed the stuff at the top ofthe form in group boxes to make it look more organized.

10/13/2008: Addressed the following:    

• Added the forgotten mostvotes.png image.
   
• Modified code to use the mostvotes image.
   
• Added a textbox to the form to allow you to specify the userID.
   
• Fixed the form resizing issues.
   
• The zip file now includes the debug folder with the images and css file.

10/13/2008: Original article posted.

You must Sign In to use this message board.

FAQ  Noise Tolerance Very HighHighMediumLowVery Low  Search Messages    Layout Expand All MessagesThread ViewNo Javascript (slow)No Javascript Preview    Per page 102550

Msgs 1 to 25 of 47 (Total in Forum: 47) (Refresh) FirstPrevNext

Flaw in Sample Application John Simmons / outlaw programmer 2:31 28 Oct '08   The sample application will crash with an exception if CodeProject is down when it tries to retrieve articles (this happened yesterday during the site update). The fix is a simple one. In the Form1.cs, in the UpdateDisplay() method, add a try/catch block around the call to RetrieveArticles() method (at or around line 331), like so: private void UpdateDisplay() { ... this.Text = string.Format("{0} - Scraping!", m_windowTitle); this.Cursor = Cursors.WaitCursor; // ---- change starts here ---- // retrieve the articles try { m_articleScraper.RetrieveArticles(); } catch (Exception ex) { // we don't need to do anything with the exception because the WebControl // will display an appropriate message if the article list is empty if (ex != null) {} } // ---- change ends here ---- m_articles = m_articleScraper.Articles; ... } I'll post an updated ZIP file sometime today. "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 Sign In·View Thread·PermaLink I am forced to give you a 5... Steven A. Lowe 17:50 20 Oct '08   ...since I was going to write something like this and you beat me to it! (and quite well, I might add) Best regards, Steven A. Lowe CEO, Innovator LLC www.nov8r.com Sign In·View Thread·PermaLink Re: I am forced to give you a 5... John Simmons / outlaw programmer 0:28 21 Oct '08   You could always write a different app around the scraping code. "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 Sign In·View Thread·PermaLink Good one N a v a n e e t h 6:57 18 Oct '08   John, Great article. I loved the color combination and icons. Some suggestions 1 - When loading for the first time, GUI hanged until it fetches the data. Implementing some progress display when data is updating would be good. 2 - If CP changed the way articles are displayed, this code will break. I think if CP can provide some webservice which gives the necessary data, program will be more stable. Good work though. 5ed! Navaneeth How to use google | Ask smart questions Sign In·View Thread·PermaLink Re: Good one John Simmons / outlaw programmer 5:06 19 Oct '08   N a v a n e e t h wrote: 1 - When loading for the first time, GUI hanged until it fetches the data. Implementing some progress display when data is updating would be good. Well, it is what it is. Like I already told someone else, it's just a sample app. I'm working on a new app/article that is SIGNIFICANTLY more complete than the sample app provided here. At least one other person is working on a WPF application. N a v a n e e t h wrote: 2 - If CP changed the way articles are displayed, this code will break. I think if CP can provide some webservice which gives the necessary data, program will be more stable. Yeah, I mention that a couple of times in the article itself. Chris is making noise like he'd lie to provide an API, but he's severely short-handed at the moment. Until he implements the API, we have to live with the possibility of changes to the page. "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 Sign In·View Thread·PermaLink Update! [modified] John Simmons / outlaw programmer 9:58 14 Oct '08   I changed the article some, and uploaded new code. I also changed the screenshot, but to see it, you may need to clear your browser's cache. [EDIT} There appears to be something wrong with CP. I submitted new article text, and it's not showing up. [EDIT] Fixed (I think). "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 modified on Tuesday, October 14, 2008 5:07 PM Sign In·View Thread·PermaLink Re: Update! Pete O'Hanlon 3:42 17 Oct '08   I've downloaded the code now, and am looking to make it WPFable - however, I'm a bit miffed here - thanks to Paul Conrad for pointing the proxy stuff out???? I'm not Paul Conrad... Right, I'm off to throw the kids toys out the pram. Deja View - the feeling that you've seen this post before. My blog | My articles | MoXAML PowerToys Sign In·View Thread·PermaLink Re: Update! Paul Conrad 12:57 17 Oct '08   Pete O'Hanlon wrote: thanks to Paul Conrad for pointing the proxy stuff out? I don't recall mentioning anything about proxy stuff. "The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham Sign In·View Thread·PermaLink Re: Update! John Simmons / outlaw programmer 16:44 17 Oct '08   I got the comments buggered up. "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 Sign In·View Thread·PermaLink Re: Update! Paul Conrad 19:04 17 Oct '08   Once again, great article. "The clue train passed his station without stopping." - John Simmons / outlaw programmer "Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon "Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham Sign In·View Thread·PermaLink Re: Update! Pete O'Hanlon 10:23 19 Oct '08   Paul Conrad wrote: I don't recall mentioning anything about proxy stuff. Indeed. Deja View - the feeling that you've seen this post before. My blog | My articles | MoXAML PowerToys Sign In·View Thread·PermaLink Re: Update! John Simmons / outlaw programmer 16:45 17 Oct '08   I'm working on a SysTray-based app for it. The article is gonna be huge (in size, I mean). "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 Sign In·View Thread·PermaLink Interesting, but a couple of points... Pete O'Hanlon 2:37 14 Oct '08   1. You'll have a problem with this code if you're running behind a proxy. You need to add the following code in RetrieveArticles (just before webClient.DownloadString(url)) webClient.Proxy = WebRequest.DefaultWebProxy; webClient.Proxy.Credentials = CredentialCache.DefaultCredentials; 2. In the same method, you are catching and consuming an exception, and there's no indication to the user that anything has gone wrong. Let the exception propagate back to the UI, catch it there and inform the user. Apart from those very minor niggles, oustanding work John and I've voted it a 5. (I might even see about creating a WPF or Silverlight version of the UI later on). Deja View - the feeling that you've seen this post before. My blog | My articles | MoXAML PowerToys Sign In·View Thread·PermaLink Re: Interesting, but a couple of points... John Simmons / outlaw programmer 2:44 14 Oct '08   Pete O'Hanlon wrote: 1. You'll have a problem with this code if you're running behind a proxy. You need to add the following code in RetrieveArticles (just before webClient.DownloadString(url)) I don't run from behind a proxy, so I would never have caught that. I'll change the code and post an update sometime today. Pete O'Hanlon wrote: 2. In the same method, you are catching and consuming an exception, and there's no indication to the user that anything has gone wrong. Let the exception propagate back to the UI, catch it there and inform the user. Well, at least the app won't just crash on you. I'll look at it. "Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997 ----- "...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 Sign In·View Thread·PermaLink Re: Interesting, but a couple of points... Pete O'Hanlon 2:48 14 Oct '08