Click here to Skip to main content
Click here to Skip to main content

Code Project Posts Analyzer for Windows Phone 7

, 23 Jan 2011
Rate this:
Please Sign up or sign in to vote.
This is a WP7 application that will analyze your most recent posts and give you summarized statistics on your forum wide posting split up.
Prize winner in Competition "Smartphone Programming Competition"

CP Posts Analyzer in action - posts view

CP Posts Analyzer in action - categories view

Introduction

CP Posts Analyzer is a Windows Phone 7 application that will analyze the last 200 posts made by any CP member (across all forums, articles, surveys etc.). You will need a developer license to be able to deploy and run this on your WP7 device, but if you don't have one, you can still run this off the emulator. I've tested the app on the emulator as well as on a Samsung Focus.

Warning

Since the application scrapes HTML from the CP website, the application can potentially stop functioning if the site undergoes a layout change (in the scraped pages). Here's to hoping that, that will not happen too frequently as to make the app tedious to maintain.

This application uses the excellent and highly recommended HTML Agility Pack for parsing HTML.

Using the app

I've included screenshots below that you can look at, but the app is quite simple to use. When you run the app, you get a panorama view and the first page prompts you to enter the member id (example, my member id is 20248). You can get your member id from your profile page. Since the app does not require you to be logged in, you can run this on any member id you want to. The app will maintain a history of the last 10 member ids that you ran it on, so it will save you some typing (which is not pleasant on such small devices with virtual keypads). The recent member ids are stored using isolated storage and are thus persisted until you uninstall the app.

When you run a fetch, the other three pages in the panorama view come into play. The first page gives you a categories based spit up of your last 200 posts. The Lounge falls under the Page category but most other forums including technical and some non-technical ones like the Indian forum fall under Forum. Any posts you make on articles are classified unsurprisingly under Article,  and then you have Survey and Member categories (self explanatory). There may be others I missed or did not encounter, those will show up as Unknown for now.

The next page is the forum-wide split up of your posts, so you can see how many posts you made in a specific forum. Since Chris chose to expose only the last 200 messages you posted, that's the limit for this app too. So the app will basically analyze your most recent posting activity. While debugging/testing the app I found that most of mine were in the Lounge until the India-SA ODI game  happened, and then my posts in the Indian forum dominated like crazy. I ran it on Chris and found that he spends most of his time answering questions in the Site Bugs / Suggestions forum (he does perform a mostly thankless job I guess).

The last page in the panorama view will list all the 200 posts to give you a quick summary of what you posted recently. Just the thread titles (not the messages).

The app has support for tombstoning, so you can switch to another app and come back and your state will be persisted. Well I reckon that's about it. If you can think of something useful that I can add to the app taking into consideration the limitation that all my input data is from html scraping, I'll be happy to try and make that change for you (provided I have the time). Check out a few more screenshots and then there's a brief section on the technical implementation details.

More screenshots

CP Posts Analyzer in the application list

CP Posts Analyzer in action - running a fetch

CP Posts Analyzer in action - categories view for Roger

CP Posts Analyzer in action - forums view

CP Posts Analyzer in action - tile icon

Implementation details

The app was written in SilverLight for WP7 and the project attempts to use a basic MVVM model. To help with data binding, I use these following types that represent the returned data.

namespace CPPostsAnalyzerWP7.Models
{
    public class PostInfo
    {
        public string ThreadName { get; set; }

        public string DisplayName { get; set; }

        public string TimeString { get; set; }

        public string ForumName { get; set; }

        public ForumType ForumType { get; set; }
    }
}
namespace CPPostsAnalyzerWP7.Models
{
    public class PostSummaryInfo
    {
        public string ForumName { get; set; }

        public int Count { get; set; }

        public ForumType ForumType { get; set; }
    }
}
namespace CPPostsAnalyzerWP7.Models
{
    public enum ForumType
    {
        Unknown = 0,

        Forum = 1,

        Page = 2,

        Article = 3,

        Survey = 4,

        Member = 5
    }
}

There is a PostsFetcher class that handles the HTML fetch and parsing, and it uses the excellent HTML Agility Pack library.

// The code has been artificially line-wrapped to fit within the 
// CP article width limits. The source code in the project is formatted
// far more elegantly.

namespace CPPostsAnalyzerWP7.Models
{
    public class PostsFetcher
    {
        private string memberId;

        public PostsFetcher(string memberId)
        {
            this.memberId = memberId.Trim();            
        }

        public event EventHandler<PostInfoEventArgs> PostFetched;
        
        public event EventHandler<FetchCompletedEventArgs> FetchCompleted;

        private void FirePostFetched(PostInfoEventArgs e)
        {
            var handler = this.PostFetched;

            if (handler != null)
            {
                handler(this, e);
            }
        }

        private void FireFetchCompleted(FetchCompletedEventArgs e)
        {
            var handler = this.FetchCompleted;

            if (handler != null)
            {
                handler(this, e);
            }
        }

        private int nextPage = 1;

        private const int maxPage = 4;

        public void Fetch()
        {
            int temp;

            if (Int32.TryParse(this.memberId, out temp))
            {
                LoadNextPageAsync();
            }
            else
            {
                FireFetchCompleted(new FetchCompletedEventArgs() 
                  { Error = new ArgumentException("Invalid memberId.") });
            }
        }

        private void LoadNextPageAsync()
        {
            HtmlWeb.LoadAsync(String.Format(
              "http://www.codeproject.com/script/Forums/Messages.aspx?fmid={0}&fid=0&pgnum={1}", 
              this.memberId, nextPage++), HtmlLoaded);
        }

        private void HtmlLoaded(object sender, HtmlDocumentLoadCompleted e)
        {
            if (e.Error != null)
            {
                FireFetchCompleted(new FetchCompletedEventArgs() { Error = e.Error });
                return;
            }

            try
            {
                ParseHtml(e);
            }
            catch (Exception ex)
            {
                FireFetchCompleted(new FetchCompletedEventArgs() { Error = ex });
            }

            if (nextPage > maxPage)
            {
                var args = new FetchCompletedEventArgs();

                foreach (var item in forumTypeCountMap)
                {
                    args.ForumTypeSummaries.Add(new PostSummaryInfo() 
                      { ForumType = item.Key, Count = item.Value, ForumName = String.Empty });
                }

                foreach (var item in forumPostsCountMap)
                {
                    args.PostSummaries.Add(new PostSummaryInfo() 
                      { ForumType = ForumType.Unknown, Count = item.Value, ForumName = item.Key });
                }

                FireFetchCompleted(args);
            }
            else
            {
                LoadNextPageAsync();
            }
        }

        private Dictionary<ForumType, int> forumTypeCountMap = new Dictionary<ForumType, int>();
        
        private Dictionary<string, int> forumPostsCountMap = new Dictionary<string, int>();

        private void ParseHtml(HtmlDocumentLoadCompleted e)
        {
            var tableNode = e.Document.DocumentNode.DescendantNodes().Where(
                n => n.Name.ToLower() == "table"
                    && n.Attributes["cellspacing"] != null
                    && n.Attributes["cellspacing"].Value == "4").FirstOrDefault();

            if (tableNode == null)
                return;

            var trNodes = tableNode.Descendants("tr");
            foreach (var tdNode in trNodes)
            {
                var aNode = tdNode.Descendants("a").FirstOrDefault();
                if (aNode == null)
                    continue;

                PostInfo postInfo = new PostInfo();
                postInfo.ThreadName = aNode.InnerText.Trim();

                var divNodes = tdNode.Descendants("div").Where(
                    n => n.Attributes["class"] != null
                        && n.Attributes["class"].Value == "small-text subdue");

                if (divNodes.Count() == 2)
                {
                    var divNodesArray = divNodes.ToArray();

                    string nameAndTime = divNodesArray[0].InnerText.Trim();
                    int byPos = nameAndTime.IndexOf("by");
                    int atPos = nameAndTime.LastIndexOf("at");

                    if (byPos == -1 || atPos == -1)
                        continue;

                    postInfo.DisplayName = nameAndTime.Substring(byPos + 2, atPos - byPos - 2).Trim();

                    postInfo.TimeString = nameAndTime.Substring(atPos + 2).Trim();

                    string[] forumLines = divNodesArray[1].InnerText.Trim().Split(
                      '\r', '\n').Where(s => !String.IsNullOrEmpty(s.Trim('\r', '\n'))).ToArray();

                    if (forumLines.Length < 1 || forumLines.Length > 2)
                        continue;

                    string forumNameInput = forumLines.Length == 1 ? "(Untitled)" : forumLines[0];
                    string forumTypeInput = forumLines.Length == 1 ? forumLines[0] : forumLines[1];

                    postInfo.ForumName = forumNameInput.Trim();

                    int leftBracketPost = forumTypeInput.IndexOf('(');
                    int rightBracketPost = forumTypeInput.IndexOf(')');
                    if (leftBracketPost == -1 || rightBracketPost == -1)
                        continue;

                    ForumType forumType = ForumType.Unknown;

                    try
                    {
                        string enumLine = forumTypeInput.Substring(
                          leftBracketPost + 1, rightBracketPost - leftBracketPost - 1).Trim();
                        forumType = (ForumType)Enum.Parse(typeof(ForumType), enumLine.Trim(), true);
                    }
                    catch (ArgumentException)
                    {
                    }

                    postInfo.ForumType = forumType;
                }

                if (!forumTypeCountMap.ContainsKey(postInfo.ForumType))
                {
                    forumTypeCountMap[postInfo.ForumType] = 1;
                }
                else
                {
                    forumTypeCountMap[postInfo.ForumType]++;
                }
                
                if (!forumPostsCountMap.ContainsKey(postInfo.ForumName))
                {
                    forumPostsCountMap[postInfo.ForumName] = 1;
                }
                else
                {
                    forumPostsCountMap[postInfo.ForumName]++;
                }

                FirePostFetched(new PostInfoEventArgs() { PostInfo = postInfo });
            }
        }
    }
}

Just basic HTML parsing there - as you can see I had to make some high risk assumptions (since some of the tables did not have identifiable ids associated with them). Oh and the reason I use Enum.Parse is because WP7's mscorlib does not have TryParse!

Tombstoning

As I mentioned earlier, the app supports tombstoning, and one of the things I had to do was to ensure that all the types I wanted to restore were fully compatible with serialization (which basically means they had to have public properties along other things). Here's the code that handles tombstoning.

protected override void OnNavigatedFrom(System.Windows.Navigation.NavigationEventArgs e)
{
    App.MainViewModel.SaveState();

    base.OnNavigatedFrom(e);
}

protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
{
    App.MainViewModel.RetrieveState();

    base.OnNavigatedTo(e);
}

And here are the implementations in the view model.

public void SaveState()
{
    var appService = PhoneApplicationService.Current;
    
    appService.State.Clear();

    appService.State["MemberId"] = this.MemberId;
    appService.State["CanFetch"] = this.CanFetch;
    
    if (this.CanFetch)
    {
        appService.State["MemberName"] = this.MemberName;
        appService.State["Results"] = this.Results;
        appService.State["ForumTypeSummaries"] = this.ForumTypeSummaries;
        appService.State["PostSummaries"] = this.PostSummaries;
    }
}

public void RetrieveState()
{
    var appService = PhoneApplicationService.Current;

    if (appService.State.ContainsKey("MemberId"))
    {
        this.MemberId = (string)appService.State["MemberId"];
    }

    if (appService.State.ContainsKey("CanFetch"))
    {
        this.CanFetch = (bool)appService.State["CanFetch"];
    }

    if (!this.CanFetch)
    {
        this.CanFetch = true;
        return;
    }

    if (appService.State.ContainsKey("MemberName"))
    {
        this.MemberName = (string)appService.State["MemberName"];
    }

    if (appService.State.ContainsKey("Results"))
    {
        this.Results.Clear();

        foreach (PostInfo item in (IEnumerable<PostInfo>)appService.State["Results"])
        {
            this.Results.Add(item);
        }
    }

    if (appService.State.ContainsKey("ForumTypeSummaries"))
    {
        this.ForumTypeSummaries.Clear();

        foreach (PostSummaryInfo item in 
          (IEnumerable<PostSummaryInfo>)appService.State["ForumTypeSummaries"])
        {
            this.ForumTypeSummaries.Add(item);
        }
    }

    if (appService.State.ContainsKey("PostSummaries"))
    {
        this.PostSummaries.Clear();

        foreach (PostSummaryInfo item in 
          (IEnumerable<PostSummaryInfo>)appService.State["PostSummaries"])
        {
            this.PostSummaries.Add(item);
        }
    }
}

I'll just mention in passing that I made some unsuccessful attempts to serialize the entire view model and it was doomed from the very beginning. I eventually gave up and decided to serialize what I specifically wanted. I guess I could have got it to work but it would have been too much hassle for no return at all, except maybe some artificially boosted self-esteem which I don't care for anyway!

Isolated Storage

The recipients list is stored via isolated storage so your most recent 10 searches will be remembered.

public MainViewModel()
{
    IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication();
    if (!store.FileExists(configPath))
        return;

    using (StreamReader reader = new StreamReader(
      new IsolatedStorageFileStream(configPath, FileMode.Open, store)))
    {
        var lines = reader.ReadToEnd().Trim().Split('\r', '\n').Where(
          s => !String.IsNullOrEmpty(s.Trim('\r', '\n'))).ToList();
        lines.ForEach(line => recentMemberIds.Add(line));
    }
}

private void SaveConfig()
{
    IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication();
    string baseDirectory = System.IO.Path.GetDirectoryName(configPath);
    if (!store.DirectoryExists(baseDirectory))
    {
        store.CreateDirectory(baseDirectory);
    }

    using (StreamWriter writer = new StreamWriter(
      new IsolatedStorageFileStream(configPath, FileMode.Create, store)))
    {
        foreach (var entry in recentMemberIds)
        {
            writer.WriteLine(entry);
        }
    }
}

The built-in support makes it incredibly easy to use isolated storage. Initially, I did consider saving posts and then accumulating them so I can analyze more than 200 posts, but that assumes the user will run it frequently enough that there won't be missing posts (which was very hard to enforce, actually quite impossible). So I gave up and thought 200 is good enough. Some day if Chris increases that limit to 1000, then I'll update the app at that time.

Conclusion

Well that's it. It's been tested reasonably thoroughly but there may be issues, specially on other phone models. The error handling is kinda silent, so if it encounters any errors it won't break down but it won't tell you either. You could try running a Fetch again, or close and re-run the app (although I have never had to do that so far) .

Feel free to submit feedback, criticism, and suggestions as usual.

History

  • January 23rd 2011 - Article first published.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Nish Sivakumar

United States United States
Nish is a real nice guy who has been writing code since 1990 when he first got his hands on an 8088 with 640 KB RAM. Originally from sunny Trivandrum in India, he has been living in various places over the past few years and often thinks it’s time he settled down somewhere.
 
Nish has been a Microsoft Visual C++ MVP since October, 2002 - awfully nice of Microsoft, he thinks. He maintains an MVP tips and tricks web site - www.voidnish.com where you can find a consolidated list of his articles, writings and ideas on VC++, MFC, .NET and C++/CLI. Oh, and you might want to check out his blog on C++/CLI, MFC, .NET and a lot of other stuff - blog.voidnish.com.
 
Nish loves reading Science Fiction, P G Wodehouse and Agatha Christie, and also fancies himself to be a decent writer of sorts. He has authored a romantic comedy Summer Love and Some more Cricket as well as a programming book – Extending MFC applications with the .NET Framework.
 
Nish's latest book C++/CLI in Action published by Manning Publications is now available for purchase. You can read more about the book on his blog.
 
Despite his wife's attempts to get him into cooking, his best effort so far has been a badly done omelette. Some day, he hopes to be a good cook, and to cook a tasty dinner for his wife.

Comments and Discussions

 
GeneralRe: nice one PinmvpNishant Sivakumar4-Feb-11 2:53 
GeneralRe: nice one PinmvpSacha Barber4-Feb-11 7:17 
GeneralRe: nice one PinmvpNishant Sivakumar4-Feb-11 7:28 
GeneralRe: nice one PinmvpSacha Barber4-Feb-11 20:02 
GeneralMy vote of 5 PinmemberPatrick Kalkman1-Feb-11 8:32 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar1-Feb-11 8:42 
GeneralMy vote of 5 PinmemberSChristmas1-Feb-11 5:16 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar1-Feb-11 5:21 
GeneralMy vote of 5 PinmemberMajid Shahabfar29-Jan-11 11:03 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar30-Jan-11 6:13 
GeneralMy vote of 5 PinmemberManas Bhardwaj27-Jan-11 5:03 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar27-Jan-11 5:04 
GeneralMy Vote of 5 PinmemberKhalil Adam27-Jan-11 4:33 
GeneralRe: My Vote of 5 PinmvpNishant Sivakumar27-Jan-11 4:34 
GeneralMy Vote of 5 PinmemberRaviRanjankr26-Jan-11 22:39 
GeneralRe: My Vote of 5 PinmvpNishant Sivakumar27-Jan-11 1:57 
GeneralReally very interesting app PinmemberPavanPareta26-Jan-11 20:14 
GeneralRe: Really very interesting app PinmvpNishant Sivakumar27-Jan-11 1:57 
QuestionTroubles installing the app on my phone PinmemberMarkus Fischbacher26-Jan-11 9:01 
AnswerRe: Troubles installing the app on my phone PinmvpNishant Sivakumar26-Jan-11 9:04 
GeneralRe: Troubles installing the app on my phone PinmemberMarkus Fischbacher26-Jan-11 9:17 
GeneralRe: Troubles installing the app on my phone PinmvpNishant Sivakumar26-Jan-11 9:17 
AnswerRe: Troubles installing the app on my phone PinmvpNishant Sivakumar26-Jan-11 9:17 
GeneralRe: Troubles installing the app on my phone PinmemberMarkus Fischbacher26-Jan-11 9:20 
GeneralRe: Troubles installing the app on my phone PinmvpNishant Sivakumar26-Jan-11 9:50 
GeneralMy vote of 5 PinmvpAbhijit Jana25-Jan-11 21:30 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar26-Jan-11 2:03 
GeneralMy vote of 5 PinmemberYusuf25-Jan-11 8:26 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar25-Jan-11 8:28 
GeneralMy vote of 5 PinmvpAbhinav S25-Jan-11 0:53 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar25-Jan-11 4:33 
GeneralMy vote of 5 PinmemberDon Kackman24-Jan-11 3:19 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar24-Jan-11 3:22 
GeneralRe: My vote of 5 PinmemberDon Kackman24-Jan-11 3:26 
GeneralMy vote of 5 Pinmemberprasad0223-Jan-11 20:23 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar24-Jan-11 2:41 
Generalthanks for sharing - have 5 PinmemberPranay Rana23-Jan-11 18:52 
GeneralRe: thanks for sharing - have 5 PinmvpNishant Sivakumar24-Jan-11 2:40 
GeneralMy vote of 5 PinmvpKunal_Chowdhury23-Jan-11 16:11 
GeneralRe: My vote of 5 PinmvpNishant Sivakumar24-Jan-11 2:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140721.1 | Last Updated 23 Jan 2011
Article Copyright 2011 by Nish Sivakumar
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid