Click here to Skip to main content
11,633,978 members (80,523 online)
Click here to Skip to main content

CP Vanity Lite

, 19 Nov 2010 CPOL 83.2K 657 22
Rate this:
Please Sign up or sign in to vote.
This is a lightweight version of Luc Pattyn's popular CP Vanity application

Figure 1 : CP Vanity Lite - the main dialog

Figure 2 : The exported CSV file in Excel

Introduction

This is a light version of Luc Pattyn's ever popular CP Vanity application. Luc tired of frequently having to modify the parsing code due to unavoidable site changes and has not updated his application since the last major site-update (a few weeks ago). That's what led me to write this app which is a miniature version of his application. Unlike CP Vanity, CP Vanity Lite does not really go into the details, instead it will only fetch the total reputation for members that are most likely to have high reputation scores. I use the same approach Luc took, namely to get the top members based on post-count and the top members based on article count, and then fetch the reputation score for each of those members. I fetch the top 125 in each category but because of certain members being in both groups, you'll find that the total number of members for whom rep scores are fetched is less than 250. In addition to the total score it also shows a daily score average that's calculated based on the total score and the number of days since a member signed up.

A word of warning: The application fetches and parses HTML from hardcoded URLs. If the URLs or the HTML content undergo changes, then the parsing code will break. Until a time in future when CP offers a web-service that will return this information, the premise on which this application has been written will be shaky. This is the same for similar applications written by myself and folks like Luc Pattyn.

Using CP Vanity Lite

There's not much to using the app, and all you do is run the application and wait for the UI to populate. The title bar will show fetching while the HTTP fetches are performed, and once all the data is available, it'll say completed. The UI is a simple ListView that's sortable on the Reputation Score column. Every time you click on the header, the sort is toggled between ascending and descending, which is quite handy if you get tired of seeing CG's name right there on top! The data is populated asynchronously via a worker thread, so the app will be responsive throughout the fetch-period. You can sort the data even before all the data has been fetched, but keep in mind that newly added rows will not be sorted although you can keep sorting as many times as you want to. Once all the data is fetched, the Export as CSV button is enabled, and you can save a CSV file that you can then open in an app like Excel to do all kinds of fancy data manipulations and charts. Take a look at Figure 2 and you'll see how far ahead of the competition CG is, he's got more than John and Pete combined! If you run the app periodically and save the CSV files, you set yourself up nicely for some date-based data parsing to determine changes in scores and how fast someone's catching up (if you are the statistically curious type).

The code

The html scraping is done in a single class called RepScoreScraper. The code will block, so it's up to calling code to use it from a worker thread or via some async/task pattern that won't block the main thread. The top potential scoring members ids are fetched by querying the Who's who pages for top message posters and for top article authors. The information is extracted using regular expressions. (I've wrapped the code blocks to prevent scrolling, the actual source code does not wrap this crazily)

private Regex memberNumberRegex = new Regex("Member No. (\\d*)");
private Regex repScoreRegex = new Regex(
  "<span id=\"ctl00_.*?_TotalRep\" class=\"large-text\">([\\s\\S]*?)</span>");
private Regex displayNameRegex = new Regex(
  "<h2 id=\"ctl00_.*?_P_Name\">([\\s\\S]*?)</h2>");
private Regex memberSinceRegex = new Regex("Member since (.+)\n");

Note that I have snipped the URLs in the code snippet here to prevent horizontal scrolling.

public void StartScraping()
{
    string[] ml_obs = new[] { "ArticleCount", "MessageCount" };
    HashSet<int> ids = new HashSet<int>();

    for (int j = 0; j < ml_obs.Length; j++)
    {
        for (int page = 1; page <= 5; page++)
        {
            string url = String.Format(
                "**SNIPPED**?ml_ob={0}&mgtid=-1&mgm=False&pgnum={1}", 
                ml_obs[j], page);
            string html = GetHttpPage(url, timeOut);

            var memberNumberMatches = memberNumberRegex.Matches(html);
            var repScoreMatches = repScoreRegex.Matches(html);
            var displayNameMatches = displayNameRegex.Matches(html);
            var memberSinceMatches = memberSinceRegex.Matches(html);

            if (memberNumberMatches.Count == repScoreMatches.Count 
              && memberNumberMatches.Count == displayNameMatches.Count 
              && memberNumberMatches.Count == memberSinceMatches.Count)
            {
                for (int i = 0; i < memberNumberMatches.Count; i++)
                {
                    int id = -1;
                    double score = -1.0;
                    double scorePerDay = 0.0;

                    DateTime memberSince = ParseDateTime(memberSinceMatches[i].Value);

                    if (memberNumberMatches[i].Groups.Count == 2 && 
                      Int32.TryParse(memberNumberMatches[i].Groups[1].Value, out id) &&
                      repScoreMatches[i].Groups.Count == 2 && 
                      Double.TryParse(repScoreMatches[i].Groups[1].Value, out score) &&
                      displayNameMatches[i].Groups.Count == 2)
                    {
                        if (!ids.Contains(id))
                        {
                            ids.Add(id);

                            if (memberSince != DateTime.MinValue)
                            {
                                scorePerDay = Math.Round(
                                  score / (DateTime.Now - memberSince).TotalDays, 2);                                
                            }

                            var handler = MemberInfoScraped;
                            if (handler != null)
                            {
                                handler(this, new RepScoreScraperEventArgs() 
                                { 
                                    Id = id, 
                                    DisplayName = StripOffHtml(
                                      displayNameMatches[i].Groups[1].Value.Trim()), 
                                    ReputationScore = (int)score,
                                    DailyAverage = scorePerDay
                                });
                            }
                        }
                    }
                }
            }
        }
    }

    var finishedHandler = ScrapeFinished;
    if (finishedHandler != null)
    {
        finishedHandler(this, EventArgs.Empty);
    }
}

That's it. If you run into any problems, please use the forum below to report those and I'll do my best to fix them. Thank you.

Acknowledgements

  • RaviRanjankr - For beta testing the application and providing useful feedback that led me to improving and simplifying the fetch code.

History

  • November 17, 2010 : First published
  • November 19, 2010
    • Added rank and daily average columns.
    • You can now refresh the scores.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Nish Nishant
United States United States
Nish Nishant is a Software Architect/Consultant based out of Columbus, Ohio. He has over 15 years of software industry experience in various roles including Lead Software Architect, Principal Software Engineer, and Product Manager. Nish is a recipient of the annual Microsoft Visual C++ MVP Award since 2002 (13 consecutive awards as of 2014).

Nish is an industry acknowledged expert in the Microsoft technology stack. He authored
C++/CLI in Action for Manning Publications in 2005, and had previously co-authored
Extending MFC Applications with the .NET Framework for Addison Wesley in 2003. In addition, he has over 140 published technology articles on CodeProject.com and another 250+ blog articles on his
WordPress blog. Nish is vastly experienced in team management, mentoring teams, and directing all stages of software development.

Contact Nish : You can reach Nish on his google email id voidnish.

Website and Blog

You may also be interested in...

Comments and Discussions

 
GeneralMy vote of 5 Pin
Amir Mohammad Nasrollahi13-Aug-13 9:34
professionalAmir Mohammad Nasrollahi13-Aug-13 9:34 
QuestionMy vote of 5 Pin
Mike Hankey17-Apr-12 3:05
memberMike Hankey17-Apr-12 3:05 
GeneralMy vote of 5 Pin
ashishkumar00831-May-11 12:32
memberashishkumar00831-May-11 12:32 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar31-May-11 13:07
mvpNishant Sivakumar31-May-11 13:07 
GeneralFinally Pin
thatraja4-Apr-11 4:34
mvpthatraja4-Apr-11 4:34 
GeneralRe: Finally Pin
Nishant Sivakumar4-Apr-11 4:39
mvpNishant Sivakumar4-Apr-11 4:39 
GeneralMy vote of 5 Pin
prasad0217-Dec-10 1:10
memberprasad0217-Dec-10 1:10 
GeneralMy vote of 5 Pin
Hiren Solanki29-Nov-10 0:03
memberHiren Solanki29-Nov-10 0:03 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar1-Dec-10 2:47
mvpNishant Sivakumar1-Dec-10 2:47 
GeneralRe: My vote of 5 Pin
Hiren Solanki1-Dec-10 2:50
memberHiren Solanki1-Dec-10 2:50 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar1-Dec-10 2:53
mvpNishant Sivakumar1-Dec-10 2:53 
GeneralMy vote of 5 Pin
Dalek Dave28-Nov-10 23:49
memberDalek Dave28-Nov-10 23:49 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar1-Dec-10 2:46
mvpNishant Sivakumar1-Dec-10 2:46 
GeneralMy vote of 5 Pin
Daniel Vaughan22-Nov-10 22:37
mvpDaniel Vaughan22-Nov-10 22:37 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar23-Nov-10 2:49
mvpNishant Sivakumar23-Nov-10 2:49 
GeneralMy vote of 5 Pin
DaveAuld22-Nov-10 0:17
memberDaveAuld22-Nov-10 0:17 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar22-Nov-10 2:20
mvpNishant Sivakumar22-Nov-10 2:20 
GeneralMy vote of 5 Pin
Pete O'Hanlon21-Nov-10 9:22
mvpPete O'Hanlon21-Nov-10 9:22 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar21-Nov-10 11:52
mvpNishant Sivakumar21-Nov-10 11:52 
GeneralMy vote of 5 Pin
Abhinav S20-Nov-10 5:21
memberAbhinav S20-Nov-10 5:21 
GeneralRe: My vote of 5 Pin
Nishant Sivakumar20-Nov-10 5:38
mvpNishant Sivakumar20-Nov-10 5:38 
GeneralRe: My vote of 5 Pin
Abhinav S20-Nov-10 5:56
memberAbhinav S20-Nov-10 5:56 
GeneralWoohoo! Pin
Bassam Abdul-Baki19-Nov-10 15:48
memberBassam Abdul-Baki19-Nov-10 15:48 
GeneralRe: Woohoo! Pin
Nishant Sivakumar20-Nov-10 2:40
mvpNishant Sivakumar20-Nov-10 2:40 
GeneralRe: Woohoo! Pin
Bassam Abdul-Baki20-Nov-10 3:24
memberBassam Abdul-Baki20-Nov-10 3:24 
GeneralRe: Woohoo! Pin
Nishant Sivakumar20-Nov-10 3:48
mvpNishant Sivakumar20-Nov-10 3:48 
GeneralIt's so Fetching .... Pin
Chris Meech19-Nov-10 5:59
memberChris Meech19-Nov-10 5:59 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:02
mvpNishant Sivakumar19-Nov-10 6:02 
GeneralRe: It's so Fetching .... Pin
Chris Meech19-Nov-10 6:21
memberChris Meech19-Nov-10 6:21 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:24
mvpNishant Sivakumar19-Nov-10 6:24 
GeneralRe: It's so Fetching .... Pin
Chris Meech19-Nov-10 6:31
memberChris Meech19-Nov-10 6:31 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:33
mvpNishant Sivakumar19-Nov-10 6:33 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 13:45
mvpNishant Sivakumar19-Nov-10 13:45 
GeneralRe: It's so Fetching .... Pin
Chris Meech21-Nov-10 10:57
memberChris Meech21-Nov-10 10:57 
GeneralRe: It's so Fetching .... Pin
Bassam Abdul-Baki19-Nov-10 6:16
memberBassam Abdul-Baki19-Nov-10 6:16 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:18
mvpNishant Sivakumar19-Nov-10 6:18 
GeneralRe: It's so Fetching .... Pin
Bassam Abdul-Baki19-Nov-10 6:22
memberBassam Abdul-Baki19-Nov-10 6:22 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:25
mvpNishant Sivakumar19-Nov-10 6:25 
GeneralRe: It's so Fetching .... Pin
Bassam Abdul-Baki19-Nov-10 6:28
memberBassam Abdul-Baki19-Nov-10 6:28 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:30
mvpNishant Sivakumar19-Nov-10 6:30 
GeneralRe: It's so Fetching .... Pin
Bassam Abdul-Baki19-Nov-10 6:31
memberBassam Abdul-Baki19-Nov-10 6:31 
GeneralRe: It's so Fetching .... Pin
Nishant Sivakumar19-Nov-10 6:33
mvpNishant Sivakumar19-Nov-10 6:33 
GeneralRe: It's so Fetching .... Pin
Bassam Abdul-Baki19-Nov-10 6:35
memberBassam Abdul-Baki19-Nov-10 6:35 
GeneralRe: It's so Fetching .... Pin
aspdotnetdev19-Nov-10 11:30
memberaspdotnetdev19-Nov-10 11:30 
GeneralNice Tool Pin
Rutvik Dave19-Nov-10 4:36
memberRutvik Dave19-Nov-10 4:36 
GeneralRe: Nice Tool Pin
Nishant Sivakumar19-Nov-10 6:19
mvpNishant Sivakumar19-Nov-10 6:19 
GeneralRe: Nice Tool Pin
Nishant Sivakumar19-Nov-10 13:45
mvpNishant Sivakumar19-Nov-10 13:45 
GeneralRe: Nice Tool Pin
Rutvik Dave20-Nov-10 20:17
memberRutvik Dave20-Nov-10 20:17 
GeneralCG Pin
Bassam Abdul-Baki19-Nov-10 2:01
memberBassam Abdul-Baki19-Nov-10 2:01 
GeneralRe: CG Pin
Nishant Sivakumar19-Nov-10 2:12
mvpNishant Sivakumar19-Nov-10 2:12 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150728.1 | Last Updated 19 Nov 2010
Article Copyright 2010 by Nish Nishant
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid