Playing Card Recognition Using AForge.Net Framework

• Download source - 148.61 KB
• Download demo - 3.1 MB

Introduction

Playing card recognition systems can be coupled with a robotic system which acts like a dealer or a human player in a card game, such as blackjack. Implementing this kind of application is also a good example for learning computer vision and pattern recognition.

This article involves binarization, edge detection, affine transformation, blob processing, and template matching algorithms which are implemented in AForge .NET Framework.

Note that this article and this system is based on Anglo-American card decks, it may not work for other card decks. However, this article describes basic methods for detection and recognition of cards. Therefore, recognition algorithm might be changed according to features of the deck that is used.

Here’s a quick video demonstration.

• http://www.youtube.com/watch?v=dui3ftwsuhM

Card Detection

We need to detect card objects on image so that we can proceed with recognition. For detection, we apply some image filters on image for helping detection.

First step, we apply grayscaling on image. Grayscaling is a process that converts a colored image to an 8 bit image. We need to convert colored image to grayscale image so that we can apply binarization on image.

After we convert colored image to grayscale image, we apply binarization on image. Binarization(thresholding) is the process of converting a grayscale image to black & white image. In this article, Otsu’s method is used for global thresholding.

Bitmap temp = source.Clone() as Bitmap; //Clone image to keep original image

FiltersSequence seq = new FiltersSequence();
seq.Add(Grayscale.CommonAlgorithms.BT709);  //First add  GrayScaling filter
seq.Add(new OtsuThreshold()); //Then add binarization(thresholding) filter
temp = seq.Apply(source); // Apply filters on source image

Original Image	Grayscale Image	Binary(Black&White) Image

Since we have binary image, we can proceed with blob processing for detecting cards in image. For blob processing, we use AForge.Net BlobCounter class.The class counts and extracts standalone objects in images using connected components labeling algorithm.

//Extract blobs from image whose size width and height larger than 150
BlobCounter extractor = new BlobCounter();
extractor.FilterBlobs = true;
extractor.MinWidth = extractor.MinHeight = 150;
extractor.MaxWidth = extractor.MaxHeight = 350;
extractor.ProcessImage(temp);

After executing the code above, BlobCounter class filters (removes) blobs whose width or height that isn’t between [150,350] pixels. This helps us discriminate cards from other objects(if there’s any) in image. These filter values can be changed according to the test environment. Suppose that, if distance between ground and camera is bigger, then cards will be smaller in image. In that case, we shall change min, max width & height values.

Now, we can get information (edge points, rectangles, center point, area, fullness, …etc.) of all blobs by calling extractor.GetObjectsInformation(). However, we only need edge points of blob to find corner points of rectangle. For finding corner points, we invoke PointsCloud.FindQuadriteralCorners function with list of edge points.

foreach (Blob blob in extractor.GetObjectsInformation())
{
 //Get Edge points of card
 List< IntPoint > edgePoints = extractor.GetBlobsEdgePoints(blob);
 //Find corners of card on source image from edge points
 List< IntPoint > corners =  PointsCloud.FindQuadrilateralCorners(edgePoints);
}

Painting Edge Points On Image	Finding Corner Points of Each Card

After finding corners of cards, now we can transform area between corners from source image to a rectangular image, so we can extract and get card images.

As it can be seen from images, cards can be placed horizontally. It's very easy to detect if a card is placed horizontally or not. We know that the height of the card is bigger than the width of the card, so we can use this information for finding out if the card is placed. If width of extracted (transformed) image is bigger than its height, then card is placed horizontal. We use RotateFlip function to rotate card.

Note that all cards must be the same size for recognition. However, size of cards might be different because of camera angle. This might produce problems at recognition step. For preventing these problems, we resize all transformed card images as 200 x 300 (pixel).

//Will be used transform(extract) cards on source image
QuadrilateralTransformation quadTransformer = new QuadrilateralTransformation();
//Will be used resize(scaling) cards
ResizeBilinear resizer = new ResizeBilinear(CardWidth, CardHeight);

foreach (Blob blob in extractor.GetObjectsInformation())
{
     //Get Edge points of card
     List<IntPoint> edgePoints = extractor.GetBlobsEdgePoints(blob);
     //Find corners of card on source image from edge points
     List<IntPoint> corners =  PointsCloud.FindQuadrilateralCorners(edgePoints);
     Bitmap cardImg = quadTransformer.Apply(source); //Extract(transform) card image

     if (cardImg.Width > cardImg.Height) //If card is positioned horizontally
          cardImg.RotateFlip(RotateFlipType.Rotate90FlipNone); //Rotate
     cardImg =  resizer.Apply(cardImg); //Normalize card size
       .....
}

So far, we have found corners of each card in the source image, extracted each card from image and normalized their sizes. Now, we can proceed with the recognition step.

Card Recognition

There are several techniques for recognition. Recognition in this system is based on features of deck cards (such as shapes on cards) and also contains template matching. Suit and rank of card are recognized separately. We have enumeration for suits and ranks as follows:

public enum Rank
{
    NOT_RECOGNIZED = 0,
    Ace = 1,
    Two,
    Three,
    Four,
    Five,
    Six,
    Seven,
    Eight,
    Nine,
    Ten,
    Jack,
    Queen,
    King
}
public enum Suit
{
    NOT_RECOGNIZED = 0,
    Hearts,
    Diamonds,
    Spades,
    Clubs
}

We also create the following Card class for representing recognized cards. The class contains rank of card, suit of card, image of card, corner points of card on source image.

public class Card
{
    //Variables
    private Rank rank; //Rank of card
    private Suit suit; //Suit of card
    private Bitmap image; //Extracted(transformed) image of card
    private Point[] corners ;//Corner points of card on source image

    //Properties
    public Point[] Corners
    {
        get { return this.corners; }
    }
    public Rank Rank
    {
        set { this.rank = value; }
    }
    public Suit Suit
    {
        set { this.suit = value; }
    }
    public Bitmap Image
    {
        get { return this.image; }
    }
    //Constructor
    public Card(Bitmap cardImg, IntPoint[] cornerIntPoints)
    {
        this.image = cardImg;

        //Convert AForge.IntPoint Array to System.Drawing.Point Array
        int total = cornerIntPoints.Length;
        corners = new Point[total];

        for(int i = 0 ; i < total ; i++)
        {
            this.corners[i].X = cornerIntPoints[i].X;
            this.corners[i].Y = cornerIntPoints[i].Y;
        }
    }
}

Suit Recognition

Suits in a standard playing card deck are spades, clubs, diamonds and hearts. It's known that diamonds and hearts are red while clubs and spades are black. Another thing we know about suits is that the width of diamond is bigger than the width of hearts, and the width of clubs is bigger than spades. These two features help us recognize the suit of card.

Color Recognition

First we start with color recognition. Color recognition will help us to eliminate 2 suits. For color recognition, we analyze suit at top-right of card image.

public Bitmap GetTopRightPart()
{
    if (image == null)
        return null;
    Crop crop = new Crop(new Rectangle(image.Width - 37, 10, 30, 60));

    return crop.Apply(image);
}

After cropping the top-right part of card image, we get 30x60 image. However, as you can see, cropped image contains both rank and suit. Since analyzing suit part will produce more accurate result, we crop the bottom half again. As a result, we get 30x30 image to analyze.

Now, we can iterate through each pixel and count the total number of red pixels and total number of black pixels. If red component of a pixel is bigger than the sum of blue component and green component, that pixel is considered red colored. If red, green, blue components of a pixel is smaller than 50 and red isn’t bigger than sum of blue component and green component, that pixel is considered black pixel.

char color = 'B';
//Lock Bits for processing
BitmapData imageData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height),
    ImageLockMode.ReadOnly, bmp.PixelFormat);
int totalRed = 0;
int totalBlack = 0;

unsafe
{
    //Count red and black pixels
    try
    {
       UnmanagedImage img = new UnmanagedImage(imageData);

       int height = img.Height;
       int width = img.Width;
       int pixelSize = (img.PixelFormat == PixelFormat.Format24bppRgb) ? 3 : 4;
       byte* p = (byte*)img.ImageData.ToPointer();

       // for each line
       for (int y = 0; y < height; y++)
       {
           // for each pixel
           for (int x = 0; x < width; x++, p += pixelSize)
           {
               int r = (int)p[RGB.R]; //Red pixel value
               int g = (int)p[RGB.G]; //Green pixel value
               int b = (int)p[RGB.B]; //Blue pixel value

               if (r > g + b) //If red is bigger then total of green and blue
                  totalRed++;  //then its red

               if (r <= g + b && r < 50 && g < 50 && b < 50) //If all less than 50
                  totalBlack++; //then its black
           }
       }
    }
    finally
    {
       bmp.UnlockBits(imageData); //Unlock
    }
}
if (totalRed > totalBlack) //If red is dominant
    color = 'R'; //Set color as Red
return color;

Total Red = 82 Total Black = 0	Total Red = 0 Total Black = 60

Note that .NET Bitmap GetPixel() function works slow, for that reason we use pointer to iterate through pixels.

To Distinguish Between Face Cards And Non-Face Cards

After color recognition, we determine card is a face or a non-face card. Face cards are King, Queen, Jack. There is a distinct feature between face cards and non-face cards. Non-face(number) cards have big suit shapes on them as many as rank of card while face cards have faces on them. This feature of non-face cards makes them easy to recognize. Instead of running template matching algorithm on them, we can simply find one big suit shape on card and analyze it. This makes recognition step faster for non-face cards.

To find out if a card is a face card or a non-face card is quite easy. Face cards have big image on card while non-face cards don't. If we apply edge detection and blob processing on card and find the biggest blob on card, we can say that it's a face card according to the biggest blob size.

private bool IsFaceCard(Bitmap bmp)
{
   FiltersSequence commonSeq = new FiltersSequence();
   commonSeq.Add(Grayscale.CommonAlgorithms.BT709);
   commonSeq.Add(new BradleyLocalThresholding());
   commonSeq.Add(new DifferenceEdgeDetector());

   Bitmap temp = this.commonSeq.Apply(bmp);
   ExtractBiggestBlob extractor = new ExtractBiggestBlob();
   temp = extractor.Apply(temp); //Extract biggest blob

   if (temp.Width > bmp.Width / 2)  //If width is larger than half width of card
       return true; //Its a face card

   return false;  //It is not a face card
}

So we consecutively apply grayscaling, local thresholding and edge detection on card image. Note that we use local thresholding instead of global thresholding to eliminate bad illumination problems.

Original Card Image	Grayscaling	Bradley Local          Thresholding	Edge Detection	Extracting     Biggest Blob

Original Card Image	Grayscaling	Bradley Local Thresholding	Edge Detection	Extracting Biggest Blob

As can be seen, biggest blobs in face cards are almost card size, and this feature allows them to be differentiated.

It’s mentioned before, we’ll use individual recognition techniques for face cards and non-face cards because of some performance considerations. For recognizing suits on non-face card, we extract the biggest blob on card and analyze its width and card color.

private Suit ScanSuit(Bitmap suitBmp, char color)
{
     Bitmap temp = commonSeq.Apply(suitBmp);
     //Extract biggest blob on card
     ExtractBiggestBlob extractor = new ExtractBiggestBlob();
     temp = extractor.Apply(temp);  //Biggest blob is suit blob so extract it
     Suit suit = Suit.NOT_RECOGNIZED;

     //Determine type of suit according to its color and width
     if (color == 'R')
        suit = temp.Width >= 55 ? Suit.Diamonds : Suit.Hearts;
     if (color == 'B')
        suit = temp.Width <= 48 ? Suit.Spades : Suit.Clubs;

     return suit;
}

Width = 52 pixels	Width = 43 pixels	Width = 47 pixels	Width = 57 pixels

These measures differ at most 2 pixels. Generally all are the same size because we resize cards as 200x300 pixels at the detection step.

In face cards, there’s no big suit images on card like non-face cards have. Face cards only have small suit images on corners. That’s why we crop top-right part of card images and will apply template matching for recognizing suit. Binary template image for each suit are added in project resources.

AForge.NET also provides a class for template matching called ExhaustiveTemplateMatching. The class implements exhaustive template matching algorithm, which performs complete scan of source image, comparing each pixel with corresponding pixel of template. Although performance of the algorithm isn’t good, we search for a template in a small rectangle ( 30x60).

private Suit ScanFaceSuit(Bitmap bmp, char color)
{
     Bitmap clubs, diamonds, spades, hearts; //Suit Templates

     //Load Templates From Resources ,
     clubs = PlayingCardRecognition.Properties.Resources.Clubs;
     diamonds = PlayingCardRecognition.Properties.Resources.Diamonds;
     spades = PlayingCardRecognition.Properties.Resources.Spades;
     hearts = PlayingCardRecognition.Properties.Resources.Hearts;

     //Initialize templateMatching class with 0.8 similarity threshold
     ExhaustiveTemplateMatching templateMatching = new ExhaustiveTemplateMatching(0.8f);
     Suit suit = Suit.NOT_RECOGNIZED;

     if (color == 'R') //If card is red then it can be hearts or diamonds
     {
        if (templateMatching.ProcessImage(bmp, hearts).Length > 0)
           suit = Suit.Hearts; //Check If template matches for hearts
        if (templateMatching.ProcessImage(bmp, diamonds).Length > 0)
           suit = Suit.Diamonds; //Check If template matches for diamonds
     }
     else //If its black
     {
        if (templateMatching.ProcessImage(bmp,spades).Length > 0)
            suit = Suit.Spades; //Check If template matches for spades
        if (templateMatching.ProcessImage(bmp, clubs).Length > 0)
            suit = Suit.Clubs;//Check If template matches for clubs
     }
     return suit;
  }

Of course, template doesn’t 100% match with sample. That’s why we set our similarity threshold value as 0.8 (80%).

Rank Recognition

Rank recognition is similar to suit recognition. We recognize face cards and non-face cards separately. Because non-face cards can be recognized by counting suit blobs on card image, no template matching is required, simple image filters will do the job. This makes recognition process simple for non-face cards. Unlike template matching, it doesn’t take too much processing time.

Following ScanRank function filters small blobs (smaller than 30 pixels width or height) and counts remaining blobs.

private Rank ScanRank(Bitmap cardImage)
{
    Rank rank = Rank.NOT_RECOGNIZED;

    int total = 0;
    Bitmap temp = commonSeq.Apply(cardImage); //Apply filters on image
    BlobCounter blobCounter = new BlobCounter();
    blobCounter.FilterBlobs = true;
    //Filter blobs whose width or height less than 30 pixels
    blobCounter.MinHeight = blobCounter.MinWidth = 30;
    blobCounter.ProcessImage(temp);

    total = blobCounter.GetObjectsInformation().Length; //Get total number
    rank = (Rank)total; //Convert it to Rank

    return rank;
}

As a result, non-face cards are recognized without template matching algorithm or OCR. However, for face cards, we again use template matching for recognizing rank of card.

private Rank ScanFaceRank(Bitmap bmp)
{
     Bitmap j, k, q; //Face Card Character Templates
     //Load Templates From Resources ,
     j = PlayingCardRecognition.Properties.Resources.J;
     k = PlayingCardRecognition.Properties.Resources.K;
     q = PlayingCardRecognition.Properties.Resources.Q;


     //Initialize template matching class with 0.75 threshold
     ExhaustiveTemplateMatching templateMatchin =
               new ExhaustiveTemplateMatching(0.75f);
     Rank rank = Rank.NOT_RECOGNIZED;

     if (templateMatchin.ProcessImage(bmp, j).Length > 0) //If Jack matches
         rank = Rank.Jack;
     if (templateMatchin.ProcessImage(bmp, k).Length > 0)//If King matches
         rank = Rank.King;
     if (templateMatchin.ProcessImage(bmp, q).Length > 0)//If Queen matches
         rank = Rank.Queen;

     return rank;
}

This time we set our similarity threshold as 0.75(75%) because it’s more difficult to recognize ranks.

Known Issues

This implementation, as it is, only recognizes playing cards that are separated by each other. Another known issue is that bad light conditions may lead to wrong recognitions.

Conclusion

Most of the image routines used in this article are based on AForge.NET framework. AForge.NET is really a cool framework that provides plenty of features for developers who works in fields of computer vision and machine learning. It is also very easy to use.

This work can be improved such as recognizing cards even when they are not separated. Another improvement could be using this system by an AI BlackJack player.

History

• 7^th October, 2011: Initial post