In Windows Forms When I Am Clicking Backspace In Textbox Error Msg Is Coming

Why should not?!
Your code checks for digit-only keys, backspace is not one of them...
You have three options
1. Use NumericUpDown[^] control (if that fits in)
2. Use the Validating[^] event to check the text-box value
3. Write your code correctly...Check this sample: https://msdn.microsoft.com/en-us/library/system.windows.forms.control.keypress(v=vs.110).aspx[^]

Use the SelectionStart and SelectionLength properties.
If Selection Length is zero, then the Selection start is the index of the cursor position in your text string, so a string.Remove will remove the character for you.
If it is non-zero, there is a selection, and you need to remove the whole thing:

int selStart = SelectionStart;
int selLength = SelectionLength;
if (char.IsControl(e.KeyChar))
    {
    if (e.KeyChar == '\b')
        {
        // Backspace
        if (selLength > 0)
            {
            myTextBox.Text = myTextBox.Text.Remove(selStart, selLength);
            SelectionLength = 0;
            SelectionStart = selStart;
            }
        else if (selStart > 0)
            {
            myTextBox.Text = myTextBox.Text.Remove(selStart - 1, 1);
            SelectionStart = selStart - 1;
            }
        e.Handled = true;
        }
    }

Part of the power that .NET and C# give you is the ability to sub-class the built-in Controls, and make them behave the way you want.

Usually, you'll add a Component to your WinForms Project, and then modify the declaration of the Class Visual Studio generates for you so that you inherit from the Control you wish to modify. Consider this sub-classed TextBox that will accept only digits, and the backspace character:

using System;
using System.ComponentModel;
using System.Windows.Forms;

namespace YourNameSpace
{
    // Visual Studio wrote this:
    //public partial class DigitOnlyTextBox : Component

    // changed so we inherit from TextBox
    public partial class DigitOnlyTextBox : TextBox
    {
        public DigitOnlyTextBox()
        {
            InitializeComponent();
        }

        public DigitOnlyTextBox(IContainer container)
        {
            container.Add(this);

            InitializeComponent();

            AcceptsReturn = false;
            AcceptsTab = false;
            Multiline = false;
        }

        private char backspace = Convert.ToChar(Keys.Back);

        // not used in code shown here
        //private char period = '.';
        //private char comma = ',';
        //private int periodCount = 0;

        protected override void OnKeyPress(KeyPressEventArgs e)
        {
            char key = e.KeyChar;

            if (Char.IsDigit(key) || key == backspace)
            {
                base.OnKeyPress(e);
            }
            else
            {
                e.Handled = true;
            }
        }
    }
}

Now where things get "tricky" are when you want to prevent the user from doing things like:

1. if you are allowing periods: stop the user from entering more than one period, and stop the user from putting a period at start, or end, of the string.

2. if you are allowing commas: stop the user from leaving a comma at the beginning, or end, of the string

3. entering two or more periods, or commas, in a row

And, what about the user possibly pasting in content on the Clipboard into the TextBox ?

The possible complexity of the code handling what the user inputs, and making sure the input is acceptable is one reason many people just use the NumericUpDownControl in WinForms. Or, use a MaskedTextBox with a custom RegEx.