How to make the focus on the form itself and not on a button?

See more:

C#

.NET

I have this form and on formMain's KeyDown event it detects the arrows to play the board game, but when I press the Restart button which I made a panel for to get rid of it taking focus when I start the game, now the game starts correctly as KeyDown event for the form fires as it has focus when the button Restart is in a panel, but when I click restart or it gets focus it cant lose that focus even when I press on the form outside the panel so I be able to play the game with the arrows with the formMain's KeyDown, when the button restart gets the control I can't make it lose it even with tabStop=false;

I want the form to get focus so it detect the KeyDown event, I can fix it with PreviewKeyDown but I want a legit solution.

https://prnt.sc/SAx3YnbLr_hm[^]

What I have tried:

private void panel1_Enter(object sender, EventArgs e)
        {
            Focus();
            Select();
        }

 Graphics g;
        private void FormMain_Shown(object sender, EventArgs e)
        {
            g = CreateGraphics();
            drawAll();


            

            buttonRestart.TabStop = false;

            ActiveControl = label1;// hidden label as its Visible=false


            Focus();
            Select();
        }

        private void buttonRestart_Enter(object sender, EventArgs e)
        {
            Focus();
            Select();
        }