First of all I would like to recommend using
select case
select case oper
case '+'
txtoutput.Text = first + second
case '-'
txtoutput.Text = first - second
...
end select
this makes the code shorter and also better readable. Also you might consider to lose the magic pushbutton. It would be more easy to test if you would simply create a function like:
public function Calculate(byval first as string, byval second as string, byval oper as string) as string
You could easily test this function, even without a user interface just simply in the direct window.
The actions of the buttons also do look a like and you could simply create a more generic implementation but this is up to you.
The reason why it isn't working is because your code is not completely properly written. You get a string value from the textbox and use it as number in various calculations. Basic has always been a very forgiving language and still looks the other way in most cases. You could try to wrap the calculation something like
Convert.ToString(System.Math.Sqrt(first))
. Also remember that to execute the square root you must enter the number, click s and then click equal to work.
Hopefully this will get you on your way. If you encounter any more problems, just let me know.
Good luck!