Start by looking at the problem:
X = Y*Y + Z*Z, where X is a perfect square.
I'd probably start by identifying all the perfect squares less than X, and then start looking for "combinations", by taking the largest, adding the lowest and seeing if it matches.
If it does, you're done.
If it's lower than X, try the next lowest and repeat.
If it's greater than X, try the next highest and repeat.
If Y==Z it doesn't exist.
But this is your homework, so I won't give you any code!