That usually depend on what you mean with 'valid filename', e.g. Should its path be formally correct? Should it be an existing file? Should it be a new file? ("Should we buy a new guitar?"
Depending on your needs you have to build your validation function (it might be very simple, e.g. just trying to open the file). Once you have the validation function you have only to keep track of failed attempts and close the program if this number becomes 3.
I would use
FILE fp = NULL;
for (i=0; i<3; i++)
printf("\n\nEnter filename :" );
gets( fname );
fp = fopen (fname, "r");
printf("Cannot open %s for reading \n", fname );
if ( ! fp ) return -1;