If your goal is to:
1) let the run-time user create a single instance of any one type of form
2) not let the run-time user create another instance of the same type
Keep a reference to the Form instances the user, or you, create: use a List or whatev3er other data structure you want. If the user closes an instance, update your List.
If you want to use Application.OpenForms to monitor which Forms are active, you'll need to cast it to IEnumerable, like this:
bool hasFormType = Application.OpenForms.Cast<Form>().Any((Form frm) => frm is Form2);
But, it's much better not to give the user the option of creating another instance of the same type.
It's called "defensive programming," and, the idea is to stop mistakes happening, rather than deal with handling choices that should not be available