Well, LabVIEW can only look at the output once the file dialog returns. The file dialog is a system service. How would you suggest it should handle the situation when a user accidentally types a name that already exists in a huge folder? Should it just beep at him? Lock the "Open" button? I think it should be up to your own code to decide if the filename is acceptable. Popping a new file dialog if the name is not acceptable seems like the natural thing to do. 🙂
If you have unlimited time to waste, I am sure you could make your own custom "file dialog" by designing a VI that looks and acts identical to the system VI, but constantly compares the entered text with the list of files and e.g. locks the OK button, beeps, and flashes a red LED if the entered name already exists. :o.