N,
I tested your VI using TestStand 3.5 and LabVIEW 8.2.
First of all, as specified before in this thread, the Windows needs to be set to dialog for modal behavior (VI Properties>>Window Appearance). Secondly, the "IsFrontmost" check is not necessary and actually obstructs the true behavior of Set Modal. The term "Modal" does not simply mean "in front of"; it also indicates that the parent application cannot be focused outside of the child application.
Additionally, since you hadn't attached the sequence file, I do need to point out that the checkbox "Show VI Front Panel When Called" should not be checked.
For an example of the modal behavior, please refer to the TestStand install directory (<TestStand>\Examples\ModalDialogs).
Finally, I do need to point out that once the modal window is actually working properly, you will run into an issue with LabVIEW 8.0/8.2 in which the window is not initially "focused". When and if you reach that point, refer to this discussion forum thread:
http://forums.ni.com/ni/board/message?board.id=330&message.id=10473&view=by_date_ascending&page=2
Thanks,
Andy McRorie
NI R&D