LabVIEW 5.1.1 is not officially supported under Windows XP. Like many software/hardware products on the market some before certain dates simply do not work with the newest versions of OS's. My suggestion would be to migrate to LabVIEW 6.1 to resolve your issues or revert back to an older OS.
Thanks but I've found what was wrong - user account I'm working on has limitations. After setting administrator's rights for account problem disapeared (except annoying box "wrong os").
I ran it under compatability for Windows 95 and that got rid of the "wrong os" box, but I still have problems with some of the functions [best curve fit in particular] not having destinations attached to them, and consequently failing.