The Good News is that I've completely solved the problem! The Bad News is that it was A Long and Winding Road (or A Maze of Twisty Little Passages, All Different).
Initial attempt -- PXI 8176 controller (old enough that it cannot accept the BIOS upgrade that puts RT Boot Mode as a BIOS option), LabVIEW 2010 and 2011 (32-bit) on a Windows 7 (64-bit) platform, MAX 5.0.
Built Desktop PC Utility USB Drive. [In retrospect, which I didn't realize until I just typed it into this Message, this was a Big Fat Hint of Trouble Ahead]. With this, I was able to reformat my 8176 (removing LabVIEW 7.0 RT) and could use it to boot into Safe Mode for installing software. I think MAX could find it, but it called it a "Generic Desktop PC", and changed its icon to that of a PC Tower (not a PXI chassis). It also set the Ethernet on the PXI to DHCP, rather than leaving it unconfigured (0.0.0.0), so I had to fiddle with my desktop NIC to be able to talk to it.
I eventually was able to install software (LabVIEW 2011 RT), and was able to change the Network address to a static 10-dot address (I changed my NIC back to this subnet, as well).
When I went into LabVIEW and tried to create a Real-Time Project, I could not get LabVIEW to "find" the "Real-Time PXI" system -- it found nothing. It didn't help to look for "Real-Time Desktop" -- that also was empty. I could "force" it to point to my PXI by choosing Real-Time PXI and specifying the PXI's IP address. However, my first attempt at deploying some software to the PXI ended in frustration -- the code seemed to deploy, but the software (using Network Streams) never connected!
At this point, so much was going awry that I was completely uncertain if this was a Hardware or Software issue, and at what level. So I started over.
Second Attempt -- same PXI, different PC (a Windows XP box with LabVIEW 2010 and MAX 4.7). The Key to Happiness is MAX 4.7 -- this version still supports creating Floppy Disks! I created a set of disks, but really only needed two -- the PXI Reformat Disk and the PXI Safe Boot Disk. Using the Reformat Disk, I "started over" on the PXI. When the reformat was finished and it rebooted with the Safe Boot Disk, MAX was able to "see" my PXI (and called it a PXI, with the correct icon!) and gave it an IP of 0.0.0.0. Using MAX, I set the desired IP, gave the system a "User-friendly Name" (I think I called it BS-Test), and rebooted it with the Safe Boot Disk. I then proceeded to install LabVIEW 2010 RT software. Once the software was completely installed, I removed the Safe Boot Disk, took a deep breath, and rebooted (to whatever boot was installed on the Hard Drive). It came up in LabVIEW RT 2010!
I then tried to create a Real-Time Project and run some test code. This time, Project did what I expected it to do -- specifying a new Real-Time PXI led it to discover BS-Test, which it placed as a target with the correct PXI chassis icon. There were a few false starts in getting my test code to work properly, but eventually everything clicked.
So back to PC #1, the Windows 7 box with both LabVIEW 2010 and LabVIEW 2011, and MAX 5.0. The Good News is that MAX 5.0 now sees the PXI as a PXI chassis (which, of course, it is!). I opened LabVIEW 2010 on this PC, again built a Project and was able to find the Real-Time PXI. However, my test program failed to work -- it acted like TCP was being blocked. Hmm, a problem with Windows Firewall?
Brief digression. My PCs have two NICs -- one is connected to the Internet (actually, to the campus network which provides Internet access), the other is given a 10-dot IP and is connected to a "very local network", namely 3-6 computers and PXI controllers connected to a single hub, with no internet access. Windows 7 persists in calling such a network a "Public Network" (a Google search shows numerous people have asked how to change this to a "Private Network", and there doesn't seem to be a realiable or easy answer).
When I disable the Firewall for Public Networks, the test code runs. But this is using a cannon to swat a fly! Figured it out -- you need to create an exception (and might as well make it for All Networks) for the NI PSP Locator Service, lkads.exe, which on Windows 7 can be found in the folder Windows\SysWOW64.
OK, so is it possible, now that LabVIEW Real-Time 2010 is installed on my PXI, to upgrade it to LabVIEW 2011? First, I removed the hard drive and imaged it, so I could recover from a disaster. On my LabVIEW 2011/MAX 5.0 system, I found the PXI, opened its Software tab, and chose to Reinstall all of the existing software components. Note that my PXI is still running by directly booting from the Hard Drive -- I haven't used the floppies since I got LabVIEW 2010 (from the XP box using MAX 4.7) installed. When I said "Install", MAX booted the PXI to "Install mode", downloaded software, then let it reboot itself. Ta daaaa, LabVIEW Real-Time 11.0!
Opened LabVIEW 2011, built a Real-Time Project (which again correctly found the Real-Time PXI), and successfully ran the test programs. I now have a fully-functional LabVIEW 2011 Real-Time System!
The Key Insight into this process was that if your PXI controller does not have a "Safe Mode Boot" option in the BIOS, you need get this functionality for things to work properly. The current version of MAX no longer has this capability, but the previous version that shipped with LabVIEW 2010 does, but only via floppy disks (I strongly suspect that controllers that don't have Safe Boot in their BIOS are old enough that they have floppy drives -- you just need to find an older PC with these drives). Note that you cannot use an external floppy, such as a USB Floppy (I tried -- it could read and write floppies fine, but showed up as the B: drive, and MAX refused to talk to it!).
