Bob -
Assuming that you are using CVI 6.0 or later, when you run a project in CVI it runs the project in an external process. When the debug session completes, the process that controlled the USB is gone. This would be similar to running TestStand with one DLL, closing it down, and doing it again with the 2nd DLL. Since TestStand is running these DLLs in process, I suspect that TestStand is unloading the DLLs, but the driver that you are using to access the USB interface is either not getting unloaded or even if it is unloaded, something about the way it deals with USB is preventing it from freeing the ports. You could see if your DLL is getting unloaded by debugging it and see if the DLLMain receives a detach message.
One issue might be that if a handle to the port is not freed properly it might not release the port and the DLL. You should be able to create a simple test case in a CVI EXE that uses your two DLLs where you open/use/close a USB ports. Do it with one DLL and then the other to see if it works, but do this inside a single CVI EXE, not two separate projects. This would mimic what you are asking TestStand to do.
Scott Richardson
https://testeract.com