Project corruption kills EtherCAT Digital I/O

[viScience]

In both LV2009 and LV 8.6.1 I have experienced a type of invisible project corruption that results in damage to etherCAT scan engine access to Digital Output resources.
Image a working cRIO 9074 with a 9144 expansion chassis that has been working perfectly for an extended period of time. A change is made to source code, compiled and
deployed to the target. Something goes wrong with the deployment and both the project explorer and the cRIO crash hard. The cRIO has to be manually rebooted and LV on the development machine has to be terminated and restarted. After restarting and redeploying again, everything is working fine except that Digital Outputs are not generating a voltage change on the cRIO 9076 module that resides in the 9144 expansion chassis. In the Distributed System Manager, digital outputs on the 9076 are forced but no electrical signal change is observed.
All other modules are working normally. cRIO is then reformated and project is redeployed but the problem persists. Swap out a new 9144 and 9076 module, still the problem persists.
Open a new project and add cRIO target and finally the outputs start working again. Open the original project, remove the cRIO target, add cRIO target, redeploy and then all is well.

What is the lesson learned?

Apparently, there was nothing wrong with the hardware or the source code or configuration within the project yet something 'invisible' had become corrupted in the project.
The problem is that there is no way to validate the project to see if something has gone wrong internally. Also no errors were generated. I cannot emphasize how disturbing this is to
our group here at NASA were we are using this system to suspend a 3 million lb launch vehicle.

[DirkW]

Hi,

This is the first time I see such a corruption to a project which causes a deployed module to not work at all.

If such a hard crash happens to a development machine, LabVIEW should be able to recover from it, if the recover feature is enabled.

It would be interesting to know whether you had "recover" enabled or not.

To prevent this from happening again I would be interested to have the corrupted project file for investigation.

If you cannot post it to this forum for confidential reasons please provide it to NI Support.

Thanks

DirkW 

[viScience]

Yes, I did have recovery enabled but when I rebooted I declined to restore and instead pulled from the last saved version thinking that it would be better to not include any recent changes to the project that might have contributed to the crash.  Unfortunately I did not think to save the bad project because I did not realize that was the case until I removed and restored the cRIO targets within the project and thus solved the problem.  I can add that over the past year I have had several problems with loss of control of Digital Outputs on a 9144 chassis.  Once, in LV 8.6.1 resoved by swapping out the 9144 chassis.   Once in LV2009 due to the distributed clock conflicting with the software sync driver, and now this.  In all cases, only the Digital Outputs were affected, is that just a coincidence?