Electrical Power Suite Synchrophasor example FIFO Read Timeout issues

[Clavix2]

Hi, I am running the synchrophasor example from the Electrical Power Suite in Labview 2014 in order to implement a PMU unit.
The main vi (Synchrophasor Measurement) runs only for 10 seconds measuring perfectly analog input data, on the next 10 it stops to read, and on the next 40 it freezes, then it starts all over again.

We modified the FPGA vi (The one who runs in the FPGA Target) and eliminated the GPS inputs, because we don't have the 9467 GPS module, the analog measurement is still the same and has not been modified, the only thing that has been changed is that the Timestamps are generated with Computer time.
It throws an error in the Read from FIFO vi: 
"ep_Synchrophasor.lvlib:pmu_Read from FIFO.vi:7210001<ERR>Read Timeout: This error occurs if the Read VI does not receive all the requested samples before the user specified timeout period. Make sure the FPGA VI was started or increase the timeout."

We think that the FIFO reading vi is experiencing timeout issues, and we didn't modify the read FIFO vi.

Someone has tested this example? Does it works fine?

Thanks to anyone who can help me

[Clavix2]

I am using a cRIO 9082 with the analog voltage and current modules 9225 and 9227

[Cavarval]

Hello Clavix,

The error is actually generated in the RT vi by the pmu_Read from FIFO subVI. By analyzing the RT code, the error means that there were not sufficient samples in the FIFO after 5 s, which seems to say that the modifications you did to the FPGA vi are delaying it or even stopping it somehow.

I am very interested on how are you using the computer timestamp in the FPGA. Can you please explain?

You can actually upload the modified project and I can take a look.

Regards,

[Clavix2]

Hi Cavarval I didn't modify the sampling of the FPGA .vi, actually, what I did was to omit the GPS signals, I eliminated all the data acquisition by the GPS module. After the sampling of the FPGA and parsing data I used the timestamp of my computer and joined it to the measured information. Strangely, I eliminated the current acquisition, just left voltage acquisition from the FPGA, and it worked. I don't have any idea why the example just runs with voltage over my CompactRIO. Attached is the project I have modified, I have only modified the P CLASS 60 HZ FPGA file. I'll appreciate any help and advice you can give me, because I still have to make it work with both, voltage and current, measurements. Thank you

[Clavix2]

Hi Cavarval
Here is the file modified with the timestamp of my computer
I don't know why it lets me to measure only voltage, or only current, but with both measurements the FIFO timeouts.
I'll really appreciate any aid you can give me

[Cavarval]

Helo,

Sorry for the delay... for some reason I did not receive notifications about your replies.

I am not really sure right now.... Does the FIFO write times out or is it the FIFO read in the RT side?

When you remove current, do you just remove it from the FPGA or do you also adjust the RT VI? I mean the decimate and interleave functions. By the way, why exactly are you using those?

I did find an inconsistency in the number of channels value being passed to the FPGA in one of the RT's subVIs. The pmu_Configure FPGA.vi seems to have a 9 channels value hard coded being passed to the FPGA when only 6 channels are being acquired. There are 9 output, but FPGA Phasor Measurement for P Class.vi (the one that uses the number of channels) only cares about the actual physical channels.

I don't think this will completely solve the issue, but I am certain that it may cause some data errors. Can you fix that and let us know if the behavior changes?

[Clavix2]

Hi Cavarval, don't worry about the notifications, thank you for your interest !

First, I removed the FPGA I/O nodes that make current measurment block, the ones that throw Fixed Point data, and I replaced them with constants. I don't know if the 9 measurements you are talking about are the 9 FIFO array outputs, because from the FPGA vi I didn't touched anything else besides what I've told you. 

 I made another change, I separated the FPGA I/O node blocks that make the Current and Voltage Fixed Point measurements from the FPGA, so right now there isn't one block giving 6 measurements but 6 blocks and each one giving 1 measurement. Currently it works until 30 seconds, then it stops measurement for 5 seconds (It measures noise, because it still throws data) and then it returns the normal measurements.I added charts to see every step of the data that is in the .vi before it gets out in a FIFO file. Also, I noticed that the raw measurements from the FPGA I/O node blocks measure some kind of disturbances.

I'll send you the actual state of the file, as I mentioned it before, I just removed the GPS measurements from the FPGA Measurement vi, it should still work and throw data perfectly. But, the example from the library seems to have an error.

Thank you for your answer in advance Cavarval! 

[Cavarval]

Hello,

I took a look back at the number of channels thing it it should not be a problem. I saw something wrong last time, but everything is fine on that part of the code.

There are a couple things we can try. I noticed this time that while you are setting equal sample rates for the AI modules and starting them together, you are not really synchronizing them. Please refer to the properties from modules 1 and 2 in the original example and notice how the master module is exporting its clock and the slave is importing it.

This is not being done in your project. I am not sure what the implications may be here, but the example assumes the data is simultaneous anyway, so it will not hurt to correct that and test. In fact, once you do this I would actually recommend going back to the single node (you can have two versions of the project, one with a single node and one with a node per channel if you don't want to loose the current project version).

I would also recommend implementing some benchmarking code in the FPGA vi to try to identify if one ore more loops are taking more time than normal to execute when the problem occurs. You can do somehting similar to the code in the link below.

http://www.ni.com/example/30439/en/

You can write a true value to the while loop conditional terminal if you will use the shift regusters so it runs only one time or use the feedback node and get rid of the loop. Then, you can turn the code into a reentrant subVI and place it in each loop. That will give you independent time measurements.

Please synchronize the modules and let us know if that improves the behavior. If it does't, perform the time measurements and let us know if you are able to identify any trends.

Regards,

[MBenassi]

Hello everyone,

I know the thread is old but I'm experiencing a similar problem.

I'm runnning the same sychrophasor example on a CRIO 9057 using Labview 20.0f1. In my case, the core of the code is the same (I'm using also the module 9467 for the GPS).

Everything worked correctly but when I tried a long-term measurement, this error appear after about 40 h:

"ep_Synchrophasor.lvlib:pmu_Read from FIFO.vi:7210001<ERR>Read Timeout: This error occurs if the Read VI does not receive all the requested samples before the user specified timeout period. Make sure the FPGA VI was started or increase the timeout."

After adding a more precise log file, the second time the problem appeared after about 60 h in the exact moment I tried to open the remote front panel. I do not know if there is a correlation (the access to the front panel should not bother the FPGA, not directly, at least) but the timing is suspicious, although I opened it multiple times during the period without consequences.

I attached the FPGA code. I added some loops for other modules (AI, AO and DO) which should not interfere with the main one. Additionally, I'm using the "FPGA Resampling VI" of the Electrical Power toolkit to process the same current and voltage data.

Any suggestion?

Thanks in advance