Network stream data loss

[M1080]

I have an application where I need to stream Data from a CRIO 9068 chassis to a PC, and i'm running into some problems with data loss.

The CRIO is sampling the data, putting it into a 2D array of floats (size depends on number of channels and sample rate) and streaming it to the PC.  The Data loss is not consistent, sometimes no data will be lost. Also the size of the array being streamed seems to not make any difference, Sometimes I will not lose data from 24 channels at 25 KS/s, and sometimes I will lose data from 1 channel at 10KS/s. 

 The Host VI (PC) runs within a non-timed while loop  It usually runs at about 50 Hz, sometimes system hiccups will make it slower for a loop or two.  The CRIO wirte loop runs at 20 Hz.  I think the Data loss is happening when the PC has one if its Hiccups,  I would expect though that the buffer size in the network stream could absorb these Hiccups.  If I make the buffer size any bigger, I seem to get a memory leak and the CRIO crashes after a couple minutes.  I also have the CRIO save data directly without streaming, and no data is lost there, so I know its dissapearing somewhere in the stream.

I also have a separate Standard TCP connection between the two for command and control, not sure if that could be causing a problem interfereing with the network stream.

  I have attached pictures of the VIs where the endpoints are creadted and where the data is written and read. 

Im Using Labview/RT 2013

Thanks

[AGJ_CR]

Hi,

Have you tried building a simpler application that just takes care of the communication? This means extracting all the other sections of your code just to test the communication.

By any chance, are you reading from several endpoints on the host? I am seeing that you are using a for loop to read there.

I also would be helpful if you could show more details of your current code.

Regards,

AGJ

[M1080]

Hi AGJ,

I have not tried cutting out all other sections that dont involve the communication.  Everything involving the communication works fine 99% percent of the time, its just now and then the Host will drop one or two of the stream writes that the CRIO chassis makes.  The Host does read several endpoints, but the problem happens no matter if I am reading from one or several endpoints.

Im not sure what else to show with my current code.  I have attached a few more pictures showing the entire aquisition loop on the target and also the loop on the host where the data is saved.  I am relativley certian that its not the save loop thats loosing data, but I thought I'd attach it anyway.  The last picture is just a block diagram to try to illustrate what happening on the host side since the block diagram is to big to make into a picture.  

Sorry about the messy code, I know I need to get better at keeping it clean.

One other note: disregard the comment in the Target aquire picture that says "every 10th iteration checks for a command"  It checks every iteration now.  That read TCP VI has a TCP read function in it that checks for the stop command from the host.

[Bob_Schor · Accepted solution]

AGJ's suggestion is a good one.  I found it difficult to follow your code -- there was so much going on, with wires everywhere ...

I've got an application running on a PXI 8106 controller streaming 24 channels of data collected at 1 KHz (or 24K S/s).  It gets streamed via Network Streams to the Host PC, and from there streamed to disk.  At the same time, I'm streaming "Events" (State transitions, DIO line changes, etc) on another Network Stream to another data file.  One key step is that as the data come off the Network Streams, they are put onto a Queue for sending to disk.  There are multiple parallel loops running, and I never lose a sample (I can tell because one of my channels is a "clock", the index of the Timed Loop running on the PXI that Produces the data being sent via the Network Stream to the Host -- I can inspect this channel and see that each point differs from the previous by 1).

Do you use Queues to buffer your data into and out of the Network Streams?

BS

[AGJ_CR]

Hi,

I would definitely go with try to make a simpler test with a simpler code. There you should be able to improve your communication and concentrate on the important. You will probably have to put in order your code anyway. Otherwise it will be harder for you to know where the issue might be.

Bob has also pointed out a really important thing to consider. You need to make sure that you are reading from the Stream fast enough and put that data safe into another buffer. You can use queues to perform this inter-process communication.

Application Design Patterns: Producer/Consumer: https://www.ni.com/en/support/documentation/supplemental/21/producer-consumer-architecture-in-labvie...

Regards,

AGJ

[M1080]

I followed your suggestion and broke the stream read out into a separate loop and fed it into a Queue.  No more data loss!  I was under the impression that the Network stream has some sort of queue built into it, but It appears that if two writes are perfomed between reads, it throws away the old data.

Thanks!

[AGJ_CR]

Excellent!

Regards,

AGJ

[Bob_Schor]

Network Streams are designed to be lossless (see this White Paper).  They don't have a Queue, they have a buffer (which acts a little like a Queue, I admit), and can (and will!) "block" if the buffer gets filled.  I'm guessing that your Writer is blocking, and you are losing data on that end ...  But All's Well that Ends Well!

BS

[smithd]

Yeah, they've got a preallocated FIFO so you shouldn't have an issue unless you're going significantly slower. And honestly the 50000-element buffer is probably not a good thing. You definitely need to have some buffer, but 50k elements is a lot and its going to mask any issues that are occurring. I'd say during development you should set it to something much smaller like 1k, and then increase as needed, so you can better characterize whats going on.

On an unrelated note, the main reason I'm posting is to say you're not actually preallocating your buffer. If you look through the help for create XYZ endpoint you'll see that for any non-scalar data types you need to pass in an array of the maximum expected size. what you're doing right now is just allocating an empty array which is not useless, but close to. Also remember that total memory usage of the endpoint will be 50k*size, so if you drop in a size-2000 array of doubles you have just attempted to preallocate 760 MB of data. So be careful 🙂

And since I'm here, have you read this: https://www.ni.com/en/shop/compactrio/compactrio-developers-guide.html

I'd highly recommend it, or at least skimming parts of it.

Also on "target_acquire loop.png" there are a few concerning items which you can learn about in that developer's guide:

-You shouldn't use a timeout on DMA reads, usually (short version is here https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000000P9SASA0&l=en-US)

-You shouldn't log to disk in a timed loop, usually

-You shouldn't do network communication in a timed loop, usually

-You *really* shouldn't branch a 2D array of doubles inside of a timed loop (there are exceptions but you are definitely copying and potentially allocating memory in the screenshot)

One solution to these things would be to correctly offload the logic to other low priority processes. The easier solution would be to just make that loop into a normal while loop. After all, the DMA fifo is buffering things for you so you don't actually have a hard time limit, do you? What happens if the timed loop finishes late?

Finally, and I know this is way too long at this point but oh well, I'd recommend looking at this: https://forums.ni.com/t5/Example-Code/Reference-Application-for-NI-CompactRIO-Waveform-Acquisition/t...

Maybe not for this current codebase, but in the future it might make the acquisition code significantly easier to write.

[M1080]

I am confused about what constitutes an "element" in regards to the stream initialization.  I have a 1-D array of 30,000 sgl values I need to send over the stream.  Does the stream see the entire array as 1 "element" ? Or is each value in the array (30,000 of them) considered an element to the network stream?