LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Data Recording application not keeping up

Solved!
Go to solution

I have an application that is reading data from a reflective memory ring and then writing to a RAID (NI 826x) in a PXI chassis.  The application must read and write 5 separate data streams, 3 at 100 Hz 1at 20 Hz and 1 at 10 Hz.  Using LV 8.61 on a regular PXI chassis.  Currently, the app has a separate loop for each data stream to read from reflective memory and a separate loop for each data stream to write to disk.  The app works fine but occasionally (3 to 4 times in a two hour period), the loops that are reading from reflective memory stall momentarily and therefore, data is lost.  The memory rings are such that for the 100 Hz streams, the memory must be read at least once each second (100 elements in the ring).  An approximation of the loop time is usually a little over a second.  When the nubmer of elements in the ring was set at 50, the 'failed' loop times was a little over 500 ms(don't understand this).  I am using a regular file write to write to disk.  In a non real-time system is this to be expected?  Can it be prevented?

Any input appreciated.

thanks

craig

0 Kudos
Message 1 of 6
(2,917 Views)

Craig,

 

With your data rates it seems likely that you should be able to keep going indefintely.

 

Posting your VI would help a lot.  Verbal descriptions ineveitably omit some small detail which is the key to the performance.

 

How are you transferring data from the read loops to the write loops?

 

Are the files occasionally being resized or moved by the OS?  All one file or multiple files?

 

How often do you write to file and how much data in each write? How is the data formatted for the file writes?

 

Lynn

0 Kudos
Message 2 of 6
(2,905 Views)

3-4 times in a 2 hour period sounds like either the OS giving priority to another process, swapping or memory reallocation.

If all 5 data streams happens to write to disk simultanously i wouldn't be surprised if it takes some time. The design must ofcourse handle this with large enough buffers (double optimal size or dynamic).

 

What i'd like is 1 user event file writing loop and your 5 data gathering loops. Once your gathering loops have gathered a decent amount of data (100 samples), send a write file event with the gathered data array and empty it.

 

The write event simply adds the data array to the indexed result file, but can do so at it's own pace, events stack and you shouldn't lose any data.

 

/Y

G# - Award winning reference based OOP for LV, for free! - Qestit VIPM GitHub

Qestit Systems
Certified-LabVIEW-Developer
0 Kudos
Message 3 of 6
(2,892 Views)

Lynn,

 

Answers to your questions:

 

How are you transferring data from the read loops to the write loops?

via queues

 

Are the files occasionally being resized or moved by the OS?

not sure how to tell if they are

 

All one file or multiple files?

multiple files, one set for each data stream, with a set file size, when reached, a new file will be created.

 

How often do you write to file and how much data in each write?

At first I was writing to the files after reading one 'segment' of the reflective memory ring then I tried reading all segments of the reflective memory ring that were ready to be read and then writing to the file.  Both ways behaved the same.

 

How is the data formatted for the file writes?

There was no formatting of data, it was read as binary and written as binary

 

I will try to post the various VI diagrams when I have access to my code.

Thanks for your input.

craig

 

0 Kudos
Message 4 of 6
(2,869 Views)
Solution
Accepted by topic author craiguno

It has been many years, but I recall a problem with a RAID controller and "writes" in an industrial PC.

 

I was running under Windows XP OS, using the vendor specific drivers. I ended up adding the "Flush File" function. I don't recall if I called it every time, or if it was every nth iteration of the write loop (quotient and remainder function).

 

 

http://zone.ni.com/reference/en-XX/help/371361E-01/glang/flush_file/

 

"This function forces the operating system to write any buffer data to the file."

Message 5 of 6
(2,861 Views)

I tried a few things at once, disabled some apps on startup, slightly modified write algorithm to perform ~70 % less write operations but I think adding the Flush File function is what fixed the problem!  Thanks Phillip!!  I think it might have taken me a while to get there.  When/if I get a chance to verify, I will post but thanks to everyone for the input!

craig

0 Kudos
Message 6 of 6
(2,838 Views)