LabVIEW
Not able to save all the data to a text file in the vi running at 1 kHz
Solved!
I should be able to save 1000 values per second but I am only able to save ~550 values. Am I doing something wrong here?
Above snippet is just an example. I want to do the same thing on a Raspberry pi running LabVIEW RTE but I save fewer values (~350) there compared to this.
File IO is pretty slow. So your consumer is falling behind. Additionally, your producer is killing the queue when it is done. When the queue is destroyed, anything that was left in the queue is gone.
So we need to come up with some ways to speed up the consumer. One option would be to put the Dequeue into a FOR loop to read off, for example, 5 elements at a time, process them, and then write to the file in a single string.
But the biggest issue I see is you need to let the consumer control the lifetime of the queue. You would send some sort of sentinel value telling the consumer to be done. I'm thinking a time value of midnight January 1, 1904 UTC (LabVIEW time epoch) should work nicely. So when the consumer finds that time, it knows to stop and destroy the queue.
Write more than one value at a time. Right now your algorithm is:
Get paper from drawer
Pick up pencil
Write number
Put down pencil
Pick up pencil
Write number
Put down pencil
...(repeat)
There is non-zero overhead when writing to a file. Maybe your computer can do it, maybe not. Depends on what else the processor decides to be working on at the time. If you change your code to do, say, 5 at a time, then it's:
Get paper from drawer
Pick up pencil
Write number
Write number
Write number
Write number
Write number
Put down pencil
Pick up pencil
...(repeat)
Your computer is doing other things every time it puts down the pencil. Batching doesn't guarantee it'll do all of it at once, but it makes it a heckuva lot more likely to try to do a lot with one "file access".
(Also solve the major queue lifetime problems that others have suggested).
@girish_jadhav wrote:
@crossrulz wrote:So we need to come up with some ways to speed up the consumer. One option would be to put the Dequeue into a FOR loop to read off, for example, 5 elements at a time, process them, and then write to the file in a single string.
Hi Crossrulz,
You mean I run the for loop 5 times every loop and at the end append the data to a text file? I put everything in a for loop but I get same timestamps and values if I do it like this.
Inside the FOR loop should be the reading an element from the queue and format the data into a string. You can build up an array of strings with an autoindexing tunnel. You can the wire that array of strings straight to a Write Text File. Each element will be a line in your file, so you will not need to add the End Of Line character(s) when you format your strings.
@crossrulz wrote:But the biggest issue I see is you need to let the consumer control the lifetime of the queue. You would send some sort of sentinel value telling the consumer to be done. I'm thinking a time value of midnight January 1, 1904 UTC (LabVIEW time epoch) should work nicely. So when the consumer finds that time, it knows to stop and destroy the queue.
How can I add this sentinel value to the queue? This should done in Producer loop and when I press stop button, this value should be sent to the consumer loop? To destroy the queue by consumer loop, should I connect release queue to the dequeue element in the consumer loop? Is that what you mean?
Use Enqueue Element after the producer loop and send your sentinel. Destroy Queue should be called after the consumer loop is done.
@crossrulz wrote:Use Enqueue Element after the producer loop and send your sentinel. Destroy Queue should be called after the consumer loop is done.
Thank you for the suggestions Crossrulz. Now I am able to save ~ 900 values per second.
When I run the VI highlight mode, VI stops when I press stop button but it hangs in normal mode. It also works fine when I remove for loop. Do you know what might be causing that?
Hi girish,
@girish_jadhav wrote:
When I run the VI highlight mode, VI stops when I press stop button but it hangs in normal mode. It also works fine when I remove for loop. Do you know what might be causing that?
Right now you check only every 5th received message as you only output the last message from that "5 iteration FOR loop": you need to check each message for that sentinel value!
Do the check inside the FOR loop and use the conditional stop of the FOR loop to "break" before the 5th iteration…
Your For loop always runs 5 times. If you hit Stop after it's only run 3 times, it'll lock up waiting for more values.
Right click the edge of the For loop and you can give it a Stop terminal just like a While loop. Check for the sentinel inside the For loop, not outside, then it'll check every element for the Sentinel.
Also right now, it only checks the *last* of the 5 elements you pull out. The tunnel mode is set to Last Value.
A quick hack would be to just enqueue 5 of the sentinel values, but that's a crummy way to do things.
Also, you're doing 5 samples per write, meaning you're still writing to the file 200 times a second. Change that to, say, 50, or 500, and you'll get a LOT faster.
@GerdW wrote:
Hi girish,
@girish_jadhav wrote:
When I run the VI highlight mode, VI stops when I press stop button but it hangs in normal mode. It also works fine when I remove for loop. Do you know what might be causing that?
Right now you check only every 5th received message as you only output the last message from that "5 iteration FOR loop": you need to check each message for that sentinel value!
Do the check inside the FOR loop and use the conditional stop of the FOR loop to "break" before the 5th iteration…
This did the trick.
As I mentioned in my post, I wanted to do the same thing to save files on a raspberry pi. I played around the sampling size but it just gives me ~400 values per second. Could it be dependent on the hardware?
