Slow Data storage issue over time

[HarshRaj18]

Dear community,

I am using three similar TSI Flow meters to record Flow, Temperature and Pressure. I am controlling the flow with a PID loop. Initially, the flow meters reflect the data every second, recording it every second; however, after saving the data in the same file, they start responding very slowly. When I changed to a new file, it works fine until the file size becomes too large and it can't store the data in the same file and flow meters start responding very slowly (once in 10-20seconds). It suggests to me that the memory is full. 
Note: I am perfectly fine with creating a new file every day if this problem involves a complex solution.

Another piece of guidance I need regarding data storage. Is there any easier way to store all the data in one file that doesn't create any memory issues. I want to save all the flow meters data in one file without moving so many connections/wires. For your reference, I have attached two png images from one of the complex VIs that store so many data values by creating something like "House Variable".
One more question, to save the data in Excel, which one is better to use: Write to measurement file or Write LabVIEW Measurement FIle? What is the difference between them?

I have saved the VI for 2015 older version. Hope anyone will be able to open it. Thanks a lot. Any help/suggestions are highly appreciated. 

[crossrulz]

Use a Producer/Consumer so that your file writing is done in a different loop than your control.

I would also recommend not saving a lot of data to an Excel file. Use a simple tab delimited text file or TDMS for saving your data. These are much more optimized for writing data.

[GerdW]

Hi Harsh,

---

@HarshRaj18 wrote:

it works fine until the file size becomes too large and it can't store the data in the same file and flow meters start responding very slowly (once in 10-20seconds). 

---

• Because you choose to write to an Excel XLSX file.
• Because you choose to write to 3 XLSX files in parallel.

---

@HarshRaj18 wrote:
One more question, to save the data in Excel, which one is better to use: Write to measurement file or Write LabVIEW Measurement FIle? What is the difference between them?

---

As has been said before: don't write to XLSX files in general!

Write to a simple text file (aka "CSV"): open the file before the loop, write to the file inside the loop, close the file after the loop!

---

@HarshRaj18 wrote:

Another piece of guidance I need regarding data storage. Is there any easier way to store all the data in one file that doesn't create any memory issues. I want to save all the flow meters data in one file without moving so many connections/wires. For your reference, I have attached two png images from one of the complex VIs that store so many data values by creating something like "House Variable".

---

These are local variables: avoid them if there are other options (Queues, notifiers, channels, …)!

---

@HarshRaj18 wrote:

Note: I am perfectly fine with creating a new file every day if this problem involves a complex solution.

---

It's complex because yoou have created complex (not scalable, messy, …) code!

• Avoid ExpressVIs, they only hide important information. You don't need to MergeSignals for a simple chart/graph, when you can bundle elements!
• Avoid duplicated code like yours! Use (more) subVIs for common tasks…
• Cleanup the code, straighten wires. Avoid hidden wires…
• To implement state machines you usually place the (while) loop around the case structure. Your code is inside-out! (No need for 3 stop buttons!!!)
• What's the point of a case structure set to TRUE case all the time?
• …

[HarshRaj18]

Hi GerdW,

Thanks for your suggestions. I am not an expert in LabVIEW. Could you please explain how I can follow this loop thing:"Write to a simple text file (aka "CSV"): open the file before the loop, write to the file inside the loop, close the file after the loop!"

I'll learn to bundle the signals.

Will clean it and use more SubVIs.

This part is not clear: "The (while) loop around the case structure. Your code is inside-out!"

Case structure is to turn keep one or two flow meters off while running another one.

I'll work on your suggestions and would appreciate if you can help with the queries or send a updated file if you can do some critical modifications. Thanks a lot.

Harsh

[GerdW]

Hi Harsh,

---

@HarshRaj18 wrote:

Could you please explain how I can follow this loop thing:"Write to a simple text file (aka "CSV"): open the file before the loop, write to the file inside the loop, close the file after the loop!"

---

Simple like that:

Open before, write inside, close after the loop…

---

@HarshRaj18 wrote:

This part is not clear: "The (while) loop around the case structure. Your code is inside-out!"

Case structure is to turn keep one or two flow meters off while running another one.

---

There should be one (1) while loop that contains your code, like your case structure(s) to handle devices.

Having a case structure as the outermost structure looks so wrong…

[Yamaeda]

I'm guessing that Express function opens the file and finds the end to append each time it's called. That's why it works decent on a small file. Do as suggested with file handling and you'll solve the issue.

[GerdW]

Hi Yamaeda,

---

@Yamaeda wrote:

I'm guessing that Express function opens the file and finds the end to append each time it's called. 

---

With XLSX files it's even worse as you cannot easily append data "at the end of the file". There's much more work involved…

[Yamaeda]

---

@GerdW wrote:

Hi Yamaeda,

---

@Yamaeda wrote:

I'm guessing that Express function opens the file and finds the end to append each time it's called. 

---

With XLSX files it's even worse as you cannot easily append data "at the end of the file". There's much more work involved…

---

Yes, being zipped html-tables it's a few more hoops to jump through. 🙂