save million datas in one sec

[Mystogan]

Hi guys, I've tried to search, but I didnt find the answer, thats why I'm posting this.

How many points/samples that can be saved in 1 sec with Labview (max)? It depends on my PC specs? I've searched in this forum and found this old thread http://forums.ni.com/t5/LabVIEW/Extremely-large-files/m-p/730139. With that program from (Herr) Altenbach, I've tried to save the datas in a .txt file using this Save1Data.vi and Signalsub.vi. The problem comes, when I'm trying to save like 100.000 points. Is there any hints to save that much samples, I need like to save about 1 million samples in 1 sec?

 

[Yamaeda]

1m samples, assuming doubles, will be 8Mb of data. In itself it shouldn't be hard to save 8mb/sec. Opening and closing files will be a problem, so keep the file open. If you append every second it might be a problem, as might the OS interfere, so a separate disc and/or a SSD is always nice, but 8Mb/sec isn't really a high value.

/Y 

[Mystogan]

Hi /Y,

thanks for the respond. But seems it doesn't help, I'm trying your suggestion by not closing the file, but still, doesn't work. Any other suggestion? (what I did is just deleting the "close file" in block diagram). 

Since I'm saving here the Samples and the time, that means, instead 8Mb, I'm saving 16Mb/sec?

I'm actually going to save the appended array of this samples, but first thing first, I'll think about it later.

[Mystogan]

It seems my Save1Data.vi is not very good program to save the samples/datas. Because its much faster/easier to export the waveform data to excel manually (right click on the graph), rather than using my Save1Data.vi.

[Norbert_B]

Even though this is not referring to DAQmx, there are quite some basics listed which are definetly of interest for you....

Norbert

EDIT: This might be a more general introduction

[Yamaeda]

That's not quite what i meant. 🙂

You should open and close the file, just not every second. 🙂

Assuming you have a loop that measures 1m samples and then write the result to file, you should open and close the file outside this loop, and just overwrite/append inside the loop (in this case in a sub-vi).

You're writing the result as text, which'll be a little bigger and will also require a transform of the data, you could write a binary file. However, it shouldn't be a real issue in this case.

I'm pretty confident the main culprit is the opening/closing time of the file.

/Y 

[Norbert_B]

Ah, this link is missing 😄

Norbert

[Mystogan]

---

@Yamaeda wrote:

That's not quite what i meant. 🙂

You should open and close the file, just not every second. 🙂

Assuming you have a loop that measures 1m samples and then write the result to file, you should open and close the file outside this loop, and just overwrite/append inside the loop (in this case in a sub-vi).

You're writing the result as text, which'll be a little bigger and will also require a transform of the data, you could write a binary file. However, it shouldn't be a real issue in this case.

I'm pretty confident the main culprit is the opening/closing time of the file.

/Y 

---

I thought I need to put the open file and close file in the loop cause I need to store the data while I'm in TRUE case (I mean, actually Im not going to save all the datas all the time, but just at the time I'm clicking the TRUE button for the real program). But I think I'll try to take out the open/close file thing from the loop and see how it works for me. 

[Mystogan]

What am I doing wrong here? I took the open/close file out from the loop, but I still cant save the datas into the file (took too long time, and the file is just 0 KB). And when I tried to export the datas to excel manually from the graph, it gave me only 1048576 rows of datas instead of 2millions datas that I put in histogramtest.vi, is that the limit of excel? 

[Mystogan]

@Norbert: thanks for the links, I'll look at it.