save million data's in one sec

[Akiel]

i believe excel has a max number of columns and rows per sheet it can cope with

[Ben · Accepted solution]

I would write the data to a temp file that is binary. As a post processing step it can be converted to text.

More often thannot the Disk i/o is the bottleneck.

TO optimize file I/O you wnat to avoid the OS calls as much as possible. THe OS gets involved in creating, allocation disk space, and re-allocating if the file grows beyond its original allocation.

So...

Pre-write your file (before the collection starts) with more data than you plan to collect. By doing so the OS only has to transfer teh mapped buffers in memory to teh disk.

If the disk is fragmented that will slow down i/o.

If there other processes using the disk, that will slow things down.

Solid state disk are fast.

Larg disk caches will also sepped up i/o.

Ben

[Mystogan]

---

@Ben wrote:

I would write the data to a temp file that is binary. As a post processing step it can be converted to text.

More often thannot the Disk i/o is the bottleneck.

TO optimize file I/O you wnat to avoid the OS calls as much as possible. THe OS gets involved in creating, allocation disk space, and re-allocating if the file grows beyond its original allocation.

So...

Pre-write your file (before the collection starts) with more data than you plan to collect. By doing so the OS only has to transfer teh mapped buffers in memory to teh disk.

If the disk is fragmented that will slow down i/o.

If there other processes using the disk, that will slow things down.

Solid state disk are fast.

Larg disk caches will also sepped up i/o.

Ben

---

Hi Ben,

I'm not pretty sure I understand what you meant. Can you possibly make example of "pre-writing"? Thanks.

[Yamaeda]

Just create a big array and write it to file so you're resereved a file. Then reset file pointer to start.

/Y 

[Mystogan]

thanks everyone for the inputs, Its not yet like what I want, but I think I'm going on the right way to that. I think my biggest problem is with that Save1Data.vi. Im trying now not to use it but to save the data by making event structure and put the property node "export to excel" thing. Would be great if I can export it to .txt, is it possible?

[aeastet]

Writing a text file is easy if you go to the file palet you will see there a vi that is call write to text file.

[Mystogan]

---

@aeastet wrote:

Writing a text file is easy if you go to the file palet you will see there a vi that is call write to text file.

---

the problem is, that I need to save about 1 million points or more in 1 sec. I have tried to use open/write/close file using that block, but seems doesnt work well with my plan (1 million points in 1 sec). I tried to make sequence structure, in the last sequence I put the export to excel thing from the graph, and it works better than open/write/close file. But with that sequence I cant control the file that occured from that "export to excel" (cant change the name, like that).

So, basically what I need to do:

1.making a high frequency (like 10MHz) signal simulation as input (I got this with using the sine waveform.vi from labview)

2.make a control, like a "SAVE" button, when I click the button, the input signal will be copied to output, saved into a file. (this is where my problem is, since I need to copy million points to output per sec)

3.when I push "SAVE" button and make it OFF, it stop copying input into output.

Sorry guys, I'm pretty new with Labview, thats why I'm a newbie

[aeastet]

Then I would reiterate what BEN told you earlier that you need to write a binary file and then convert it when you are done.

[Mystogan]

hi tim,

I wonder if you may have a little .vi example to help me understand more about how things works?

thanks,

Yan

---

@aeastet wrote:

Then I would reiterate what BEN told you earlier that you need to write a binary file and then convert it when you are done.

---

[Hooovahh]

---

@Akiel wrote:

i believe excel has a max number of columns and rows per sheet it can cope with

---

Excel 2003 and earlier has 65,536 rows 256 columns per worksheet.

Excel 2007 and newer has 1,048,576 rows and 16,384 columns.  This increased limit is only applicable if the workbook is not ran in compatibility mode.  If compatibility mode is on then the limit is set back to the 2003 values.