Maximum write speed to log file, trying to get a log entry every 100ms.

[LennartM]

I have a DAQmx application which data I want ot log. The aquistion is ran at a high speed, 1 KHz. I want to log the measurements to a text file which can read in Matlab or excel. I a wondering what the best approach is for this and what is the maximum speed.

I have created a program below to test. But it seems that the log is not consistent at 10 Hz alread. The log interval is set to 100 ms while the loop is running at 10 ms.  If I look at the time between the sampes 6 of them are higher than the 100ms. The time between some steps is 700ms which is quite high. Is this implementation wrong or is this just due to the undeterminstic computer?

[Bob_Schor]

The fancier the output format, the slower the writing (because you need to do other things).  Writing text entries, separating elements with commas (so-called "comma-separated variables"), and separating groups of elements with <CR><LF>, should be quick.

And it is.  I just wrote a test routine that generates a 1000 lines, with each line being 1000 random numbers, written as text, comma-separated with six decimals of precision.  The file size is 9.8 MB, and it took 3.6 seconds execute (see below).

Fast enough for you?

Bob Schor

[Dennis_Knutson]

Your acquisition rate is at the low side. Instead of text, you can use tdms streaming. With the correct plug-in Excel and Matlab can read them.

[LennartM]

Thanks, this seems to be faster than the write to excel  file.

I still see that the time step sometimes differs, but it is a lot better than the previous implementation.  I do have a problem with the stop button. If I implemend it as described here (local variable, Switch Until Released) the program does not start. If I use 'switch when released' it works but the program does not stop. With the probe I can see that both the lines to the stop buttons become true but the program does not stop. What could be the problem with this?

Dennis, also thanks for the suggestion. I already tried this but the Matlab convert tool is unfortunatly not working..

http://forums.ni.com/t5/LabVIEW/Read-TDMS-files-with-Matlab/m-p/3125710#M897790

[Sam_Sharp]

You need to initialise your stop button to 'False' before the loops start if you're using local variables.

[LennartM]

I did that be that seems not to of influence on the stopping behaviour. The funny thing is that both loop count indicators stop counting. But the program does not stop...It seems it is waiting on another notification to stop or something?

[Sam_Sharp]

1) You can use the same notifier reference for both loops.

2) You have no timeout set on the notifier so if the notification isn't received, it will sit and wait indefinitely for a new notification.

[crossrulz]

---

@LennartM wrote:

But the program does not stop...It seems it is waiting on another notification to stop or something?

---

That is a very likely scenario.  The simple solution here is to destroy the notifier after the top loop completes and make your bottom loop stop when there is an error from the Notifier (when it was destroyed).  That would eliminate the need for the local variable and would ensure that both of your loops stopped.

[ToeCutter]

Rather than using the PC clock to time your writes, make it data driven.

If you sample at 1kS/s, then 100ms=100 samples. So perform a write on every 100 samples. The log timestamp will be 'log start time+sample index/1000'.

[Bob_Schor]

LennartM wrote:

Thanks, this seems to be faster than the write to excel  file.

---

I think I demonstrated (with the snippet I posted) that you can write very fast using LabVIEW.  If you want to write an Excel file, and if you use the Report Generation Toolkit instead of Write to Excel file, and structure your code in a sensible way, you can achieve amazing speeds.

I took the demo I posted earlier here and changed it in the following ways.  First, instead of 1000 lines of 1000 random numbers, I generated 100 rows of 100 columns, one row at a time (so that I had 100 "writes" to the Excel Worksheet).  I also opened the Excel Report "minimized" (so it didn't need to write all of the evolving rows to the screen).

The timing includes the following:  Calling New Report to open Excel, running 100 loops in which I write an Excel row using Excel Easy Table VI, calling Save Report to File, and doing a Dispose Report to close Excel.

So how many seconds do you think it would take to write 100 x 100 = 10,000 Dbls to Excel?  An hour?  A minute?  15 seconds?  Less than a second?

Bob Schor

P.S. -- pick the most astounding answer ...