Direct streaming to disk is much slower than the convinient one. Error 200560.

[Alexej_P]

I usually measure and process (real time) 1) magnetic hysteresis (MH) loops and 2) Barkhausen noise (BN) simultaneously.

1) For MH-loops I use the PCIe6351 card.

AIN: 5 channels at 50kSs (3 of them for Hall sensors). Process all of them  on the fly.

AO: 2 channels, one for triangular waveform with 1-bit resolution.

2) For BN I use PCIe6361 card.

AI: 1 channel at 1MSs. Calculate RMS and ARV from each 1-10 msec. If I need the lock-in instead of BN, I process the signal by mixing with sin(t) and cos(t). For BN the signal is filtered by the Butterworth Filter VI. There is an option to save raw BN file.

AO: 1 channel. Sinusoidal waveform generation for lock-in measurements.

All these tasks run in parallel without errors (no raw BN saving at 1MSs ) up to 100 sec measuring time (I simply did not make longer measurements). I attached the screenshot of the BN part of the measurement.

If I choose "raw BN saving at 1MSs", then I can measure up to 20 seconds if I am lucky - the file size up to 260 MB. After this time I get an error that some data is lost.

It is not enough for me, I need raw data up to 100 seconds.

I decided to use the  direct streaming to TDMS with no processing. I opened the example "TDMS Logging - Log Data Only.vi", tried it at 500kSs for 100 seconds, but always after 13 seconds appr. I got "Error -200560 occurred".

I am shocked - only one direct streaming (one card, 1 channel) is several times slower than the convenient NiDaqmx streaming (two cards, two AO and two AI) with many other tasks in parallel. How is it possible?

Is there a way to stream to RAM, say up to 1-2Gb and then save to a disk?

[GerdW]

Hi alexej,

do you mind to cleanup your block diagram so it would be much easier to read/understand?

(Btw. while doing so you might give labels to those terminals, use polymorphism instead of this sin/cos FOR loop, think about using initialized arrays instead of build them using shiftregisters with BuildArray, avoid hidden code, remove Rube-Goldbergs, …)

I don't see any file write operations in this image, so what is the question about?

(When you want more detailed help with your VI then you should attach your VI. We cannot debug images using LabVIEW.)

[Alexej_P]

Hi GerdW,

it is just an illustration that this complex part of the complex program works several times faster than the simplest "TDMS Logging - Log Data Only.vi" (found in LabView examples).

The problem is in the direct streaming - it is much slower. I expected the opposite - direct streaming should be much faster.

At 500kSs the direct streaming works up to 12 seconds and then I get that error 200560.

Maximum output size 11Mb. I expected that it should work up to gigabytes.

[GerdW]

Hi Alexej,

where do you store those data? Fast or slow harddrive?

[Alexej_P]

Hi GerdW,

Windows shows "WD1003FBYX-01Y7B SCSI"

The program "Write To File Speed Test.vi" from Labview examples says: 4 Gb at 78.19MB per second in 1000 writes (4MB). So it is ~ 70 times faster than necessary for 500kSs 16bits.

[GerdW]

Hi Alexej,

a single test with specific behaviour doesn't rule for any other test with maybe totally other behaviour!

The same is written here about that harddrive… (At "4k random mixed IO" you just get ~1MB/s from this drive according to this website.)

Again: from what I see in your image there is a lot of room for improvements…

[altenbach]

As other have already said, you picture are not suitable to troubleshoot the problem, for us to identify problem code, or suggest solutions. What is outside the big case structure?

Can you show the code doing the logging? Can't you decouple the file IO using a produce consumer architecture? How much RAM do you have? Is this LabVIEW 32bit or 64 bit? Why are you growing all these arrays if you are streaming the data to disk? Since you can calculate the exact number of iterations before the while loop starts (based on time and rate), you need to use a FOR loop and autoindex at the the output tunnels. Much more efficient because allocation is now exactly known. (You can still stop early or on error using the conditional terminal if needed.)

Suffice to say, your coding style does not show much experience, so it is likely that your file IO might need code cleanup too. We can help if you attach the VI.

[Alexej_P]

Hi GerdW,

It is not a disk problem. I've just tried a new one WD My Passport 2TB USB3 - the same error after 13 seconds. The speed test says 117 MB/sec

So the same question - why the direct streaming is very slow?

[Alexej_P]

Dear altenbach,

as I said above, I need help with "TDMS Logging - Log Data Only.vi" from LabView examples.

I already regret that I attached a part of my program diagram. It is just for you to see that "very bad written" "with no experience" complex program performance is much better than the primitive program "TDMS Logging - Log Data Only.vi" by Labview. I attached the block-diagram of that VI from the National Instruments. I have problem with that  "TDMS Logging - Log Data Only.vi" from National Instruments. I need to find the answer - why is this "TDMS Logging - Log Data Only.vi" from National Instruments is very slow. Not my program, but the program from National Instruments.

[GerdW]

Hi Alexej,

I don't find this example VI in my LV2019 installation…

Are you also using "finite samples"? Why? Which samplerate, which "samples to read"? Which timeout for DAQmxWait?

(It really would help to get your VI with typical values in all controls! You making it very hard to help you…)

No problem on my side to run the same commands like in your image at 1MS/s for 1M samples with a simulated DAQmx device on a rather slow harddrive in my not-so-new laptop…

(Same VI with 1MS/s for 24M samples with DAQmxWait using a proper timeout value runs just fine. I don't mind to run this test VI any longer with this less information from your side.)