LabVIEW and large data sets are incompatible. T/F? I say T.

[*3d0g]

Here's the scenario.

An oscilloscope is capable of capturing 50MSa/channel.  You want to work with the captures, differentially; therefore, you need two such captures for a single signal capture.  The manufacturer has provided a driver to the IDnet (the instrument driver network, here at NI.)  The driver exhibits a sample-capturing vi that brings the capture to LabVIEW via a waveform object (indicator).

You've found in the past that the best you can do with some scope software you work with is 6.6x10^6 Sa.  You figure it's just due to really inefficient LabVIEW-inexperienced code.  Luckily, 6.6MSa is more than enough to get the two channels you need, for now.  But what of tomorrow?  What about when you need to work with more samples?  Just how much can LabVIEW handle?  You try a simple experiment to get the answer.

You create a no-frills practically pointless vi.  The vi simply creates an array and displays the array's contents in an indicator.  That is a totally useless vi.  For the vi to be of any value there'd have to be some means for writing something other than a constant to the array.  Nevertheless, this is the created vi.  Its algorithm is dirt simple: 1) create an array, and 2) display the array's contents in an indicator.  Since you'll be needing double-typed values for the oscilloscope's data set, it's reasonable to create the array to store doubles.

Instead of using every bit of the oscilloscope's storage per channel, you choose a fraction of it, only 32MSa, and you get this by sampling at 500MSa/s for 64ms.  To store this information will require (32MSa)(8 Bytes/Sa) = 256MB.  Preparing for this future real-McCoy situation, you set your array parameters to double (8 Bytes/Sa) and 300M elements.

You then run the simplest vi in the world and...

LabVIEW doesn't have enough memory.

You then back the number of elements down from 300M to 200M and try again.

LabVIEW still runs out of memory.

You reason that your only hope to work with the oscilloscope with the above parameters is to discard LabVIEW for the Object Oriented Programming world and its heap, because LabVIEW doesn't know anything about the heap and is totally incompatible with it.

Conclusion: LabVIEW is incompatible with large data sets.  10KSa?  Fine.  50MSa?  Forget it.

What say you?  

[*3d0g]

Incidentally, 32M elements for a double-type array, done four times, since it's a four-channel oscilloscope...

Yep, LabVIEW runs out of memory.  Hence, two differential measurements using a fraction of the oscilloscope's storage is out, due to LabVIEW.

However, if you disagree, and you know how to do it, then please tell me how to do it.  This is simply the creation and display of four arrays.  This is a useless vi, a vi cut to its bare bones.  I need vis that'll do something useful with these huge arrays.

LabVIEW can't do it.

[santo_13]

If you think about it, it is not LabVIEW that ran out of memory, it is the OS ability to allocate memory that LabVIEW requested.

You are missing a lot of details about your experiment to claim LabVIEW is incapable. Please share your experiment so fellow LabVIEW users can evaluate whether you're doing things right.

[Bob_Schor]

I tried an experiment.  You are sampling 2 channels of dbls and say " To store this information will require (32MSa)(8 Bytes/Sa) = 256MB."  My first question is I assume 2 channels of Dbls take 2 * 8 = 16 byte, so you would require 512 MB, if I understand your situation.

I wrote a little demo routine to generate a big 2D Dbl array.

This generates 30 M samples which requires 480 MB, and there is no problem with exceeding memory.  Pushing it to 32 M, however, does "run out of memory".

I have always run 32-bit LabVIEW.  I don't have 64-bit LabVIEW installed, but my understanding is that the extra memory is to allow larger Data Structures.  I could imagine that simply running the above code with 64-bit LabVIEW might still have a memory problem, but I would expect using something like a DVR structure would allow using the expanded memory.

Some of my colleagues on the Forum who run 64-bit LabVIEW might be able to confirm or refute my assumption, but my "guess" is to say that LabVIEW is "probably" compatible with large data sets (I'm guessing at least tens of GB).

Bob Schor 

[Bob_Schor]

I tried an experiment.  You are sampling 2 channels of dbls and say " To store this information will require (32MSa)(8 Bytes/Sa) = 256MB."  My first question is I assume 2 channels of Dbls take 2 * 8 = 16 byte, so you would require 512 MB, if I understand your situation.

I wrote a little demo routine to generate a big 2D Dbl array.

This generates 30 M samples which requires 480 MB, and there is no problem with exceeding memory.  Pushing it to 32 M, however, does "run out of memory".

I have always run 32-bit LabVIEW.  I don't have 64-bit LabVIEW installed, but my understanding is that the extra memory is to allow larger Data Structures.  I could imagine that simply running the above code with 64-bit LabVIEW might still have a memory problem, but I would expect using something like a DVR structure would allow using the expanded memory.

Some of my colleagues on the Forum who run 64-bit LabVIEW might be able to confirm or refute my assumption, but my "guess" is to say that LabVIEW is "probably" compatible with large data sets (I'm guessing at least tens of GB).

Bob Schor 

[mcduff]

Hmm. I can take 32 channels at 30 MSa/s continuously which includes saving the data, I can take a smaller subset of channels at a higher rate, issues can arise with the display and processing of data, that is where you need to be careful.

[santo_13]

I was able to generate a 32M x 2 array of DBL without any error. So, in short your claim that LV can't handle has been disproved.

@mcduff hit the point, it is not about whether LV can handle that data, it is whether the way you handle the data is inefficient that you run of memory.

In fact, I easily generated a 100M x 2 DBL array like breeze.

Edit:
I successfully created a 500M x 2 DBL array on LV 2021 64-bit on a Win 10 64-bit with 32GB RAM Laptop

[*3d0g]

Either interpretation, the memory isn't available for me to use.  I cannot work with two differential signals of 32MSa.  That'd require four 32MSa arrays of double.  LabVIEW doesn't have the memory, for whatever the reason.  It is incompatible with large data sets.

[*3d0g]

I was only focusing on one channel's capture, first.  I could create an array of doubles of 32M elements.  That'd need 256MBytes.

I could not do it four times, though.  

Hence, the driver dutifully gives me a set of four waveforms, two differential signals in waveform indicators.  But I can't do anything with them, because I can't even create the arrays of the waveforms' data.

Conclusion: LabVIEW is incompatible with large data sets.  The bare bones is creation of the arrays and do zero else.  It can't even do that.  I see OOP as my only option, because w/OOP I get the heap.

[*3d0g]

I'm noting your arrays being created via a loop.

How do I loop through a 32MSa waveform?  Can it be done?

The assumption of course is that's the reason my simple vi ran out of memory: because the array was defined into existence and filled instantly vs being appended into existence as a loop ran.

...Windows kept finding memory when LabVIEW asked for more, 10,000,000 times in a row...