LabVIEW and large data sets are incompatible. T/F? I say T.

[Eric1977]

I agree with other forum members here. You haven't given us any code for us to look at to even begin trying to understand what you are doing or even where the problem lies. All you've done is babble about what's wrong.

Your VI will say more than your words will.

I'm certainly not going to go back and read all those posts. No one will.

As stated by many other forum members, you need to give us an example VI of the exact problem you are seeing. We don't need the full blown app if it's proprietary, just an example of the problem. And save it back to version 18. Do you know how to do that?

If you can do that, the many brilliant minds here can help! Help us to help you!

[JÞB]

---

@Eric1977 wrote:

I agree with other forum members here. You haven't given us any code for us to look at to even begin trying to understand what you are doing or even where the problem lies. All you've done is babble about what's wrong.

Your VI will say more than your words will.

I'm certainly not going to go back and read all those posts. No one will.

As stated by many other forum members, you need to give us an example VI of the exact problem you are seeing. We don't need the full blown app if it's proprietary, just an example of the problem. And save it back to version 18. Do you know how to do that?

If you can do that, the many brilliant minds here can help! Help us to help you!

---

And while you are posting the pngs.

•   Take the time to label the wires
• Add comments 
• Explain hidden cases

You show no shift registers yet keep referring to them.

You still have not stated a requirement.   "I want to work with massive arrays " is not a requirement.   A requirement would be similar to "the system must record the maximum differential Voltage between two channels within 500mSec after an event with +/-100uV resolution "  Note: the requirement may not involve moving massive arrays. 

[Intaris]

---

@*3d0g wrote:

One last thing, for now, unless I suddenly get a solution, of course, is what I'm doing right now technically didn't hi-jack my own thread, technically, for what I am doing right now is exactly what I would be doing with the LeCroy, as far as I know.  I'd be drilling down to the raw data, at the API, bringing it out and processing it.  Their presented vi can't handle this number of samples, at least not with what I have to work with, bits-wise.  (There's actually a term for that, but I can't recall it and haven't for a while; it's not "bits-wise."  It's a single word. Nor is it bitness.)

Change to 64-bit LabVIEW!  I can't do that!  That is not an option.  If there is a solution, I want to find it -- I'd like to find it with your assistance, because that would reduce the time I must search and test hypotheses.  Granted, yes, I started this thread with a scenario, but it's actually real.  I'm trying to solve a real problem.  I figured in the end I'd either solve the LeCroy problem, or I'd solve the PicoScope problem; the thread was just launched in a generic/hypothetical situation. I was asking you, experts and novices alike, what would you do to prove LabVIEW, in my situation (32-bit), can handle this?  So far the only hypotheses for solving this real problem are change to 64-bit LabVIEW or use shift registers.  Hence, so far, reflecting back to the title of my thread, the answer is T.    

---

One problem we ran into was similar to this. I ended up utilising a trick of Windows memory management and file caching.

I designed an LVOOP version of an array which assumed a fixed-size array. I then made all of the methods possible without requiring memory copies, and functions like Max&Min or Average operated on the data in chunks so that only parts of the array are ever in memory at a time.

Then, when populating the "array" I would write to a binary file. Windows 64-bit has a great habit of caching all open files up to any including the full OS memory limit. The end result was not quite native array speed, but very surprisingly close. In the background, the OS was reading and writing to disk, but since the whole lot of the data was being cached (in 64-bit Windows) we were kind of exploiting far more than 32-bit LabVIEW could ever offer. Not only was the entire array not in memory at once, practically all of the operations we needed could also be implemented on a chunk-by-chunk basis, massively reducing the overall memory footprint.

It's not a simple thing to do, and I'm not at liberty to share code, but it has made working with large datasets (similar to your here) much much easier, if a slight bit slower than native. But getting sometrhing to work is always better than not having it work.

See HERE for the background that actually led to me thinking of this approach.

[altenbach]

---

@*3d0g wrote:

Now just follow line one with a disable block that either keeps the trace or injects the local variable.   

---

I can guarantee that any "solution" involving local variables will not solve any memory problems. I also don't understand the need for all these dlls running in the UI thread. A clean, "pure G" solution with data in wires (not in front panel elements and their locals and buffers) will allow the LabVIEW compiler significantly more places to optimize things.

[*3d0g]

Hello Intaris!  Thank you so much for participating in my thread.  What a great post!

Now we're talkin'!  That's the kind of thing I'm looking for in this.  (I'm making the assumption you have read the thread to this point.)  Kudo heading your way!

[*3d0g]

"I can guarantee that any "solution" involving local variables will not solve any memory problems."

exactly

"I also don't understand the need for all these dlls running in the UI thread."

Please explain a little.

"A clean, "pure G" solution with data in wires (not in front panel elements and their locals and buffers) will allow the LabVIEW compiler significantly more places to optimize things."

Again, please explain.  You understand that the local is a workaround, right?  I don't want it there.  It has to be there because LabVIEW doesn't work as advertised.  You understand that, right?

(...no offence, LabVIEW.  That's what a bug in a consumer product means.  It means the product does not work as advertised.  Tunnels are supposed to work as tunnels; that's why they are tunnels not brick walls ...that some how retain their proper size going through them??  No LabVIEW, I'm no giant, but I know what a tunnel is and what a broken one looks like.  That tunnel is broken.)

[*3d0g]

Here is a picture of great importance to me:

[Bob_Schor]

---

@*3d0g wrote:

(...no offence, LabVIEW.  That's what a bug in a consumer product means.  It means the product does not work as advertised.  Tunnels are supposed to work as tunnels; that's why they are tunnels not brick walls ...that some how retain their proper size going through them??  No LabVIEW, I'm no giant, but I know what a tunnel is and what a broken one looks like.  That tunnel is broken.)

---

Another post that I cannot understand!  What "tunnel is broken"?  I haven't seen any code with a tunnel -- as others have noted, you've been ranting and raving, but haven't shown us any data/examples/code that we could say "Yes, you found some interesting bug or feature", or "No, it is supposed to be like that".

I'm reminded of some code I wrote once where I knew there was an error created on the Error Line inside a loop, yet the Error Line came out "clean".  I wrote a little snippet to illustrate what I mean --

Where is the Error?

There are two output tunnels -- one passing the number (5, in this case) passed in to "Result", and the other generating a User Error 5000 and passing it to "error out".  So what will be the values in Result and in "error out" when you run this?

[Remember the Laws of Data Flow].

Maybe this is "byting" you somewhere in the Code you refuse to show us.  Does it make you feel superior to the hard work that the LabVIEW developers did creating LabVIEW that you cannot improve your own code, and are unwilling to share it with us and maybe teach us something about LabVIEW we didn't know (at the risk, of course, of us "finding the bug" and teaching you something about LabVIEW that you didn't know)?

Come on, show us the code, let us try to explain why "it's supposed to do that!", or "It's not supposed to do that!".  We don't even know what "It" and "that!" are, for goodness sake!

Bob Schor

[*3d0g]

Bob:  You're going to hate me for this, as you seem like a nice reasonable person; I believe this as I just happened to catch a post you left responding to a student only recently.  I sent you a kudo for returning to help when they (I'll assume) didn't get it and you calmly tried again.  (I am super busy.)

But Bob, it's always a choice of whether to take a test.  It's about the value to be gained at the end.  If the student sees little to no value, they won't take the test.

You'll see the code if you are at National Instruments.

[*3d0g]

Bob: Also, this thread isn't about me teaching anyone anything.  It's me learning.