Storage and processing of large volumes of data

[daisy_11]

At present, it is Labview2017x 64. I will combine this 3X90000000 into 2d array data and write it into DAQmx for multi-channel analog voltage output. In addition, I will carry out for loop to realize dynamic waveform display

Is there any other method suitable for my needs

[altenbach]

so why isn't the data combined from the beginning into a 2D array?

---

@daisy_11 wrote:

At present, it is Labview2017x 64. I will combine this 3X90000000 into 2d array data and write it into DAQmx for multi-channel analog voltage output. In addition, I will carry out for loop to realize dynamic waveform display

---

Your description is way too generic to comment. What is the output rate? What is the hardware? What is the dynamic range? Why do you need 8 bytes per value (DBL)?

I have no idea what most of your words mean. What is a "dynamic waveform display" and why does it need a FOR loop?

---

@daisy_11 wrote:

Is there any other method suitable for my needs

---

No way to tell form the given information. How many other places is the same FGV data needed?

[daisy_11]

The number is a decimal like -3.918338, so double is used

The 90000000 data were displayed dynamically in the images chart, so the for loop was adopted to display the waveform similar to the simulated oscillograph

[altenbach]

How many bits in the AO converter?

I still don't understand your descriptions because you are throwing more and more undefined terms at us. For example, what is an "images chart"?

[daisy_11]

I just want to know what is the best way to store this huge amount of data. FGV, while satisfying my needs, is too slow in reading and writing

[Martin_Henz]

We can't help, because you do not give us enough informations.

[altenbach]

Unfortunately we still don't know your "needs" at all. This needs to be solved with an overall design, focusing on the FGV is way too narrow!

Can you attach a simplified code skeleton so we get a better idea? How often is data written to the FGV? How often is it read out?

I am sure once we have more complete information, we can start with better suggestions.

[Yamaeda]

I'm assuming you're getting memory copies and possibly swapping if it's that slow. Assuming you're only Reading small parts at once, one thing that can help is In Place structure.

I made a post about a similar issue with an Array of clusters that had large arrays. I'll see if i can find it.

[altenbach]

Another option would be to change the read function to only return the needed subset (e.g. a column of 3 elements or suitable subset required for one AO call)  instead of the entire gigantic array. Continuously shoving gigabytes of data across subVIs boundaries is just insane.

How much data are you giving to the AO function at any given time? That's how much should be retrieved from the FGV (e.g. by providing inputs for offset and length).

You really also should get rid of the init case. I can't see any reason to ever clear the shift registers except if there is a new state that is guaranteed to never require the FGV data ever again. Each write will overwrite and (currently) each read gets all the data. Arbitrarily resizing to zero momentarily might lose an existing array allocation, requiring extra work. Your FGV is so simple, it could even be replaced with a global variable.

You still haven't really told us about the "processing" needs.

[daisy_11]

Attached is a simple program framework

1，Read_TDMS.vi    reads the 900,00000 data required

2,  FGV_Datas.vi       Store the processed data for writing to DAQmx for multi-channel analog voltage output

3,  FGV_ChannelDatas.vi     Raw data used to display waveforms

4,  showdata.vi           Display dynamic waveform

First read the TDMS data, store it in FGV, then dynamically display the old data waveform, and write FGV_Datas,vi  data into DAQmx