LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

How to save a 3D array? Or how to save a spreadsheet string for future retrieve?

I want to save a 3D array, preferable to Excel spreadsheet.
Common VIs for writing to spreadsheet are for 1- or 2-D arrays.

I first thought to convert the 3D array to a spreadsheet string. But then I did not know what to do with the spreadsheet string. I have to be able to put the original 3D array in a local hard drive for future reading from.

Thanks for any hint.
0 Kudos
Message 1 of 12
(7,203 Views)
hi BLPig,
You can take a look at this thread http://forums.ni.com/ni/board/message?board.id=170&message.id=14628

hope it helps.
0 Kudos
Message 2 of 12
(7,190 Views)
hi, thank you for your information. The hints are really helpful. However, since my 3D array is way too huge, 101x181x2500, so if I:

1)try to save it as multiple 2D spreadsheet files, it would be too many;
2)try to use "array to spreadsheet string", LabVIEW popped the msg that "not enough memory" (PS: why??? I have no other windows threading running)
3)try to save all the 2D arrays into a single spreadsheet file by setting "append to existing file". All the 2D's goes to a single sheet ... and would not be able to display completely when the file is opened.

I do want to try the way save all 2D's into a single Excel spreadsheet file but have each sheet to have only a 2D array. Could anyone tell me how to write up the header to make this?

Many thanks.
(refer to my post at http://forums.ni.com/ni/board/message?board.id=170&message.id=14628 if you have a second)
0 Kudos
Message 3 of 12
(7,183 Views)
"2)try to use "array to spreadsheet string", LabVIEW popped the msg that "not enough memory" (PS: why??? "

because

101 X 181 X 2500 = 45,702,500

Assuming you use 5 ASCII characters to represent each value means that you would need

229 Gig of memory just to buffer the data.


As in all Q's of this sort the answer is to "divide and conquer".

FYI
Excel will only display 65535 rows of data.

Ben
Retired Senior Automation Systems Architect with Data Science Automation LabVIEW Champion Knight of NI and Prepper LinkedIn Profile YouTube Channel
0 Kudos
Message 4 of 12
(7,179 Views)

I would strongly second Ben's suggestion to divide and conquer. If you have not already done so, you might want to check out the tutorial on Managing Large Data Sets in LabVIEW. You can either search for it by name or try this link.

I don't often push people to non-ASCII save formats, but putting about 45 million data points into an ASCII file is going to be very slow and cumbersome, not to mention inefficient. You need a binary solution. If you have some time, I would suggest you check out HDF5. It is fast, multi-platform, efficient, hierarchical, and has native data compression. On the minus side, it is not thread-safe (on Windows, it is on Linux) and very low level (so somewhat user unfriendly). There is an unsupported Windows LabVIEW solution on the DevZone here. It does not support 3D data sets, but a minor modification on one of the save routines will do that for you (take the save 2D routines and change the dimensionality to 3, then change the input arrays to 3D). NI has been using HDF5 on shipping products for about four years.

If you don't have time, stream your data to disk as binary using the LabVIEW file primitives. Make sure you record your data format so you can open the files at a later time (this is an advantage of HDF5, it is self-describing, so you can always find the format).

Good luck.

0 Kudos
Message 5 of 12
(7,167 Views)
I just noticed my math error here.

I have always had trouble counting decimal palces before I have had my coffee!

Please forgive,

Ben
Retired Senior Automation Systems Architect with Data Science Automation LabVIEW Champion Knight of NI and Prepper LinkedIn Profile YouTube Channel
0 Kudos
Message 6 of 12
(7,167 Views)
Nobody reads 45 million data points in ASCII.

Save it as binary, and then write an app that will chop out what you really need (or average it, or graph it, or whatever) and save that in a human readable form.

A quick and easy way to save it as binary is to flatten it to string (Functions -> Advanced -> Data Manipulation) and write it out as a text file (but don't use a .txt extension). To get it back, read it in as text and then unflatten it back to a 3D array.

Good luck.
0 Kudos
Message 7 of 12
(7,147 Views)
Thank you all for the help. I attached a simple testing program modified from some example I searched in this forum. I had the msg "memory not enough" and all applications were halted thereafter .. and I have to reboot my computer. I am not familiar with file IO at all so I am not sure where I am wrong. If anyone could take a look into the attachment, thanks a lot!

Thank you all again. Have a nice holiday
0 Kudos
Message 8 of 12
(7,145 Views)
In your example, the wire coming out of the init array contains about 400MBytes of data. You split the wire, creating data copies, in several places. I have never been able to get LabVIEW to use more than about 1200MBytes of RAM - this is two extra copies (and yes, I have tried). You probably have somewhere between four and eight copies, depending upon what version of LabVIEW you are using. Read the tutorial I mentioned earlier. You will need that information to survive this programming experience.

I must respectfully disagree with VI Guy about saving using flatten to string. First, flatten to string makes a copy. You have enough copies already. Second, when you read the data, you must know exactly how it was flattened or you lose it. My experience has been that such information gets lost in about three to five months. Third, you can write the data directly to disk using the write file primitive - it takes any data type. You will still need to know how you wrote it, but if you interleave it in a obvious fashion, it will not be as opaque as a flatten to string and you can figure it out again if you need to in the future. Use the example finder to browse and look at all the examples on file I/O, especially the binary ones. You can also open the binary file storage VIs and look at how that was done, then modify for your case.

Is your data being generated all at once or is it coming piecemeal from somewhere? If piecemeal, store it as it is generated. If all-at-once, use the techniques in the tutorial to store it, then save it piecemeal. You want to avoid 400MByte wires at all costs. Dealing with this much data is an advanced topic, so don't be surprised if it takes you some time to figure out. The first real question you should ask yourself is “Do I really need this much data?”

If you do need that much data, I recommend you use HDF5 for this. It will take you two weeks to figure out, but you will be glad you did.

(NOTE: I will be out for vacation until next year.)
Message 9 of 12
(7,135 Views)
One more comment, you can get a browser for HDF5 files from the HDF5 website. This allows you to look at your hierarchical data as if it were a file system - very nice. However, it will not display a 400MByte data set. Fortunately, writing a “browser” in LabVIEW to do this is pretty trivial. There is an example for a 1D data set in the LabVIEW API I mentioned earlier.
0 Kudos
Message 10 of 12
(7,130 Views)