how to save and retrieve 3D data

[ccyang]

Hi, all:

I have several 3-D data and each one has a very large and different size. I saved them in 1D using Write to binary file after reshaping the data, and retrieve it after the user key in the size of the data so that my code can reshape the data into a correct size.

My first question is, is there any way to save the data directly into a 3D format so that the user can retrieve the data without knowing the size? For example, I have the data A in 55(row)x512(column)x500(frame) and the data B in 3(row)x1004(column)x1500(frame). Is there any way to save and retrieve them without the need to know their size in advance?

My second question is, I will generate several data set like this and save and retrieve them during data collection. I found that it always takes a long time to save the data, and longer time to retrieve the data. Is there any way better than saving in binary to reduce the time?

I am using LV 8.2. I tried to search the previous posts for the answers but found that some downloaded examples gave me a warning that my current version couldn't support the function.

I would very appreciate for any suggestion and example code to help me. Thank you in advance.

[sture]

I suppose you could just add information about the 3 sizes to the data you save. Then retreive the information aftre loading the file again.

Something like the picture attached.

http://forums.ni.com/ni/attachments/ni/170/353619/1/3d-array.JPG

BR

Sture

Message Edited by sture on 09-04-2008 04:11 PM

Message Edited by sture on 09-04-2008 04:12 PM

[smercurio_fc]

---

sture wrote:

I suppose you could just add information about the 3 sizes to the data you save. Then retreive the information aftre loading the file again.

---

The Write To Binary File does this automatically if you set the "prepend array size" to True (which is is by default). Thus, all you need to do is wire your 3D array directly to that function. Example:

 

---

My second question is, I will generate several data set like this and save and retrieve them during data collection. I found that it always takes a long time to save the data, and longer time to retrieve the data. Is there any way better than saving in binary to reduce the time?

---

The amount of time is related to how much data you're writing to disk. You can try using the ZIP functions to compress the file, but you probably won't get much out of it.

Message Edited by smercurio_fc on 09-04-2008 04:25 PM

[altenbach]

By default, LabVIEW will prepend the array size when you write binary data, so all you need to do is wire an empty 3D array of the correct type to the "data type" input when you use "read binary file" and everything will fall into place. Have you tried?

(Edit: Ahhh, Smercurio... was faster once more. :D) 

Message Edited by altenbach on 09-04-2008 02:32 PM

[ccyang]

Thank you all! I did not expect such a fast reply. I am going to try your suggestions next week. Looks like I don't need to worry about the size of the 3D array. Thanks again!

[ccyang]

Thanks again for everyone's suggest. I think my question is solved, and for alterbach, I wonder how you made a 3D data comparison to receive one boolean output? I tried to repeat your code, but I found I must connect a 3D boolean indicator to the equal comparison. What did I do wrong?

By the way, I found to save one of my example data in 1000(column)x55(row)x500(frame) of DBL may usually run out of my memory and receive a warning message from LV. Is there any way to avoid such a problem? The users don't like the result to be saved in zip format, and the code may need to repeat saving and retrieving for several times. I really appreciate for any further sugestion. Thank you again.

[altenbach]

---

ccyang wrote:

Thanks again for everyone's suggest. I think my question is solved, and for alterbach, I wonder how you made a 3D data comparison to receive one boolean output? I tried to repeat your code, but I found I must connect a 3D boolean indicator to the equal comparison. What did I do wrong?

---

Right-click on the "equal" and select "comparison mode...compare aggregates". Voila!

---

ccyang wrote:

By the way, I found to save one of my example data in 1000(column)x55(row)x500(frame) of DBL may usually run out of my memory and receive a warning message from LV. Is there any way to avoid such a problem? The users don't like the result to be saved in zip format, and the code may need to repeat saving and retrieving for several times. I really appreciate for any further sugestion. Thank you again.

---

This is over 200MB of data. So you need to code very carefully to avoid extra datacopies in memory and always re-use the same buffer. Also, arrays are stored contiguously, posing additional problems.

[ccyang]

Thanks, altenbach! However, I don't really understand what you meant by "avoid extra datacopies in memory and always re-use the same buffer". In the attached picture, you see what I tried.  I put "Close File" after "Write to Binary File" and "Open Bianry File". I keep comparison and showing data to simulate the work. I re-run the code for 4-8 times then the code run out of memory. I believe that my computer's memory is large and nothing else occupies it. Any suggestion to improve the memory usage? Thank you again for your help.

[altenbach]

Can you attach the VI instead of a picture?

[ccyang]

Yes here it is.