Help!! Trying to writing an array to a file without having to rewrite the old data every time.

[dlovell]

Hey Everyone,

I have a vi that takes a 2D array and writes it to an xml file. The purpose for this is to characterize the pathloss through a dowkey 10x10 matrix at varies frequencies. I use this program to create a look-up table for the switch matrix so when I conduct one of my automated tests I can obtain an accurate measurment. The problem with this is I am taking about 299 data points per switch matrix combination leading me to over 32000 data points in the look-up table. The reason I use xml is because each data point needs a header so I can parse through the look-up table with another one of my vi's when I need that pathloss. What I am trying to fix is that when my vi writes to a file it writes one array at a time to save memory space. When it writes it rewrites the old data then the new data. As the number of data points increase so does the lag time to write to the file. By the time I am done it takes about five hours to fully write to the file. Does anyone know of a way to write continuesly write to a file without having to rewrite all of the old data? Attached is my vi for writing to the file, my vi for searching through the file, and an example of one of my look-up tables.

Thanks,

Dustin

[crossrulz]

---

dlovell wrote:Does anyone know of a way to write continuesly write to a file without having to rewrite all of the old data?

---

The only way to do that is to only append to the end of your file.  This would probably require some rewriting of your header information.

[dlovell]

Hey crossrulz,

Thanks for replying but how would I only append to the end of the file only? I have tried many different methods in my "Write to XML.vi" but if I don't attach the xml file to string 0 of the "Concatenate Strings Function" it just overwrites the entire file...

Dustin

[hecmar.arreola]

XML files are a compromise between human readability and machine iterpretability.

I recommend you rework your project to save the data in a binary format (regularly) and provide "Export to XML" and "Import to XML" functionality.

[dlovell]

My problem comes from the fact when I try to write to the file if I don't read and store the contents first it over-writes the entire file... I have been using a concatenating string function to do this. Is there a better way to do this where you don't have to store all of the previous data and rewrite it everytime? 

[KathrynB]

It might be beacuse of using the write to XML VI. I have a system that is writing waveform data continuously (as in the writing speed is just about the same as the aquisition speed, and is luckily quite slow as it runs at about 1kHz), which I had to do as ASCII as TDMS or binary are not human readable and the final file needed to be.

I am just creating the file in a write only mode using 'Open/Create/Replace File', and using the refnum to just write the text to the file with 'Write to Text file' (which does actually take a refnum if you wire one to instead of a path - it's this use of a refnum that gives me the access speed) in a loop. This always appends the data.

Of course if you go into the advanced file functions then you get a whole host of options such as offsets from starts of file to use, where the current position is, but this all relies on that same refnum capability. The downside to all these is that YOU will have to format your dat instead of getting LabVIEW to do it for you, but then, is the coding time worth the saved user time, and the improved user experience?

[dlovell]

Hey KathrynB,

I was looking into that but I am having trouble wrapping the design around my head... Is there anyway you can send me a screen shot of vi where you are able to continous write to a file without having to rewrite all the old data? I do much better with visuals...

Thanks,

Dustin

[KathrynB · Accepted solution]

Hi,

Just in case others are having an issue, something along these lines as a snippet: