deleting data from binary file

[daksoy]

I need to acquire data for a very long time, however I have limited disk space. Therefore I need to delete data while I am writing it to the disk. For instance, after writing 1 terrabyte of binary data to a single file, I should start deleting it starting from the beginning of the file such that it won't exceed 1 terrabyte. Should I use TDMS or what?

Any info appreciated!

[YongqingYe]

If you are using TDMS files in LV 2010 and later version, I would suggest you trying out TDMS Advanced API. It has a VI called "Set Next Write Position" which can set the cursor in the file back to the beginning of the file.

[johnsold]

If you corrupt that file you may lose all of that terabyte of data.  Can you possibly write ten 100 GB files each day.  When you are ready to start the eleventh file, delete the oldest one.  This would also simplify making backups or transferring data to archival storage, shoudl you decide to keep more than one day's data.

Lynn

[daksoy]

This can certainly be a solution, however what I have in my mind is much more complex than simply deleting the oldest data. When I reach to a certain size (lets say 1 terrabyte) I want to start deleting chunks of data from the beginning of the acquisation, however at the same time keep some of them (lets say I keep 1 gigabyte for every 10 gigabyte removed) and then keep going on with deleting. Eventually, I should have a data which looks like:

10GB removed - 1GB stored - 10GB removed - 1GB stored - ............. - 1GB stored - 10GB removed - 1TB (or more) stored

Do you think that the only solution is to write 1GB files?

[daksoy]

This can be a solution if I also store the timing information along with the acquired data. Also I should upgrade to LV 2010 🙂

[johnsold]

It is not easy to remove data from the beginning or middle of a file.  Typically the way that is done is to read the entire file, extract the data to be saved and then write it to a new file.

For the 1 GB - 10 GB process you described you might be able to to use 10 GB files. One file would be the "working" storage file.  When you remove the first 10 GB, write 1 GB to the working file. When you remove the second 10 GB file, append 1 GB to the working file.  Continue until you have 10 GB in the working file, then start a new one.

The details of the bookkeeping for the fresh, removed, and stored data will need to be determined according to your needs for the data.

Lynn