Fifo on disk for hundreds of channels

[GabbyG]

Hi
I'm working on a project, where Labview is used to build a user interface for an OPC - data acquisition system. The data coming in via OPC are all kinds of status info from a plant in chemical industry. Currently selected channels are stored in a SQL data base on a one entry/sample basis. 
For the future more (>1000) channels and higher sample rates are envisioned, together with the option to only store statistical aggregates in the database, whereas the real waveform is only needed for some duration.
So I think of using a kind of TDM(S?) streaming to disk, but with the option to work as a FIFO. Is there a way to have this in TDM?.
I thought of having always the last 1 hour of data on disk, store say every 10 Minutes the average in th DB, and when an alarm-condition is triggered store the whole waveform as blob.
But if everything goes easy, just adapt the start time on disk and remove the first ten samples or so.

I remember, that TDMS gets _very_ huge if you do not compact it from time to time. Do you think erasing the head on disk does also present a performance problem ?

Thanks in advance for any ideas

Gabi

[AdnanZ]

The FIFO feature as you described isn't inherently available in the TDMS VIs. The best way is to program the implementation.

I am not sure if you will be able to purge a selection of data in the TDMS file but you should be able to keep writing an hour's worth of data in every file and then delete files whenever appropriate. Have a look at the following KB article that shows the use of TDMS Defragment:

Why Are My TDMS Files So Large?

[GabbyG]

Thanks Adnan
That's not a bad idea. I probably don't need an infinitesimal time fragmentation. If I start a new file every say 10 minutes I can simply erase  the old ones without going through fragment, make my mean, median etc. , and if really necessary can combine the earliest and latest and get almost double the time span.

Thanks so far

Gabi