How to restrict Datalog file size

[Gurdas]

Hi!,

I am acquiring waveform data (at 40K/sec)and writing them into a Datalog file. My application is required to run for 30 days non-stop.

However, I wish to retain only the last 10 minutes of data in my file. Any data previous to the last 10 minutes is waste.

How can I do this?

This is urgent.
Thanks in advance!

Gurdas
gurdas@qagetech.com

[aderogba]

You can time the acquistion and access the file with a different access mode when it's about time to acquire the data you want to keep. This may involve closing and opening the file or dynamic file access.

[Alberto]

Hi Gurdas,
you need to create a circular buffer on file of the last 10 minutes acquired data.
First take a look at the low level Write File function (you find it in the advanced file functions), the input Position mode sets where the data is written in the file: Start, End, Current. If no connected pos mode is set to Current, this means that each new data is added at the current offset position that, if unwired, corresponds to the end of file.

What you should do is to write data at offset position equal to offset after read (append data and fills buffer); when the buffer is full you force offset input to zero , this overwrites previous data.

I attach an example that shows the principle; note that in my vi I write a fixed string, but you can also conn
ect a waveform to the data input of the Write File function.

Let me know if you need more help,
Alberto

[Gurdas]

Thanks Alberto.

What you suggest is excellent but some issues remain in my vi (see attachment):

1) I am unable to connect "pos mode" and "pos offset". Shows error. Is it because the file type is Datalog?

2) I do not have a "wait for ms" function in my while loop. The acquisition has to take place at the fastest possible rate. What you see is just a simplified version of the actual vi in which I am acquiring 5 channels, one of which is vibration. So, is there a way to restrict file size in terms of seconds or do I have to resort to restriction via size (as I did in the attached vi)?

Looking forward to your reply.

Gurdas
gurdas@gurdas.com

[JeanPierre]

Datalog file are append-only so you can't use them as a circular file buffer. Use a bytestream file.

[Alberto]

Hi again,
I modified your example, it should work.

Anyway, if you need a fast acquisition and stream to disk I'd suggest a buffered acquisition. Lv has several examples, take a look at Cont Acq to File.vi you find in Stream to disk examples. You can apply to these vis the principle of the circular disk buffer.
You see that in these examples the data stored are numeric arrays, not waveforms; these increases performance because in waveform you have additional informations (t0 and dt).
One more thing, do not place the file info function in the while loop because it has to access the file at each loop iteration and could slow your acquisition. You can check the file size by the value of mark after read (offset), you won't cause any overhead.

To calculate the filesize corresponding to a time period you have to calculate the number of bytes written for this time.
Suppose you are acquiring 5 channels at 1Khz, you have 5K samples/s. If you save the data in I16 format (this should give you enough resolution 16bit), each data is 2 bytes long; thus you save 10K bytes/s.

Let me know if you need more explainations, good luck,

Alberto

[Gurdas]

Thanks Jean.

That means I am back to square one!

Can I work around by using the intermediate level File open/write/read VIs?

Also, in the circular buffer, when the pos offset is set to 0 doesn't one loose ALL previous records? So, how do I ensure that at ANY given time, my file contains data from the previous 10 minutes?

Another issue while reading files is that I want to index my 10 minute data using time stamps. Thus when I replay my latest 10 minute data file, I should be able to directly go to say 4th minute, 37th second.

The files being written in my vi are from endurance test of a bearing and thus it is important that data immediately prior to bearing failure is recorded in the file.

Also, I do not know what bytestrea
m files are.
Do you mean binary files?

Thats a lot of questions, and goes to show my stuck position 😉

- Gurdas
gurdas@qagetech.com

[JeanPierre]

Look at your example modified by Alberto. It should work.

You asked:
Also, in the circular buffer, when the pos offset is set to 0 doesn't one loose ALL previous records? So, how do I ensure that at ANY given time, my file contains data from the previous 10 minutes?

No. Files are random access. Provided that you write records of equal length, you can overwrite record N at offset N*recordlength without erasing other records. the file length stays the same after it has been filled once. Because the file is a circular buffer, you have to keep track of where is the oldest record in the file in order to retrieve in the order they were written.


You also asked:
Another issue while reading files is that I w
ant to index my 10 minute data using time stamps. Thus when I replay my latest 10 minute data file, I should be able to directly go to say 4th minute, 37th second.

The same to read data. compute the offset of the required data and read the file at this place.

Bytestream file are not typed like the datalog files. You can read/write any datatype in any order. Open/Create/Replace VI opens bytestream files.

[Gurdas]

Dear Alberto and Jean,

I must thank both of you for such excellent advice.
I finally managed to configure the write vi to my satisfaction.
Thank you again.

However, I still need some help with the read vi.
The file I am writing (and thus reading later as playback) is approximately 700MB in size and contains data collected over a period of 30 days non-stop testing. Time stamp is contained since time is relevant information. The file is read and 're-played' on the screen on a chart display.

My concerns:

1. How do I generate a single sheet printout of this data with X-axis as time and Y-axis as data value? The objective of such a plot is to give the user trend data for diagnosis and not precise values (similar to sales charts for
the last 5 years).

2. How can the I read the file from record number x whose time stamp I specify? For e.g. I am interested in the data from Day 6 (H1 hour, M1 mins and S1 secs) to Day 6 (H2 hour, M2 mins and S2 secs). Then, can I directly start reading from Day 6 (H1 hour, M1 mins and S1 secs) and NOT Day 1?

As always, I look forward to your insights.

- Gurdas
gurdas@qagetech.com