tdms first write time delay

[Snowpac]

I'm trying to stream a large amount of continuous digitization data from a Flexrio/FPGA front ent to

a HDD-8265 streaming disk. I'm using teh advanced TDMS functions, preallocating the

file size with the correct sector size, etc and when I stream 2 G -50 G samples (u32)  It works no

problem. 

As I increse my sample size from 2 G to 50 G I noticed that the Target-to-Host FIFO through which I am sending

my data starts to get fiulled up upon the first time I try to write the .tdms file. Once the adviced write

is called it's firt time, everything is ok and the advanced write catches up.  

I notice when I exceed 80G samples the Target-toHost FIFO overflows and I lose events. I am trying to fix this problem; I need to retain all events as I am providning an absolute (GPS)  timestamp

which is generated by a  the fixed number of events from the start of the data acquisition. If I drop events, I lose my ability to TimeStamp any particular DAQ sample.

I am keeping my sampling rate the same throught the tests, so it looks like the problem is associated with 

some sort of seek time for the first write to the .tdms file; it seems to get longer and longer 

the larger the file size, and then once it does the firt write it is no problem.  Once the HDD-8265 starts to 

write, it has no problem keeping up; it is just some sort of initialization delay which is proprtional to the

allocated file size.

I'm preallocating the space using the advanced .tdms functions and I've tried putting a wait after that function but

that has no effect on the time delay of the first write.

Is there some way to prime the HDD-8265 to get it ready to take data without actually writing events? I suppose I coudl write a dummy block of fixed length, wait a second, and then enable the DAQ system, but this seems like an awfully silly thing to do. I'd rather not write dummy blocks of data to the HDD-8265 just to get get rid of that delay before the disk can fully handle the cointinuous data stream.

[Bo_Xie]

Snowpac,

Could you provide a simple VI so that I could reproduce your issue?

Best Regards,

Bo Xie

NI R&D

[Snowpac]

Here is my previous message with typos fixed. Somehow I cannot post more than one file. ....here is the main .vi ---------------------------------------------------------------------------------------- I'm trying to stream a large amount of continuous digitization data from a Flexrio/FPGA front ent to a HDD-8265 streaming disk. I'm using the advanced TDMS functions, preallocating the file size with the correct sector size, etc and when I stream 2 G -50 G samples (u32) It works no problem. As I increase my sample size from 2 G to 50 G I noticed that the Target-to-Host FIFO through which I am sending my data starts to get filled up upon the first time I try to write the .tdms file. Once the adviced write is called it's firt time, everything is ok and the advanced write catches up. I notice when I exceed 80G samples the Target-toHost FIFO overflows and I lose events. I am trying to fix this problem; I need to retain all events as I am providing an absolute (GPS) timestamp which is generated by a the fixed number of events from the start of the data acquisition. If I drop events, I lose my ability to TimeStamp any particular DAQ sample. I am keeping my sampling rate the same throughout the tests, so it looks like the problem is associated with some sort of seek time for the first write to the .tdms file; it seems to get longer and longer the larger the file size, and then once it does the first write it is no problem. Once the HDD-8265 starts to write, it has no problem keeping up; it is just some sort of initialization delay which is proportional to the allocated file size. I'm preallocating the space using the advanced .tdms functions and I've tried putting a wait after that function but that has no effect on the time delay of the first write. Is there some way to prime the HDD-8265 to get it ready to take data without actually writing events? I suppose I could write a dummy block of fixed length, wait a second, and then enable the DAQ system, but this seems like an awfully silly thing to do. I'd rather not write dummy blocks of data to the HDD-8265 just to get get rid of that delay before the disk can fully handle the continuous data stream.

[Snowpac]

It looks like the opera browser doesn't allow attachments in the forum. Her ei am postin gthrough Google Chrome....

Her eis the fpga .vi, the main program, and the .lvproj file.

I'm running on a NI 5761 Flexrio, pxie7965R virtex5- fpga, HDD-8265 disk and pxie interface, pixe controller and PXIe1082 backplane.

The controller is the new Win 7 unit quad-core...4 GB ram..PXIe8133...

[Bo_Xie]

Snowpac,

Thanks for providing test VI files!

I have not NI 5761 Flexrio at hand and then I could not run your test VI to reproduce your issue.

Could you try the following workaround? (I have not verified this workaround by myself because I could not reproduce your issue)

[Workaround] Append "TDMS Flush" to "TDMS Configure Asynchronous Writes" as shown by attached Append_TDMS_Flush.png.

BTW, you might set "pre-allocate(F)" terminal of "TDMS Configure Asynchronous Writes" to "F" if you are using LabVIEW 2011 to get better performance.

Please let me know whether this workaround solves your issue.

Best Regards,

Bo Xie

[Snowpac · Accepted solution]

Ok, this definitely helps. If you have a large file, say 1TB, then you still need to wait several seconds after the flush before teh 8265 is ready to take data. It seems that the drive starts to map out the file allocation after the flush.

I put a time delay of 5 seconds afterthe flush and this seems to work perfectly for a 1 TB file. I suspect 5 TB 

would require maybe 10 s or more. See the attached .png

I did not consider that you could mix regular and advanced .tdms functions..Thanks for the help!