Trouble with Concurrent Acess of TDMS file in LabVIEW 8.6

[aartjan]

nope. just error 2503 and the index_index_index issue.

I am going to examine my applications some more to see what might cause the 2503 error.

My real goal is to have analysis applications running files while the measurement application is writing data, so my ultimate goal is not needing to close and create a new tdms reference for every read. (it doesn't read any new data either, not just the channel length)

[YongqingYe]

OK, that's good, then let's focus on the error of -2503 and the index_index_index file. It looks a little bit weird to have the index_index_index file, seems there's something wrong when generating the index file. The error code -2503 means the file (either .tdms or .tdms_index) is corrupted, so

• Does this problem only happen in LabVIEW 2009?
• Would you please try to delete the index file and then read the tdms file?
• And is that convenient for you to post a VI which can reproduce this problem or the weird index_index_index file here?

Thank you!

[aartjan]

Sorry, I am not going to share the tdms writer app I am using; it's a production app for our business. It doesn't matter which reader I use (my own or the one shared before). The tdms is not corrupt as the -2503 error indicates; they read just fine as soon as the writer stops writing. So; the error is actually erroneous

The fact that reading the index file creates an index file of the index file is not of interest to me. I just mentioned it because you may want to know about such peculiar behavior.

What I can tell about my tdms writer executable: It is using the same runtime (9.0.1). I even played with several tdms open modes (create, open or create, create or replace etc): no change. Also I have several local variables of the tdms reference and multiple write locations. Just one reference open to the tdms, though.

Of course both reader and writer are in tdms 2.0 mode

[Herbert Engels]

The VIs I posted earlier in this thread are special only in that "disable buffering" is set to "false" on TDMS Open. That's what fixed the problem back then. Have you set that flag in your application (note that default is "true" on this terminal, which is not what you want here)?

Background

If we write with Windows buffering enabled, every write operation extends the file exactly by the space needed for the data you passed into the TDMS API, so every byte of the file is occupied by valid TDMS content. If buffering is disabled (which is necessary so we get reasonable performance e.g. out of RAID arrays), we need to write everything to disk in chunks the size of an integer multiple of the disk sector size, so there will be some uninitialized content at the end of the file most of the time. We do truncate the file on TDMS Close in order to get rid of this unused area. But we can't do that any earlier, because that causes a pretty massive performance hit.

Error -2503 occurs when we scan the file for tokens that we annotate our meta data with, and we run into a section of the file that, in operating system terms, is part of the file, but, as far as TDMS is concerned, has not been written to yet. Within the same process, we can keep track of our "real" writing process with variables that refnums can share. Across processes, we don't have a solution for this right now. We do know what the solution is, but it'll require either a change in the binary layout of the file or a really dirty workaround. We are trying to keep version changes in TDMS to a minimum, so we basically collect incoming CARs and feature requests that require these changes until the combined urgency of these requests hits a certain threshold. Based on what we know at this point, we're still well below that threshold. Ideally, the buffering flag will fix your app. Otherwise, we would have to reconsider either the format change or the workaround. Please let us know how this works out.

Thanks,

Herbert

[aartjan]

Hi Herbert, I will try the buffering flag.

If I understand you correctly the reading of new data is not going to happen until a "tdms 3.0." Unfortunate, but I understand the reasoning.

Thanks for the detailed reply. I will post my test results shortly.

[blawson]

I'm sorry to butt in here, but it seems like Aartjan's issue is within the known limitations of concurrent read/write:

With a process for writing and one for reading, the read reference only has access to what was written at the point the read VI's TDMS reference was opened.  To get newer data, close the reading reference and reopen it.

With different references in the same process, there can be a gap between the data written on one reference and what's read on the other, and you can close that gap by calling TDMS Flush on the write reference.  (and buffering affects this, too?)

is that accurate?

[Herbert Engels]

That is accurate. The reading process appears to catch an error, though. That might be a side effect of how unbuffered writing on Windows works.

Thanks,

Herbert