High Speed data collection throws error 4 (EOF reached) after 11 hours (1.5 GB file every hour)

[run321]

Hi folks,

I have a problem with a loop writing TDMS files. Every Hour a new file is generated. But after arround 9-11 hours the .vi stops with error 4 (EOF reached). Of course the drive is still not full (57 GB still free). From the timestamps, I see that the error comes inside a loop with no wait command inside. I added a 1ms wait for each iteration but it ended in an overflow of the input buffer -> waiting is not allowed at all. So I changed the value to 0. I do not know yet, if it help as I need to wait some more (11 🙂 hours. I use Labview 2016 and the Development is done on Windows 10 (32 bit), while the compiled project is running on a Windows 7 machine without Labview (only RTE). If you have a look at the attached figure: the motor current has a different timestamp, so the error was between the intermediate Curcuit voltage and the motor current.

I just want to ask:

- what more can I do to debug,... check the errors of all TDMS write functions and set a breakpoint in case of error 4?

- is there any other experience with high data throughput with TDMS file writing...

Thanks for your Help!!

Best regards

Matthias

P.s.: It is a huge project, so not so easy to send the original labview code. It was my first Labview program ever.

[stockson]

Do you use buffering?

Do you close the file after each writing event?

Do you access the file from any other place in the program?

[run321]

Hi Stockson,

thanks for your reply.

I use buffering for the incoming data (this is the buffer which overflows, if I add a delay of 1 ms). There is no buffering in the TDMS write function (I could not find one).

Of course I close the file if it is completely written. This happens arround 11 times (one per hour) without problems. But the error comes between two write commands (same loop), so there is no file closing inbetween :)....

Yes I access the file from different cases of a case structure,... in every case there is one or more TDMS write commands.

Best regards

Matthias

[stockson]

So is it possible that you try to access the file while the writing process is still in progress?

Since you are operating on large data sets in might be also a good idea to use the "defragment" function after each write.

[run321]

Hi Stockson,

it is a single while loop with a case structure inside and in every case there is a loop and minimum 1 TDMS write command. So I see no chance of two writing processes at the same time.I hope the writing process is finished when the TDMS write block has passed. I've also checked that if there are more write commands in one case, the error path is connected between both, so even the order is fixed.

I will try to use the defragment. Hope it will not take too much time, as putting it in every loop might take too much time.

The loop in the figure I posted (at the beginning) needs to run in 4 ms all 6 itrations => ~ 650us!! Seems not enough time for a defragmentation of 1.5 GB 🙂  ..... 

Best regards

Matthias

[stockson]

You can run the defragmentation in a separate loop, after the reference to the file is closed (between two writings).

[run321]

Hi Stockson,

sorry if I was not clear enough: if the file is closed, it is never opened again (in this vi).... to be opened later in Excel / DIAMS. The file is closed only every hour (changed from every day to every hour).

If the file is open, there is no time for defragmentation as I can not imagine that 1.5 GB can be defragmented in about 600 us. If all data is written in one file (start a new file once a day) produces the same error in about the same time (up to 9h), even the content is much more (contains 9 times more data)....

Best regards

Matthias

[stockson]

Ah, clear. I assumed that writing happens every hour and for the rest of the time the file is closed.

At this point I do not have any good ideas. What I would try to do is maybe add a case structure that in case of Error 4 reopens the file and tries to write again.

[JÞB]

Well, first off get rid of that wait 0.  It does things you don't understand that are bad in this case.  read the help file for details.

Next lets take a good look at your TDMS data layout.  it seams very odd.

your "Groups" are:

<GroupName>current position

<GroupName>expected speed

<GroupName>current speed...

Where each Group has the same channel list

and the data for each groups channel of the same name is created from a single scalar input from the cluster element named similar to the last part of each group

Now, I don't pretent to understand what is happening in the sub-vi that takes 2 scalars and returns a 2D array of Floats (sgl, dbl ext, complex varieties of the above? I just see orange) But I bet it is wrong.

[run321]

Hi Jeff,

and you think that this could be the cause of the "EndOfFile"-error? What is the difference to the first 11 hours before the error is thrown? 

Do you have suggestions what I can do to debug this problem? Timing is tough and the problem occours very seldom, so I do not know how I can get rid of this problem...

thanks for all

Matthias