LabVIEW
TDMS flexibility / performance
Never mind, I started up a VM with LV 2015 installed.
I saw this:
So also an increase in write time (again, you need more data to observe a trend). Final file size after this was only 33MB. Far away from the TB I will require and already a significant increase in write time is observable.
Shane.
NOTE: You made a mistake re-writing my VI, you wrote one dataset and then no more (the Remove Inside Loop version). Regardless how long it ran, it never added more data than 8200 points per channel, as such the benchmark is fatally flawed. I attach a fixed version here.
Which one of these should have a constant write time?
Hi Guys,
Just having another look through this thread. To backup some testing:
1. Intaris - your modified example still shows the issue unless I disable the inner writing of meta data. Based on everything else I suspect this merely slows it significantly.
2. Jie Zheng - Running the modified example on my system (native OS, LV2014, SSD (shared with OS) shows the same issue.
This backs up what I have seen before that TDMS performance only really works well when you are writing to fixed channels. Any time that you have to change channels there is a significant performance hit.
I've not seen anything in the format that determines this and can't exactly explain it in this case. I would love a write only mode which I think would require less caching of file structure although it sounds l ike it still has to go back and write a point in the lead in. That said would that really make a massive impact here? You would see a step change when it starts missing caches but then it should be fixed time on an SSD. It would also only need to maintain a fixed address in the file to modify so you wouldn't need to re-run over the whole segment structure
Anyway, hope that helps add to the data set. If I were you I would probably be discarding TDMS from this use case if you have to use multiple groups and channels this regularly. Clearly this use case doesn't work, perhaps it is a CAR but that wont help you now!
Thanks for the input James.
Fan of your work BTW. 
Yeah, it seems to be either a dodgy design choice for TDMS or it's a bug where the driver is doing more than it really needs to. I still would like very much an explanation as to which of these two it is because I'm surely not the only person to have made these assumptions on the performance of TDMS.
I don't want this issue to be swept under the rug because at the end of the day we (as a community) invest significant effort in the NI product range and really gain most benefit when the different parts of the ecosystem work well together. A proper technical explanation as to why things are as they apparently are will either save others the effort of going through the same crazy headache I've just been though with this issue or perhaps lead to a TDMS 3.0 or 2.1 which fixes this. I say fix because I cannot really understand why this performance degradation MUST be. I can think of a few workarounds to make the administration overhead (As you yourself have alluded to) to be constant time as opposed to linearly increasing time. I think this problem can technically be fixed and that must be our goal here.
But yeah, it's looking more and more likely that TDMS is dead in the water for us. We can't wait for a fix (if that would even be backported to LV 2012 which is a whole other can of worms.). Shame.
