NCL to TDMS Converter

[Eyeseeyou]

Hey Guys, this is going to be my first post so please forgive me if I forget to supply any crucial information.

What I'm trying to do with LabVIEW is creating a VI that uses the pre-built NI examples "Display NCL Logfile as 'Target-BUS' Frames" to write the content of those logfiles into a single TDMS File so that you can view multiple logs as one file.

My approach so far contains the actual top-level-VI NCL_Viewer. You first have to supply a file or directory path, in either case the VI lists every .ncl-file in the specified folder and calls the 'Display Logfile...'-VI for each of them.

This SubVI then determines whether the NCL-file is of the CAN, LIN or FlexRay type and converts it into the respective table but - different from the original NI-VI - doesn't write it to a table indicator but a TDMS-file. This, however, results in ~500MB consisting of the TDMS-file and the corresponding index - 50/50 ratio - which is bad, as I know and also takes approximately 10mins to write. Defrag reduces this to ~25MB TDMS and <1MB index.

FIRST QUESTION: Can I avoid this somehow? I'm not quite sure on how to use the buffering in this case, but considering the fact, that a defrag can severely shrink the filesize, I'm bound to believe it has to be possible to create smaller files in the first place.

Furthermore all those logfiles in one TDMS-file would be sort of chaotic to display, as TDMS allows only two levels of internal file structure. This is why I assemble my own new group names in a 'Filename-ID' scheme (e.g. CAN_01-963 for ID 963 in the first CAN logfile). I made my own modified version of the built-in TDMS-Viewer-VI so that it splits the group name into CAN_01 and 963 and creates a new tree structure like:

CAN_01

   - 963

       -Channels

Even though this is not the way the TDMS-file is built, I can make it look like it was.

SECOND QUESTION: I keep coming up with a number of CAN_01 entries corresponding to the number of different IDs logged on this BUS, instead of just one entry, listing all IDs. What did I do wrong here?

As to help you reproduce my problems I uploaded all the VIs in my project as well as one NCL-Logfile of each BUS-type.

Thank you in advance and forgive me if any passage is hard to read - English is not my native language.

PS: Check the path-controls/constants so they are valid for your PC.

[YongqingYe]

Hi Eyeseeyou,

For your first question, what the problem you have for your TDMS files? Do you mean the TDMS file is too large and after defragement of TDMS file, the size is reduced a lot, and you don't want to have a too large TDMS file and defragment the file by yourself?

Yongqing Ye

NI R&D

[Eyeseeyou]

I'd really appreciate it, if I could somehow just create smaller TDMS files to begin with. Of course a defragmentation solves the problem, but if I could work around the problem the whole VI might run considerably faster than it does now (large files render me kind of unable to do anything for 10mins).

And as I said, as a defrag really does help it's proven to me, that the resulting file is larger than it needs to be, else the defrag couldn't reduce the size by such a large amount.

[YongqingYe]

Hi Eyeseeyou,

There are a couple of ways for you to generate a smaller TDMS file:

1. You can use the property "NI_MinimumBufferSize" to control the number of times the data written to disk, for more details, you can refer to the help of TDMS Set Properties.
2. If you are using LV 2009 or later versions, there's a feature of TDMS called "one header only", that means, if you always write to the same channel, with the same number of data values, the size of TDMS file will be reduced.
3. If you are using LV 2010, we have a new menu palette "TDMS Advanced API" introduced, and using this Advanced API, it also can generate TDMS files with relatively smaller size.

[Eyeseeyou]

Thanks again for your help.

1. I tried to change the Buffer size. From what I understood, not wiring the channel and group inputs applies the change to every group and channel in the file, so I used that approach, but if there was a change it was to the worse, meaning even slower file creation than before.

2. I use LV 2010 SP 1 but can't seem to find that feature you're speaking of and I'm not sure if it would help in my case. I write to different channels during the execution, although I finish writing to one channel completely, before moving to the next.

3. The Problem seems to be the header being almost as big as the TDMS file it belongs to, so the non-advanced API is most probably not the cause.

I've already been told that the final product would never have to work with more than 30sec of logs so the ncl-files would be a lot smaller than some of those I have now but it still bother's me, that the execution takes like 15mins to finish.

[YongqingYe]

Hi Eyeseeyou,

1. I'm afraid this property is a little bit different to others, you have to wire both the group name and channel name and then the property can be applied.
2. If possible, you can write to one channel, complete it and move to the next, but please pay attention to the number of the data values when you writing to one channel, you'd better always write the same number of values.
3. OK, do you have a lot of properties in the channel? Or do you have a lot of channels, and the raw data is small?

[Eyeseeyou]

1. So I'd have to write a recursive VI, as I do neither know names nor number of the group names, prior to receiving the logfiles I need to convert.

2&3. I don't know how familiar you are, with CAN, LIN and FlexRay logfiles.

The program I write is designed to cope with any number of NCL-files with any amount of data written to them, increasing the number of iterations respectively.

For each file the main conversion routine calls a SubVI that processes the data corresponding to the logged BUS type. At first the raw data (refer to 'Display NCL Logfile as CAN Frames' example VI) is sorted from lowest ID to highest, each ID will be a group in the TDMS file created later. So the number of groups varies from file to file. Each ID, however, has a fixed amount of channels, 13 for CAN, 17 for FlexRay and 14 for LIN (I split the payload entry into 8 entries for each byte respectively)  BUT sometimes there is no data written to some of the entries.

[Eyeseeyou · Accepted solution]

Just to close this topic, I got the second problem solved and the first one omitted.