Appending tdms files problem

[Saloutios]

Hello all,

I've developed a VI that logs data from a 6 hours test. Sampling rate is 10k resulting to 3,600 TDMS files each of them saving 6 columns (AI channels) of 60,000 rows. These are raw data that are going to be post processed calculating i.e rms values, FFTs and so on. 

I am in a stage where I should take a decision of whether I should use DIAdem for post processing or develop my own LabView code. By far I would prefer DIAdem as I am not very experienced Labview programmer. My questions are:

1) Drag&dropping 3,600 tdms files to the Data Portal results as expected memory overflow. Is there anyway of appending all these massive data for processing? I.e being able to have a chart of 6 hours and the corresponding rms values?

2) Is there any way of (if it is required) compressing my samples? I.e get an average of rms values every 10 or 50 cycles?

I have been through most of the webcasts on DIAdem and I really wanna take a swift decision of how I should orient my post processing work and your help and suggestions are most welcomed.

Thank you

Panos 

[Brad_Turpin]

Hey Panos,

Wow, small world, I happen to be working on exactly this scenario right now for two other projects.  DIAdem can absolutely efficiently concatenate a span of those segmented data files for detailed viewing/analysis, though that requires a VBScript (which I have).  But in most cases it is preferable to calculate wrap up summary properties for each segmented data file and query those property columns with the DataFinder, loading the segmented properties instead of the segmented data.  This is a great use case for DIAdem, and I can offer you lots of advice and lots of pre-built VBScripts that integrate into the DIAdem GUI as menus.

Why don't you send me a few of your segmented data files and we can chat details-- brad.turpin@ni.com

Brad Turpin

DIadem Product Support Engineer
National Instruments

[Saloutios]

---

Brad Turpin wrote:

Hey Panos,

Wow, small world, I happen to be working on exactly this scenario right now for two other projects.  DIAdem can absolutely efficiently concatenate a span of those segmented data files for detailed viewing/analysis, though that requires a VBScript (which I have).  But in most cases it is preferable to calculate wrap up summary properties for each segmented data file and query those property columns with the DataFinder, loading the segmented properties instead of the segmented data.  This is a great use case for DIAdem, and I can offer you lots of advice and lots of pre-built VBScripts that integrate into the DIAdem GUI as menus.

Why don't you send me a few of your segmented data files and we can chat details-- brad.turpin@ni.com

Brad Turpin

DIadem Product Support Engineer
National Instruments

---

Hello Brad,

Thank you for the reply! Sure thing, that would be great! Gonna send you a batch of these files.

Regards,

Panos

[YongqingYe]

I have a VI which merges 2 TDMS files into a new TDMS file. The VI uses binary file I/O operations. I attached the VI here for your reference.

[Saloutios]

---

YongqingYe wrote:
I have a VI which merges 2 TDMS files into a new TDMS file. The VI uses binary file I/O operations. I attached the VI here for your reference.

---

Thank you YongqingYe,

Unfortunately I cannot open your VI coz Labview 9 cannot open it. And the main problem is that i can merge TDMS files but I am talking about mass data, a total of almost 9.5GBs in 3,600 files. That causes severe memory problems. 

[Herbert Engels]

You can concatenate large TDMS files using the Windows command prompt (you can choose to either delete the TDMS_INDEX files or concatenate them in the same order):

copy /b "C:\temp\0.tdms" + "C:\temp\1.tdms" + "C:\temp\2.tdms" "c:\temp\merged.tdms"

Here's an example of how you could do this in LabVIEW:

I ran this on Windows 7 the other day, it seems that there is a security issue with launching a new process from a running process (requires additional user privileges). That might give you an error 267 from the operating system. I don't know yet how to fix this. Other Windows versions should be fine.

Hope that helps,

Herbert

[Saloutios]

Hello Herbert, 

thank you very much for the reply. I will check this VI even though it seems that probably I will need to follow the opposite direction meaning breaking down my files to smaller ones coz I need to perform calculations on a 1sec time window. Brad above is already is helping me out on this.

Thanks for the help!

Cheers

[Saloutios]

Hi there Herbert,

just wanted to let you know, I ran the VI on a 32-bit Vista system and gave me a 267 error too.

Cheers,

Panos

[Herbert Engels]

As long as you don't exceed the max length for command line strings, this should work.

Herbert

[TCPlomp]

Herbert,

does this mean you can stitch TDMS files together?

Like this:

Ton