Using nilibddc dll with Python ctypes

[Allen_Windhorn]

I am trying to use this "DiaDEM Connectivity Library" with Python to read files produced by our data acquisition system in the *.tdms format, but the dll examples are all for C and Matlab. Has anyone else done this who could give me advice? I've made some progress (see below) by trial and error, but it's slow and I need to get it up and running soon. I'm trying to duplicate the functions of the C example for reading a file (don't need to write yet).

I'm using Python 2.6 on a Windows XP machine.

Here's what I have so far:

------------------------------------
# Module to handle *.tdm and *.tdms National Instruments TDM
# format files
#
from ctypes import *

#Define short names for required dll functions
DDCOpen = windll.nilibddc.DDC_OpenFile
DDCNumGroups = windll.nilibddc.DDC_GetNumChannelGroups
DDCReadGroups = windll.nilibddc.DDC_GetChannelGroups
DDCGroupStrPropLen = windll.nilibddc.DDC_GetChannelGroupStringPropertyLength
DDCGroupProperty = windll.nilibddc.DDC_GetChannelGroupProperty

# File handles are pointers to 32 bit integer
czero = c_int(0)
FileHandle = pointer(czero)

# Retrieve a handle to the file
DDCErrCode = DDCOpen('Test_0919.tdms', 'TDMS', FileHandle)
print(DDCErrCode) # Should be zero
myfile = FileHandle.contents #The actual file handle

# Find number of groups
grpnum = c_int(0)
numptr = pointer(grpnum)
DDCErrCode = DDCNumGroups(myfile, numptr)
print(DDCErrCode)
# if DDCErrCode == 0: print(numptr.contents, grpnum)

# Read the groups [UNTESTED]
groups = c_int(0)*grpnum
grpptr = pointer(groups) # Needs to be a pointer to memory to hold grpnum
# of DDCChannelGrouphandle fields
DDCErrCode = DDCReadGroups(myfile, groups, grpnum)
print(DDCErrCode)
if DDCErrCode == 0: print(groups)

# Length of group name [UNTESTED]
namelen = c_int(0)
nlenptr = pointer(name)
DDCErrCode = DDCGroupStrPropLen(groups[0], 'name', namelen)
print(DDCErrCode)
if DDCErrCode == 0: print(namelen)

#Make a buffer to hold the name, then retrieve it [UNTESTED]
grpname = create_string_buffer(namelen+1)
nameptr = pointer(grpname)
DDCErrCode = DDCGroupProperty(groups[0], 'name', nameptr, namelen+1)
print(DDCErrCode)
if DDCErrCode == 0: print(grpname.value)

# Read the channels in the group and get the names [UNTESTED]

# Read the channel # of data values [UNTESTED]

# Read the channel data values [UNTESTED]

# Process the data values [UNTESTED]

# Write the processed data to a file in a different format [UNTESTED]
------------------------------------

The "UNTESTED" sections are more or less pseudocode representing things I need to do yet. The others work. These will eventually be functions in a module.

Cross-posted to a couple Python forums.  Thanks in advance for any help.

Regards,
Allen

[Brad_Turpin]

Hi Allan,

I wish you the best of luck with this, but I've never heard of someone getting the TDM(S) C++ API to work in a scripting language other than MatLab's m script, which has built-in DLL connectivity.  I wish I could help, but I've never done anything with python and have no idea what to suggest.

Have you considered using the (free) Excel TDM Add-In to load the data into Excel, then export to *.txt files?  It's unfortunately not automatable, and Excel does have limitations in the number of rows it will load, but it's at least a workaround to consider.  Of course the NI software solution for post-analysis and reporting of TDM(S) files is our DIAdem software, and you could also download a 30 day evaluation version.  You could then use the 30 day evaluation version of DIAdem to automate converting the TDM(S) files into something you can read with python.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Allen_Windhorn]

On 30 March 2009, Brad Turpin wrote:

> Have you considered using the (free) Excel TDM Add-In to load the data into Excel, then export to

>*.txt files?  It's unfortunately not automatable, and Excel does have limitations in the number of rows

>it will load, but it's at least a workaround to consider. 

A typical file has around 500,000 rows, so this is unfortunately not an option.

> Of course the NI software solution for post-analysis and reporting of TDM(S) files is our DIAdem

> software, and you could also download a 30 day evaluation version.  You could then use the 30 day

> evaluation version of DIAdem to automate converting the TDM(S) files into something you can read

> with python.

I have DIAdem but it's not the easiest program to work with as far as automating stuff, and besides, I'd like to have our customers be able to manipulate the files too.

I could use the C dll to write a program that just translates the relevant portion of the files to text, then process those.

Alternatively, I could write the decoding algorithm in Python (slow though) if I had a better definition of the tdms format.

Thanks for your help.

Regards,

Allen

[Herbert Engels]

Let me ask the other way round. If what you have is a generic DLL wrapper for Python, it should be able to handle the calls to the DDC library rather easily. It would just come down to translating the C code example into Python. Did you run into any problems doing that?

Herbert 

[Brad_Turpin]

Hi Allen,

Note that Excel 2005 and later supports 1 million rows, so a 500,000 row file would not be a problem for loading purposes, although Excel still limits their graphs to only 30,000 points.  Also, the Excel TDM Add-in supports windowing in part of a data set, say the first 60,000 values or the last 30,000 values.

I would not recommend trying to read the TDMS files with binary primitive file I/O functions.  We have a fair amount of documentation posted on the structure of TDMS files, but they have not been sufficient for customers to successfully decode existing TDMS files.  TDMS files were written for performance, not transparency, and we have provided the C++ API as really the only viable option to read the files in a 3rd party compiler.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Allen_Windhorn]

On 30 March, 2009, Herbert E wrote:

> Let me ask the other way round. If what you have is a generic DLL wrapper for Python,

>it should be able to handle the calls to the DDC library rather easily. It would just come

>down to translating the C code example into Python. Did you run into any problems

>doing that?

 You've hit the nail on the head -- that's just what I've been trying to do.  My problems have mainly been with types.  I figured out by trial and error that the DDCFile type is a 32-bit C integer and DDCFileHandle is a pointer to that type, which I can duplicate in Python using ctypes.  The property variable is a pointer to a C character array that has memory allocated and can be written to, which ctypes knows how to do.  The next challange is the DDCChannelGroupHandle (another integer pointer, I think), and the variable groups which is defined as a pointer to DDCChannelGroupHandle type.  DDCChannelGroupHandle in turn is defined in the header as a pointer to DDCChannelGroup, which is defined as a struct in a manner I don't understand, with the word _DDCChannel which is nowhere defined.  I need to duplicate the variable groups in Python somehow.  ctypes lets me define simple C types, but I don't know how to make a struct or array (question for the Python newsgroup), and I don't know what structure to make (question for this forum).  I had a class in C++ once but I've never used it in my work, so I'm rather hazy about typedefs.

Thanks for your help.

Regards,

Allen

[Herbert Engels]

The C example doesn't declare the struct types for DDCFile, DDCChannelGroup or DDCChannel, because all access to contents of the underlying classes goes through API functions. You don't need direct access to the actual class instance, all you need is something that references it when you call an API function.

You should be fine if your Python script declares DDCFileHandle, DDCChannelGroupHandle and DDCChannelHandle as pointer-sized integers.

Hope that helps,
Herbert

[T.Gilbert]

Hi Allen.

Sorry for re-opening the old thread. I junst wanted to know if you were able to proceed in this issue. I am also trying to write a python script that is processing some tdms files.

Thank you very much in advance for answering.

Best regards

Tobias

[Kyle_M]

Tobias,

You may want to send Allen a PM if you would like to get an update on his project in case he does not frequent this forum super often. I believe it sends them an email to notify them. 

Regards,

Kyle Mozdzyn

Applications Engineering

National Instruments

[longwalker · Accepted solution]

May want to explore:

http://pypi.python.org/pypi/cTDMS/0.9

or alternative link

http://code.google.com/p/ctdms/

they appear to be exactly what you are looking for.... (couple years later)