Re : Re: Re : Re: tdms vs cluster where when speed doesn't matter

[tinnitus]

hello,

The question is in the title: in case speed doesn't matter wich could be advantage using one or other of methode.

I 'm working on calibration bench at low speed ( T° of water , humidity, barometer) so hdd access is not a parameter.

i have to store , author, date  ~20 data information like that an then test data ( ~10 * 2Darray(~10*10) ).

I'm new to this kind of programming . I think both are possibles , but i would like to start knowing what can be the 

best before involving to much in one method before realizing the other was better adapted.

What are avantage disavantages like  easy of coding,  scalability,  ...

Best regards

Tinnitus

[Christian Mergl]

First of all TDMS vs. Cluster is not comparable, TDMS is File Format or a API for file I/O and a cluster is a specific datatype.

So you still didn't specify which method you want to use for writing this cluster to the harddrive. I assume you are talking about the Datalog API, so binary.

Well if you dont have any constraints in speed or data evaluation (like using NI DIAdem) then its more or less independet of what you use in terms of File I/O.

However from the programming architectural view it might be better to use a cluster where you have all your data collected once. So you can create a typdef

which makes future changes easier then adding atrributes to TDMS.

So you should first think about how you organize your data within the VI, and if your best choice would be a cluster, you can directly use it for the datalog API.

Just wanted to give some hints! 

Christian

[waldemar.hersacher]

I additon to Christian I would take another thing into account:

If you write a cluster with the datalog API you can only read this file with the same cluster. After changing the cluster by adding new elements you cannot read in the datalog files containing the old cluster data.

Using a LV class instead will allow you to change the class by adding new elements. Reading the old data files will read in the existing data and fill all elements. New added elements will contain their default value. Data for deleted elements will just be ignored.

Another simple way to store clusters is to use the XML functions. You need just "Flatten to XML" and "Write XML to File.vi". This method has also the disadvantage that changing the cluster will not read in the old data.

I have added a project I used to measure the speed of using config files, XML and LV classes. Do not try TestConfig.vi it refers to some private subVIs and they are not designed for general datatypes. TestFile and TestXML will work. Project is in LV 8.5.

[tinnitus]

Hello,

Tank you both for comments

The probleme is that demands of customers can evoluate and himself seems not to know exactly what are their needs.  

It could be that in future months they will need to store data in data base or having possibility to transfert results file with info (builder, operator,date, coeff calibration at a specific  date...) into differente table  and maybe need .xls outpouts.

Thank you waldemar for LV class, for the moment it's a little fuzzy for me but concept seems nice,i hope obtaining time to train myself or having training on this kind of programmation.

Best regards

Tinnitus

[DFGray]

For scalability and future expansion, I would strongly recommend you save your data as discrete units, not clusters.  Be sure to include a file format version number.  This will allow you to change the format in the future and still read old files.  More importantly, it will allow old code to fail gracefully (if necessary) when attempting to read newer, different format, files.  Make sure you take this into account in your design.  Some options in LabVIEW:

1.  TDMS - would be perfect for you, except it does not support 2D arrays.  In your case, the workaround (storing multiple 1D arrays) is not very onerous.  This is probably your first choice.  It can be easily converted to ASCII and/or read by Excel if needed.
2. Config File - this API set would also work fairly well for you.  It is text based, so be careful of your formatting for any numbers you save.  It is easy to lose resolution and get into problems.  Once, again, arrays are not natively handled.  You can work around this easily enough by converting the arrays to strings before writing.  Again, you will probably want to create a key for each line of the array, since extra CR/LFs will cause problems when reading.
3. HDF5 - this is a far more difficult method, but avoids the issues of the previous two.  You can get the API here (a Google search will give you more LabVIEW APIs, this one is rather old). HDF5 is difficult to learn but highly flexible and can be read by most commercial analysis packages.

Good luck.  Let us know if you need more information.

[tinnitus]

 Hello,

a little resurrection

I habe been working on other part of project, i'm actually returning on this aspect of saving data file.

DFGray,

Thank you for your notification about including version number. 

 
about 1) you said its easy to convert Tdms file to ascii is there something like openG doing this kind of thing ?
    
about 2) could it be an idea to work within and through vi(s) with cluster then recording it with openg 'cluster to ini" file
or there is another problem with this methode i don't see.

for resolution problem,for the moment it goes well for 1 digit, what have i to do if i need more precision
 splitting data before saving ?

about 3) thank you for this link i did not kown iwill take a look but i think i 'll reserve this for the next project...

  Best regards

Tinnitus

[DFGray]

There are a couple of ways to convert TDMS to ASCII.

1. Using the Excel plug-in, read the TDMS file into Excel, the save as ASCII.
2. Using LabVIEW, read the data and then export as ASCII or LVM.

The first method is pretty easy.  The second gives you more control.

I don't see a problem with using the OpenG cluster to ini, but have never used it.  So, I cannot recommend it one way or another.  Make sure it gives you some way to account for file storage versions before using it.

Resolution is only a real issue if you are storing ASCII.  That is why I recommend a binary method, if you can.  I have seen too many people get burned by resolution issues.  LVM does not give you control over the number of digits stored, but the INI file API does (inputs on the bottom of the VIs).  I do not know if  the OpenG VIs export this functionality.

[tinnitus]

Thank you for your answer,

it seems that openg ini doesn't managed digits option,  could some one tell me if mine are not up to date or not ?

Best regards

Tinnitus

[James_R]

I used the Openg ini VIs in the past but found problems with them recently concerning paths and other datatypes.  So I switched to the native LabVIEW flatten to XML VIs.  These are nice because you can use either the cluster to unflatten them or use the XML DOM parser.

[DFGray]

Unfortunately, the native LabVIEW flatten to XML does not give you control over the resolution of save strings.  Older versions saved six digits.  Newer versions save the full resolution of your data type.  Be sure to check before you use it.