How to organize variables for file saving and scalability?

[Aaron T.]

Hello,

I have created several CVI applications that store production data for numerous machines.  To organize the data for file saving I have implemented structures.  This has worked well with one limitation, the inability to scale the structure at a later date without invalidating existing files.  I would like to consider alternative approaches that would allow scalability.

Here's an example of my current method...

// Definition of structure per machine.

struct machine_1

{

   int  int_param_1, int_param_2, int_param_3;

   double  dbl_param_1, dbl_param2, dbl_param3;

}

struct machine_2

{

   int int_param_1, int_param_2, int_param_3;

   double dbl_param_1, dbl_param2, dbl_param3;

}

// Definition of inclusive structure. (Member name and structure tag name are the same.)

struct

{

   struct machine_1   machine_1;

   struct machine_2   machine_2;

} machine_parameters;

To assign a value to a structure variable:

// Assign value.

machine_parameters.machine_1. dbl_param_2 = 77.47;

Then when it comes time to save the populated structures:

// Save structure.

error = fwrite (&machine_parameters, sizeof(machine_parameters), 1, dest);

The problem comes later when multiple files already exist and one of the machine structures needs an additional variable added.  For example, if I need to add int_param_4 to the machine_1 structure.  Adding this variable will invalidate the previously saved files because they were saved with a different structure and will not be able to be opened with a new structure containing one additional variable due to the structure definition mismatch.

I have added spare variables per data type to the structures for each machine, but it's a losing game.  If I add 10 spare variables, I end up needing to store 11 more pieces of data.

Is there a better approach?

Thanks,

Aaron T.

[Raj_V]

Hi Aaron,

Unfortunately there's no better way than to know how many elements you would be needing ahead of time in the design phase. This is just a limitation of the language, unless you want to create a copy of the struct with added elements. Since this is not objected oriented programming, you don't have the flexibility to inherit from a parent class to have a number of child classes with varying properties. I hope this answers your question.

[menchar]

One simple way is to output the data as ASCII comma separated values, with a newline character at the end of each row of data.

I.E., the only structure to your file data is a "row" of CSV's, with the file containing some number of rows.

Then, you load the data into Excel, and it will parse the CSV's for you and when it sees the newline, put the next set of CSV's on the next row of the worksheet.

If you ever need to expand the number of items in a row, you just add them as you generate data, pushing the newline to the right, the extra data  extending the row.

So you get an Excel worksheet filled with rows of (possibly varying length) data.   So long as you add data at the end of the row when you redefine what you're saving, anything reading the file should see the same stuff that was always there.

You can write a macro to reformat or parse the CSV's once they're in the spreadsheet.  With Excel 2007 supporting very large worksheets, you can put a lot of data into one.  I think they expose a C interface for writing fast data manipulation of cell data now too - sort of a fast macro from the Excel viewpoint.  I think the number of columns is 16384 and 1 million rows in a "Big Grid". The Excel 2007 engine is multi-threaded and you can tell it how many cores to use on a multicore machine.

So the only problem I see is the loss of local structure (your C structs get serialized and get concatenated to one another) but you could re-introduce the structure with a macro.

Or, if you were to write out serialized binary values and then view the file data using a hex editor like Neo, you can tell Neo what your C structs were and it will pick up binary file data and put it back into the C structs for viewing.

Or use MatLab to read the CSV's and reformat it.

Or use the CVI SQL interface and write it out as database records.  I think the SQL toolkit costs extra, maybe it comes with the FDS. 

Menchar