How do I neatly replace my excel sheet logging with database capability? + general questions about distributed computing with LabVIEW

[Bob_Schor · Accepted solution]

I'm embarrassed to admit I've never used TDMS, but if I had to guess, I would guess that for your application, many LabVIEW Gurus would suggest TDMS as an output format rather than getting involved with databases.

Also, you are correct that Excel, if given a "delimited" text file with a tab delimiter that has the "wrong" (meaning "not according to the expected Excel proprietary format) extension of .xls, will be able to open it and treat it as a tab-delimited file.  To me, this is a little bit of a perversion, but then I'm a stickler for clarity (and an extension having the "expected" meaning).

BS

[ijustlovemath]

TDMS sounds interesting. I'll have a look at it. By the way, one of you asked for CPU + RAM utilization while the application runs:

50-70% of my 2012 i7 (this is inside a windows 8.1VM) with ~200MB of RAM. It's definitely beefy.

Another shot of just after execution completes:

[altenbach]

OK, you are clearly using slightly more than one CPU core (your processor is only a dual core, optimized for low power laptop use.). Using full parallelization would not even double the performance so that might not be worth it unless you can get a better CPU first.

[ijustlovemath]

As I said, this is currently being run within a VM, so performance is limited. Simulations hopefully would be distributed among multiple computers, as outlined in the original post. TDMS looks like a great format, though it still seems like I would have to make one TDMS file for each simulation. I suppose that can't be helped!

[Dennis_Knutson]

With multiple computers, it might be simpler to write to a common database. The performance is going to be better than what you have now but not as fast as binary/tdms. Something you can easily experiment with.

[ijustlovemath]

For my use case, which is to analyze data from all 8k simulations, which do you think would be better?

[Bob_Schor]

Why are you asking our opinions about you processing your data?  We've suggested several methods that can work, given some plusses and minuses.  It is going to come down to your trying one or the other (or all) methods, seeing which works, seeing which is "too much trouble to program/understand", seeing which "cannot cope with the data".

From what I know about TDMS, I have no doubt that it is up to the task of handling the amount of data you contemplate.  Whether or how it would handle data from multiple machines I can't say, as I've not worked with it, but I would venture to guess that if you coded it up, you could make it work.

Binary would probably be the fastest, but I think maintaining the "meta-information" (format, etc.) would be a royal pain.

I know even less about linking LabVIEW with a "foreign" database -- it would surprise me if that was a simpler (from a program development standpoint) solution than TDMS.

But in the end, it's your data, and it needs to be your effort that gets the job done (unless, of course, you are purchasing the services of a consultant to do the work for you).  Spend a day or two (or more) trying one of these methods and see how it goes -- you'll make much faster progress than spending days asking our opinions about which method is "best".

Bob Schor

[Hooovahh]

TDMS Guru chiming in.  Maybe I tend to recommend TDMS more often than I should but I know many of the benefits of it and try to use that as justification.  

A few things TDMS does well is fast writing.  It just throws it down very quickly, and streaming tons of data can be done very fast.  The amount of data is almost never the limitation, but instead the hard drive or network speed.

Another neat thing is that multiple TDMS files can be combined without having to use any of the TDMS functions.  If you have two 1GB TDMS files, combining them takes as much processing power as it would to just copy the file.  No need to open, read, interpret, write, and close for each file.  You can even do it using the command line.  Just run a command line like this:

copy /b "C:\Test File 1.tdms" + "C:\Test File 2.tdms" "C:\Output.TDMS"

So if you have multiple stations, all logging to their own TDMS files, in separate groups, or channels, you can copy them all back to one machine and combine them quickly.

[Bob_Schor]

Aha, a TDMS Guru responds!  And (trust me) you can trust Hooovahh's expertise -- after all, his name refers to the vacuum you can see in the MRI he uses as his avatar (I'm seriously kidding -- I certainly have learned a lot from his posts ...).

BS

[Yamaeda]

From a quick look it seems like it could be adapted to a database fairly easily. Just add a GUID-field and add a generator to the code and all tests should be able to write to the same DB. The GUID is just to a unique ID to all tests so you don't manage to mess data up.

I suspect you can set your virtual computer to use more cores, then Altenbachs performance suggestions might be more useful.

(as a style note, i prefer to have string array constants instead of string constants building arrays, but it does no difference in the end)

/Y