How Do I Convert LV7.1 Datalog Files to LV8 Format?

[Larry_Stanos]

I have several LabVIEW 7.1 datalog files that generate Error 71 (File data type conflict) when I try to open them in LabVIEW 8 using the new Open/Create/Replace Datalog vi with my original typedef constants wired to the record type terminal.  Likewise, my original LV7.1 code recompiled in LV8 often generates error 116 (unflatten or byte stream read operation failed due to corrupt, unexpected, or truncated data) or error 4 (end of file encountered) when randomly accessing records in the LV7 datalog files, presumably due to changes made to the datalog file structure in LV8.  I recall one time when a dialog box popped up asking if I would like to convert the file to LV8 format.  Since this was a file currently in use on the server by LV7.1 clients, I had to reply no.  With copies of the datalog files now on my hard drive ready for conversion, I haven't seen the conversion dialog box pop up again.

How do I convert the LV7.1 datalog files to LV8 format?

Thanks,

Larry

[Travis_M.]

Good afternoon Larry,

Thanks for posting!  The Datalog file support in LabVIEW has changed, but the underlying principles of the technology remains the same.  Converting to the new format should just involve converting a few VIs to the new Datalog-specific versions.  As you can see, the new Datalog VIs appear in their own sub palette in 8 which will help keep things simple.

Before we address the concerns with the errors, let’s first make sure that the code has been properly converted. 

To convert your VI to the new format:

1)      Replace the Open File VI from LabVIEW 7 with an “Open/Create/Replace Datalog” VI in LabVIEW 8.  Keep your type constant or control wired to the “Datalog Type” input.

2)      Replace any “Write File” VIs in LabVIEW 7 with “Write Datalog” and triple check that your data matches the datatype wired into the open exactly.

3)      Replace any “Seek” VIs with “Set Datalog Position”.  The input to this VI is the number of records to advance (check the mode set by the “from” parameter).

4)      Replace any Reading VIs with the corresponding “Read Datalog” VI.

I suspect that if you are receiving errors saying that the end of file has been encountered it is a result of a bad input to the “set Datalog position” VI.  I attached a basic example of writing a Datalog file in LabVIEW 8, and reading the results.  The rest of the functions in the new Datalog palette should be pretty straightforward. 

Does this help?  I would be more than happy to answer any other questions on this if you need any additional clarification, or if this response does not address your question.

[Larry_Stanos]

Travis,

Thanks for replying!  The problem has not been with the use of the new LabVIEW 8 datalog file vi's per se.  My problem is with trying to read version 7.1 datalog files containing variable-length records, for example, clusters containing variable-length arrays.  I've attached an example to illustrate the problem.

Create LV71 Datalog File.vi is a LabVIEW 7.1 vi that creates a datalog file containing records based on the typedef Record Definitiion.ctl.

LV71 Datalog File.dat is one such file created by the above vi.

Read LV71 Datalog File in LV8.vi is a LabVIEW 8.0 vi that uses the new datalog vi's to read a datalog file containing records of type Record Definition.ctl.  When this vi tries to open LV71 Datalog File.dat, error 71 (File data type conflict) is generated.

If Record Definition.ctl is a simpler (fixed length?) data type, records written by LV7.1 are read fine by LV8 without generating an error.

So my question again is: "How can I convert my LabVIEW 7.1 datalog files containing variable-length records into files that are compatible with LabVIEW 8?"  My files are large databases that must be ported with the next release of the client application.

Thanks!

Larry

[Travis_M.]

Hello Larry,

This is indeed an interesting problem -- sorry I believe that I misunderstood your question earlier.  However, I still can not reproduce the behavior you described.  I ran the LabVIEW 7.1 write Datalog file and created a new Datalog file, and used the reader in LabVIEW 8 without any problems!!  When I ran the reader on the dat file you sent sure enough -- CRASH.  The only difference I can see here is that I am running LabVIEW version 7.1.1f2 which is a patch for LabVIEW 7.  I could not find specific record of this being something that was fixed, but its worth the time to upgrade!  Additionally, I'll attach the data file that I generated with LabVIEW 7.1.1f2 -- please reaname to a .dat file and try to open it with your 8 reader and let me know if you get the error.

Have a great weekend!

http://digital.ni.com/softlib.nsf/websearch/6C5F342A4AF4EF8986256F5400693C3B?opendocument&node=132070_US
http://digital.ni.com/softlib.nsf/websearch/E0768D56ECF0BDB9862570BB005C4CD9?opendocument&node=132070_US

[Larry_Stanos]

Travis,

Thanks for the follow-up. It was very helpful as you'll see below.

I had no trouble opening your attached datalog file, so I tried to reproduce the error on my side again. I created five LV7.1 datalog files in the same manner as before, and this time had no trouble opening them up in LV8. They were all between 2578 - 2638 bytes in size - similar to your file of 2628 bytes. (Some variability expected due to use of random number generator.) I then discovered that I had two files named LV71 Datalog File.dat on my hard drive. The correct version could be read just fine, but I had mistakenly performed my read tests using the wrong version which had been created with a different record structure. This explains the type conflict error produced with the previously attached datalog file. My apologies for making this blunder.

So I'm back looking at my larger application which still generates errors intermittently. I'm seeing that even when re-reading the same record over and over, errors are inconsistent. Sometimes one of the two Read Datalog.vi instances generates an error (either 4=eof encountered or 116=unflatten or byte stream read operation failed due to corrupt, unexpected, or truncated data) and sometimes they don't. My plan is to try to distill a test program down to the essentials that still causes this behavior.

Larry

[Travis_M.]

Hello Larry,

I am glad to hear that we’ve resolved one of the issues here!  I’m not too sure what could be the problem here – one thing that really bugs me as an engineer nondeterministic failure.  I guess the big question here is not only why is test failing, but why does it fail only sometimes.  There are a couple things we could try: 1) create a small set of simple VIs which could replicate the behavior then we could send this off to R&D for investigation, 2) investigate and determine if there is a pattern to the seemingly random failures, 3) write a VI in LabVIEW 8 which reads the LV 7 datalog file one record at a time, and writes that to a new LV 8 datalog file one record at a time (making a copy basically), then determining first if the new LV 8 datalog file fails and second if there are any differences between the two (using a diff program or something similar).  Best case for us might be choice 1 because if there is a problem we could easily track it down then, but choice 3 is tempting if it works.  I’ve searched our database here of known issues like this, and the only one I can find relates to the writing of waveform data to files (the waveform datatype and not array data).

I hope some of these help – please let me know if further investigation sheds any more light on the situation!

[Larry_Stanos]

Travis,

You've outlined three excellent approaches.  I've already completed the first.  I extracted the essential features of my larger application and put them into a new version of the test vi called Read LV71 Datalog File in LV8.vi (included in attached zip file).  Surely the non-standard approach I've taken to opening the file is the culprit that summons a bug of some sort.  When I wrote several general-purpose datalog file routines in the past (e.g. Open Datalog File with Timeout), I had to use a variant record type terminal for defining the datalog type.  Then, to simplify subsequent file operations, I would typecast the returned refnum in the calling vi using a file refnum constant containing the same typedef record structure that had been wired to the variant record type terminal.  This worked reliably in version 7.1, but apparently has issues in version 8.0.

I have included two datalog files originally created by my larger LV7.1 application in the zip file together with the test vi so you can see the behavior for yourself.  I noticed that if I DON'T convert the record type to a variant before wiring it to Open/Create/Replace Datalog.vi, I receive error 71.

Regarding your suggestion #2, I have not been able to discern any patterns beyond the fact that when a file opens "properly", records can be read indefinitely without an error.  When a file opens "incorrectly", an error occurs almost always upon the very first attempted read operation.

Your third suggestion is a good one for exploring the problem further and I will likely try that next.  Upon successfully opening a database file "properly", I should be able to read each record and then write the records to a new file with a more conventional use of the datalog file vi's (i.e. without the use of variant record types and typecasting of file refnums).  This would render my general purpose vi's that rely on variant record types obsolete, however, so other workarounds would certainly be of interest.

Thanks for your continued attention and helpful advice.

Larry

[Travis_M.]

Hello Larry,

First, let me say that I cannot open the attached example because about a dozen typedef controls are missing from the control attached so I was unable to run the VI.  Second, I am pretty confused as to why you are typecasting the datalog file refrence from the create datalog file VIs to a datalog file constant refnum before reading the records.  If the type of the datalog refnum created with the constant is the same as the type of the input “Measurement File Record” what is the need to typecast at all?  Furthermore, if the type is different in anyway I would expect undefined results.  If we can method 3 described above and read all the records in your file in a desired format (say a cluster, or a variant) we might be able to find a better way to write these to a datalog file.  What are the characteristics of the datalog file you are looking for?

Keep me posted (so to speak)!

[Larry_Stanos]

Travis,

So sorry about the missing typedef controls.  I had meant to disconnect the ZVMS Measurement File Record subcontrols from their sub-typedefs before attaching but I forgot.  The new set of files reference just the one included typedef so you shouldn't have a problem running the vi's.

I also modified the main vi (Read LV71 Datalog File In LV8.vi) to better illustrate why I am typecasting the datalog file reference.  The code is commented with an explanation.  Because I wish to use several user library vi's such as the included "Open Datalog File with Timeout.vi" which is designed to open datalog files containing any type of record, datalog record types like "Measurement File Record" must be converted into a variant.  (Creating an extensive set of polymorphic instances for every record type seems wasteful and time-consuming but may be my only option in LV8 if record variants cause problems when opening datalog files.)  Let me know if you see a better way of implementing datalog-related sub-vi's that must receive record typing information.

Larry

[Travis_M.]

Hello Larry,

First, I believe that there are still missing controls that prevent me from running this example (see the attached screenshot).  Additionally, I believe that there might be some sort of corruption with this VI because any time I attempt to make a copy of the “Record Read From File” indicator I get a LabVIEW crash (in fact this happens any time I try to copy anything with the record type in it – the constant, the indicator, or the control)!  This might be related to the custom controls being absent, but I’m not exactly too sure on that. 

I believe that while complex, what you are trying to do should work.  I am not convinced however that there’s not an easier way.  First, to address the open datalog with timeout – you pass in the variant representation of the cluster control.  Since the datalog file is written with cluster-records and not variants I do not see how you can do anything other than open the file reliably (but your program seems to be able to ascertain the number of records from the file).  If your number of records in your datalog file is somehow incorrect it could throw off the rest of the program – interestingly enough the number of records is computed using the file size in an equation with the size of the record.  If the record type wired to your “open” file is different than the record of the file (even slightly) it could compute erroneous results.  It’s a little early (in the morning) for me to say for certain, but that is one possibility.  Another question about this VI, assuming that we would want to move the size detection outside of this VI, is why create a variant with a real type to begin with?  If all the VI is going to do is open the file depending on its availability, and the file’s refnum will be cast to a datalog file type of your choosing I think it’s a possibility that we wouldn’t need to send anything in but an empty variant constant.  I realize that this shouldn’t alter the operation of the program, but it will speed things up and reduce some complexity, as creating/copying variants is no quick task.

Finally, I would like to note that it appears that your control is made up of many smaller custom controls which may have been changed over time.  You have a constant of your type which is used with the typecast to declare to your program what type of datalog file is present.  I’m sure you have tested this, but you want to make EXTRA sure that everything in the constant matches exactly what LabVIEW expects in the file, and the indicator on your front panel.