where is Read from I16 File.vi ?

[CodeMunkee]

Hi,

At this link, http://forums.ni.com/ni/board/message?board.id=170&message.id=40715&query.id=1064265#M40715, I found a very helpful vi called ReadBigBinFile that uses two vi's that I cannot find when I search my LabView Help Manual.  The vi's are called Read from I16 File.vi and Write to I16 File.vi.  I incorporated the code I found and it works effectively, although not "precisely" for my needs.  When I click on the Detailed Help link in the Context Help, my search retrieves no additional information and says it cannot display the webpage.  I use LV 8.5.

Attached is my code.  My goal is to split a large binary file into smaller pieces at specific locations so that I may process the smaller sub-files in Matlab. My binary file contains records that are 25 bytes long preceded with an ASCII header that is less than or equal to 1024 bytes long.  I would like to be able to split the file precisely at the end of a 25-byte record.  Right now, I can split the file after a certain number of "points" (the original code I adapted was setup and worded this way), but I'm not sure how these "points" correspond to bytes.  Thus, the search for more info on those vi's that led me to a dead-end.  

Will someone please tell me where I can find the info on Read/Write from/to I16 File?  I'm unclear on the definitions and how they relate to my binary file and how I can precisely define where to correctly split my binary file.  Until then, I'll keep hunting.

Thanks,

~Tristan

p.s. I tried to attach a sample data file, but the website denied me due to invalid file extension (*.nim, *.bin)

[Darin.K]

On my system it is located in vi.lib/Utility/file.llb

[smercurio_fc]

That thread is from 2002 and the example is for LabVIEW 6. Those VIs are still available only for backwards compatibility so as not to break old code. They should not be used for new code. It also doesn't make any sense to read your file as a series of I16 numbers since your file format isn't a series of I16 numbers in the first place.

The VIs themselves are just wrappers around the regular file I/O functions. I would not recommend using it in the manner shown since it opens/closes the file in each iteration of the loop, and this is silly. The Read Binary File and Write Binary File functions automatically pick up from where they left off if you wire a file reference to them instead of a path. Thus, you just need to use the Open/Create/Replace File function for the input and output files before the loop and feed in the references inside the loop. Close the files after the loop is done.

As for the split operation, is the header present before each 25-byte record? You said that the header is less than or equal to 1024? What determines this, and how are you supposed to know if you just look at the file in a hex editor?

As for the code, I don't understand what you're trying to do. Is the bottom part supposed to be a replacement for the top, or is that supposed to run in parallel? If the latter, I understand it even less. 

[CodeMunkee]

Ah, thank you, I thought that might be the case - that the I16 stuff was obsolete.  I agree about the silliness of opening and closing the original large binary file multiple times.  That logic had me confused as well, but it worked.  I believe my data is in the form U8 instead of I16, and I admit I tried to shortcut actually thinking about the problem by grabbing some previous code.  Thanks for explaining that these old vi's are simply wrappers around the File I/O, as that also had me confused.  I will rewire the loop correctly.

 Addressing the rest of your questions might be easier with this explanation:

When deployed in the field, this instrument records/writes 25 bytes of binary data per second - usually for 1-6 months.  When the data run begins, an ASCII header of metadata is attached to the beginning of the file, sort of like a column header.  Unfortunately, the creator of these instruments allowed for a nebulous amount of metadata to be added, to a maximum of 1024 bytes.  As our field technicians write the metadata somewhere miles from me when deploying the instruments, I do not know how long the header is until I open the file upon its return.  Sometimes, there are reasons (more than simply the file size) to split our data files at certain points to make data processing easier.  What I am attempting to do is write a vi that allows the user to open a specified file, view the ASCII metadata header, see the size of the ASCII metadata header, separate it from the binary data, and then define where to split the binary data, or into how many files to split the original binary data file.  As the records are 25 bytes in length, I'd like to be able to split the file inbetween the end/beginning of data records.  So the code attached previously is incomplete work, and I apologize for the confusion as well as appreciate your thoroughness in following the code's logic (which is a little unsound at the moment!).  I'll remember to not post incomplete work again to avoid frustrating you and possibly wasting your time.

So to answer your last question: no, they are not supposed to run in parallel but I haven't figured out how to optimize the user interface because I haven't actually figured out how to properly define where to split the files so that no data records get mangled.  Do you have any suggestions?

Currently, I am struggling with another (extremely pressing) problem: I have attached modems to my instrument in the field and I am using LabView to access the streaming data via the Internet.  I cannot identify what is going wrong with the telnet sessions and think it might be in the way I am writing the software, but I will post another thread for this issue containing details.

Thanks,

~Tristan

p.s. attached is an example of one of the data files (binary data columns are translated into ASCII, but you'll see the header)

[smercurio_fc]

OK, so you have a single header at the beginning of the file of unknown size. But you said " do not know how long the header is until I open the file upon its return. ". How, exactly do you know? 

The text file you attached doesn't really make things clearer as far as your file structure. Is the header the stuff up to (and including) the text "High Gain"? How do the numbers in the columnss relate to 25 bytes of data? Is each row supposed to correspond to this 25-byte record? 25 is not a very even number, so I don't see how you're getting all those numbers into 25 bytes. 

If I take things based on dealing with a file that contains a header of some size then you just need to read out a certain number of records at a time. Well, this is easily done with a for-loop. You can easily calculate the number of iterations required based on the remaining number of bytes after the header, the number of records at a time that you want to read, and the record size (25). See attached examples as a starting point.

[CodeMunkee]

OK, so you have a single header at the beginning of the file of unknown size. But you said " do not know how long the header is until I open the file upon its return. ". How, exactly do you know? 

I know because of the number of ASCII characters I read from the header file ("Header Chars" on the Front Panel).

The text file you attached doesn't really make things clearer as far as your file structure. Is the header the stuff up to (and including) the text "High Gain"? How do the numbers in the columnss relate to 25 bytes of data?

Yes, "high gain" is the last sentence of the header file.  The columns that follow the header are usually a splurge of information in binary format.  I cannot attach a binary file to your website to show you, so I gave you a textfile where the science data had been broken into columns, after being translated from binary to ASCII.

Is each row supposed to correspond to this 25-byte record?

Each row represents the science data that has been pulled from a 25 byte record, yes.

25 is not a very even number, so I don't see how you're getting all those numbers into 25 bytes. 

Well, I agree, 25 isn't a very even number.  Can't explain the logic, it was created in the 80's by mad genius scientists.  I just get to deal with it.  There are much more data squeezed into the 25 bytes of binary data than the numbers you see in that textfile - it's big endian little endian stuff combined with some mathematical post-processing voodoo.

If I take things based on dealing with a file that contains a header of some size then you just need to read out a certain number of records at a time. Well, this is easily done with a for-loop. You can easily calculate the number of iterations required based on the remaining number of bytes after the header, the number of records at a time that you want to read, and the record size (25). See attached examples as a starting point.

Thanks!  I'll take a look at the vi.

[smercurio_fc]

---

CodeMunkee wrote:

OK, so you have a single header at the beginning of the file of unknown size. But you said " do not know how long the header is until I open the file upon its return. ". How, exactly do you know? 

I know because of the number of ASCII characters I read from the header file ("Header Chars" on the Front Panel).

---

This to me implies that you are looking at the byte value and seeing if the value is within a certain range. If so, you are considering this to be an ASCII value and thus part of the header. Is this correct? 

---

Well, I agree, 25 isn't a very even number.  Can't explain the logic, it was created in the 80's by mad genius scientists.  I just get to deal with it.  There are much more data squeezed into the 25 bytes of binary data than the numbers you see in that textfile - it's big endian little endian stuff combined with some mathematical post-processing voodoo.

---

Dumb question: why don't you just change the file format to something that makes sense to sane people?

Message Edited by smercurio_fc on 10-22-2009 09:56 AM

[CodeMunkee]

---

CodeMunkee wrote:

OK, so you have a single header at the beginning of the file of unknown size. But you said " do not know how long the header is until I open the file upon its return. ". How, exactly do you know? 

I know because of the number of ASCII characters I read from the header file ("Header Chars" on the Front Panel).

---

This to me implies that you are looking at the byte value and seeing if the value is within a certain range. If so, you are considering this to be an ASCII value and thus part of the header. Is this correct? 

I use the number of the offset for the first character that begins the binary feed (thanks to small miracles, it's a constant value of hex01) as an indication of how many ASCII characters are in the header preceding this offset.  One byte is one ASCII char, therefore the offset of the beginning of the binary file equals the number of characters in the header file.

---

Well, I agree, 25 isn't a very even number.  Can't explain the logic, it was created in the 80's by mad genius scientists.  I just get to deal with it.  There are much more data squeezed into the 25 bytes of binary data than the numbers you see in that textfile - it's big endian little endian stuff combined with some mathematical post-processing voodoo.

---

Dumb question: why don't you just change the file format to something that makes sense to sane people?

You feel my pain.  Changing the file format involves (1) getting the mad genius to, in his mind, recreate the wheel because it's difficult to retrofit, or (2) building an entirely new system from nuts and bolts and flux capacitors.  As I have the power to do neither, I sit in a tree, eat bananas and try to outsmart him using LabView.

[smercurio_fc]

---

CodeMunkee wrote:

---

CodeMunkee wrote:

OK, so you have a single header at the beginning of the file of unknown size. But you said " do not know how long the header is until I open the file upon its return. ". How, exactly do you know? 

I know because of the number of ASCII characters I read from the header file ("Header Chars" on the Front Panel).

---

This to me implies that you are looking at the byte value and seeing if the value is within a certain range. If so, you are considering this to be an ASCII value and thus part of the header. Is this correct? 

I use the number of the offset for the first character that begins the binary feed (thanks to small miracles, it's a constant value of hex01) as an indication of how many ASCII characters are in the header preceding this offset.  One byte is one ASCII char, therefore the offset of the beginning of the binary file equals the number of characters in the header file.

---

OK. In that case for the example I had provided you would place a loop around that first Read From Binary File so that you keep reading one byte at a time until you see 0x01. The rest of the code is written to read the records. 

[smercurio_fc]

Actually, scratch that last suggestion. That's a bad way to do it since you'd end up performing a file I/O operation up to 1024 times. A better way is to read 1025 bytes since you said the header is a max of 1024 bytes. Then search the array for that magic 0x01. This will tell you how many bytes the header is. Then you can set the file pointer to the location right where the records begin, and let the rest of the code read the records as is.