Binary file of unknown formats

[Roberto83]

Thanks again for your assistance Altenbach. It is great to be able to  see the values in the file now. I had been struggling with if for about three days.

The file now provides an output. At first glance, the numbers displayed seemed too large. Also, I was wondering what happened to the Echo information (or maybe it was the Velocity information that was missing, at this point I didn’t know). I tried changing the representation and see what kind of output I got. Besides The I32 representation in the output of the array I also tried I16 and I8. I was surprised to see that the I16 representation in deed provided some of the numbers of the text file.

I used Profiles 0 and 1 of the text file to compare against the values in the output array. Then, I divided the output of the I32 representation by powers of 2 (Thanks for the tip). By dividing the output by 2¹⁶ I could get more numbers that were present in the text file. Well, it turns out that half of the velocity values are obtained by using I16 representation and the other half is obtained by using the I32 representation (and dividing by 2¹⁶). The same is true for the Echo values, half of them are represented as I32 and the other half as I16.

I thought of adding another read binary file function and then use a I16 representation to display the values that have that format. It does not work, I get a message that says: “Error 4 possible reason: end of file encountered”. I am still missing half of the information so I don’t know why I get such a message.

I was wondering if there is another way to split the numbers that have I32 format and those which have the I16 format. I believe that in this way if those numbers are displayed, then I can figure out a way to manipulate them and separate the velocity from the echo values.

In case my explanation lines above is confusing I added an excel sheet with the values of the velocity and echo of profiles 0 and 1. The values obtained by using representations I32, I16 and I8 are also included.

Thanks in advance,

Roberto

[altenbach]

---

@Roberto83 wrote:

Also, I was wondering what happened to the Echo information (or maybe it was the Velocity information that was missing, at this point I didn’t know).

---

Remember, there is also binary data in the header, starting from about byte position 23..104 in the file. Most likely it contains the distance information.

---

@Roberto83 wrote:

Besides The I32 representation in the output of the array I also tried I16 and I8. I was surprised to see that the I16 representation in deed provided some of the numbers of the text file.

I used Profiles 0 and 1 of the text file to compare against the values in the output array. Then, I divided the output of the I32 representation by powers of 2 (Thanks for the tip). By dividing the output by 2¹⁶ I could get more numbers that were present in the text file.

---

You would never want to use a plain division for integer stuff like that. You can use the "scale by power of two" which operates directly on integers using bit shifts. Well, it seems that maybe each row contains a mixed set of representations. The first four columns seems I32, but maybe the 65 data columns need to be split further. Currently, each values is an I32. Instead of dividing by 2^16, you should split the Integer into two 16 bit of four 8 bit numbers. Maybe each value needs to be further scaled e.g. by e g the distance or distance*cos value for each sensor? There is an infinite number of possibilities!

---

@Roberto83 wrote:

I thought of adding another read binary file function and then use a I16 representation to display the values that have that format. It does not work, I get a message that says: “Error 4 possible reason: end of file encountered”. I am still missing half of the information so I don’t know why I get such a message.

---

Once you read most of the file as I32, and you try to read more as I16, you'll run out of file unless you reset the file position to whatever it was before you read the I32. Still this is not the way to do things. You already have all the data, so there is no need to read it again. All the type manupulations can be done in memory.

Silce out colums 4..68 into a new 2D array, then you can use e.g. "split numbers" or "typecast" to further subdivide the integers.

[Roberto83]

Silce out colums 4..68 into a new 2D array, then you can use e.g. "split numbers" or "typecast" to further subdivide the integers.

Thanks a lot Altenbach. I did slice out columns 4 through 68 into two new 4096 by 65 arrays (I used the “split number” function. The array that shows the I16 numbers displays both positive and negative numbers correctly; however, the array that displays the I32 numbers displays only the positive numbers correctly. The negative numbers are displayed as number of value ~65000.

I tried changing the representation of the array to U32 but it did not work (I did so because the help file of the “split number” function mentions that this functions take “8-bit, 16-bit, or 32-bit unsigned integers”.

I also tried changing the representation of the data type in the read binary file function to U32 but it didn’t work either. Do you have any idea how I can display the negative values of the I32 array correctly?

Remember, there is also binary data in the header, starting from about byte position 23..104 in the file. Most likely it contains the distance information

You are right; the distance information and the velocity resolution are still missing, most probably they are in the header. The good thing is that I have enough information in the footer to calculate both parameters. For now I will try to calculate them if I can not extract that infomation from the header.

 

[Roberto83]

I forgot to add the file, sorry

[altenbach]

I am not sure why you are reshaping the same array twice. Once should be enough ;).

When you split a 32 bit integer, both halves are 16bit (U16), so don't leave one indicator (binary data 2) at I32 or you'll get the wrong results.

[Roberto83]

Thanks again Altenbach. I got it this time. I left the upper array to compare the differences in the output when I changed the representation in binary data 2. But now that I feel comfortable with the results I delete binary data 1. Reading the binary data has been for me the most difficult part of this project. Again, I really appretiate all your guidance and support you have provided me.

Sincerely,

Roberto