Help parsing .csv hex values that are not byte aligned

[gEEkPockets]

I need to parse a csv file with bytes of hex values.  I need to look for the x16,x16 (RS232 bisync) then grab the next 375 bytes which make up a message.  Most of the files have bytes that are not aligned on the byte boundary of “x16,x16” (b00010110, b00010110) pattern (See file 1.txt).  File 2.txt is more representative of the data I have to work with.  I have NO choice on the data as it is produced by someone else – so I can’t change the collection method.  The “x16,x16” is in the file, but it is not on the byte boundary.  For example in the file “2.txt” “x16,x16” or “ b00010101,b00010101” is starts three bits past the xCE  and xC2 swapped (xC2xCE) and concatenated (1100001011001110), using the

xC2                       xCE

11000010            11001110    (first x16 sync)

x22                        xC2

00100010            11000010    (second x16 sync)

x00                        x22

00000000            00100010    (x01 soh)

x22                        x00

00100010            00000000    (x10 dle)

xE0                        x22

11100000            00100010    (x01 first half of message byte count)

xAE                       xE0

10101110            11100000     (x77 second half of message byte – x0177 = 375 bytes).

Any ideas on what would be the best way to approach doing this.  I am very new to LV so any help is appreciated.  I know I need to look at the hex data as binary to find the pattern, then shift the data by the number of bits it is offset.  I then take the data and calculate all kinds of things from it.  I am able to do this already (I know because I have files that I have recorded that are always aligned on the byte boundary).  I am just stuck on what to do to convert the files generated by another group.  Thanks!  Please look at the attachments for clarification and data.  PS - I'm not a student - I've just been struggling with it for a week. 

[tst]

I didn't understand exactly how your files relate to what you're describing, but if I understand you correctly, you want to treat the file as a stream of bits instead of bytes, and then look for a specific pattern of bits and know how much it's shifted from a multiple of 8 or 16.

I'm assuming performance isn't a big concern, so I would suggest the most naive approach - assuming your data is coded as a string, use the String to Byte Array function to convert it, then convert each byte using the Number to Boolean Array function, then use Resize Array to convert it to a 1D boolean array, which is your array of bits. Then go over the array and when you find a bit which starts the sequence, check the next N bits (use Array Subset) and compare to the pattern (right click on the compare function to change to an aggragate comparison mode). If it matches, you're done, and you can use Quotient and Remainder to see how much off the 8 bit alignment you are. There are various improvements which can be made to this method, as well as other methods you could probably use, but that's the first one that comes to mind.

[gEEkPockets]

I get what you're saying up to the point of how to find the pattern.  I have attached what I have so far.  I am not parsing a file in yet as I want to get this part right so I just put constants in for the hex values.  I put four hex values in, then concatenated them and created a binary array.  I want to look for a x16 (0001 0110) which is in the array but not in the byte. Can you please expand - and sorry for such newbie questions. 

[altenbach]

please attach the actual VI instead of a gigantic oversized picture. We cannot run or debug an image.

[tst]

I'm with Altenbach on attaching actual code, but before you do, there's some stuff that you should do first - your biggest problem here is that you're working on individual bytes, when you should be working on an array of bytes. This will get rid of your duplicate code and will make it possible to actually do what you want.

Anyway, to actually do this, you have to always be aware of what data you're working on. You should start with a 1D U8 array of size N and then go to a 1D array of booleans of size 8N. That can be accomplished by going over each byte and converting it. If you're using LV 2013, you can configure the output tunnel to concatenate the arrays, otherwise you will need to do this yourself using Reshape Array or inside the loop.

Once you have the boolean array, you should just iterate over it in a for loop and get the next 16 elements to compare them, as I described before.

[gEEkPockets]

Thanks for the advice.  I am unable to access the forum from work and unable to take work home.  So my attempts at describing the problem are obviously inadequate.  The huge pic of the vi was just my attempt to master the bit shifting thing and not actually code I'm using.  I will try to scrub the actual vi or write another vi and upload the actual code.  It won't happen for several days because of the holidays, but I will try to get it done asap.  Thanks again!

[gEEkPockets]

I have attached the actual data and actual vi that I have written.  The resulting files are to going to feed into another vi that has already been written.  I need to generate files that have found the sync pattern, then the next 375 bytes of hex values (8 bits), then repeat for the next message until the end.  I'm not sure how to approach the byte swapping, looking for the bit pattern with x16,x16,x01,x10,x01,x77 (which should all be shifted by the same amount), then grabbing the next 375 bytes in the same way (i.e. same shift). 

A.txt results in a file that has a shift of 0 bits.

B.txt results in a file that has a shift of 3 bits. 

Hopefully this makes more sense. 

Well, maybe not.  I can't upload the binary files that I am converting. Do I have to change the extension?  I have attached the A.txt & B.txt which are the files generated from my vi. 

[gEEkPockets]

Okay - after several attempts, I have figured that it doesn't like my trying to upload the binary input files to my vi. Is there a way I can do so? 

[gEEkPockets]

Got them uploaded. I have attached the A.xyz & B.xyz which will need to have their extensions changed to .bin to work with the vi.  I have also attached another pic that shows the pattern I am searching for and how it looks in A.txt & B.txt.  My goal is to have a file that has the x16,x16,x01,x10,x01,x77, then the 375 bytes that follow and repeated for every message - not the data that has to be shifted going into my next vi.

[ShishKeBobby]

Hi gEEkPockets,

I'm not familiar with this format; can you break this down into single steps where we can work on each step then move on? I think that would be the best way to attack this.

Regards,