escape character code optimization

[jerry12345]

Cameron,

Alright, here's the decode:

strings in quotes, hex in hex notation

"0" is "0"

"\0" is 0x00

"\\0 is "\0"

"\\\0" is "\" and 0x00

"t" is "t"

"\t" is 0x09

"\\t" is "\t"

"\\\t". is "\" and 0x09

"\\\\t" is "\\t"

"\\\\\t" is "\\" and 0x09

"\\\\\\t" is "\\\t"

"\\\\\\\t" is "\\\" and 0x09

\t\\0\0\\t would be 0x09,"\0",0x00, "\t"

Does that make more sense now?

Jason

[camerond]

Maybe this can help you, it's not quite the same, but based on similar decoding of a data stream:

(see attachments) We have a serial data string coming in, from which we use 0x1003 as a stop signal. In order to prevent stopping the read prematurely (if the 0x1003 is actually data, not the stop signal), we check each byte, and use each two-byte string for the stop signal. For this, we have to modify the incoming stream, as below.

Wherever we have a 0x10 to send as data (not as the stop byte), we double it, so it gets sent as 0x1010.

Then to decode, we run through the attached VIs to check for 0x1010. If we find that, we interpret it as data, strip off one of the 0x10 bytes, and send it to the output buffer.

If there is just a single 0x10 coming over, followed by something else, we check for 0x1003 (the VI actually checks for 0x0310 because it's a stream at this point, not a file). If we get this, we stop the read and proceed to the next operation, if we don't get 0x0310, we flag an error (never happened yet).

It's pretty simple code, just using a single shift register, I haven't clocked it bur it seems to be quite fast. Maybe the structure will be useful for your problem.

Cameron

[jerry12345]

camerond,

I'm finally off furlough and back to work!  Thanks again for your input.  However I don't think your example is very helpful to me.  You are reading data from your instrument one byte at a time.  I/O operations generally have a lot of overhead and are pretty slow.  Also you are using a couple "concatenate string" blocks in your main loop, which I believe make a memory call each iteration of the loop and are slow as well.  I did try to write a VI using your methods, but it was taking seconds to process my data...much too slow.

I'm looking for some advanced string manipulation techniques.  The native LabVIEW "search and relplace string" takes only 2 or 3ms to perform the operation.  Is it not possible to mutually program a similar operation in LabVIEW that isn't 100 times slower???  Why did my attached attempt at using an "in place element structure" (KATCP - Escape String_read number array3.vi), take a full minute to perform the same operation?  Any ideas out there???

[Nathan-M]

A few things:

1. Using an in-place structure only allows for memory optimization, not speed. Typically there is a trade-off between the two (i.e. faster execution means more memory, less memory means slower execution). The in place structure that you are using may be slowing your application down. 

2. Hex values are bytes, which are a different data type from strings. It is still unclear what functionality you would like to achieve when you say that you would like to see both. 

3. If you've got this subVI open when you run the application, you may be getting performance problems due to the overlapping front panel objects. When onjects on the front panel overlap, they must re-draw constantly.

[jerry12345]

Nathan-M,

Thank you for your comments!

1.  This is very helpful!  In-place structures were EXTREMELY slow.  Memory is cheap on PC's so am I correct is saying, the only time you would want to use in-place structures is if you have a very large array or cluster and you have limited hardware resources (memory)?  Are in-place structures used more in LabVIEW Realtime?

2.  My instrument returns data encoded in an ascii string.  I don't need to read or display this ascii string, I just need to decode it to get usable data.  Strings are made up of bytes.  When I talk about bytes in this thread, I am referring to the non-printable characters that I can't easily write as a string.  I find it more clear just to write the byte code of the non-printable character, especially since the notation I usually use for non-printable characters include the escape character I'm trying to remove in my code.  So if my instrument returns "\0" (two bytes: 0x5C, 0x30) I need to decode those two bytes as one byte: 0x00.  If the instrument returns three bytes "\\0" (0x5C,0x5C, 0x30) this needs to be decoded as two bytes "\0"  (0x5C, 0x30).  "\\\0" (0x5C,0x5C,0x5C,0x30) decoded to (0x5C,0x00). 

3.  I've heard about this performance hit when you have subVI's front panels open...but honestly I've never seen it.  I can't imagine re-drawing front panel objects causing a 100x decrease in performance.

Again, what I'm really looking for in this thread are some advanced LabVIEW string manipulation techniques.  The native LabVIEW "Search and Replace string" is awesomely fast.  When I try to manual implement a similar function with other LabVIEW primitives and I'm getting a 100x increase in execution time!  Processing this stupid string is now frustratingly slow.  It's the smallest bottleneck in a once very responsive data acquisition system.  Before, the only problem was that occasionally I'd get some strange data when the string didn't decode correctly.  Now I get correct data but it's SLOW!  I'm trying to figure out if this a LabVIEW limitation or a programmer limitation?  Take a look at "KATCP - Escape String_one at a time.vi" and tell me if you can make it faster.

[Nathan-M]

You are correct on the in-place structures. Imagine trying to do video processing on a 10MB frame at 30 frames per second, memory would be consumed far too quickly if operations weren't done in place. 

Thank you for clearing up the byte/string thing. 

I've tried running that VI you referenced with a test file that was around 150 kB in size (specifically, your last comment copy and pasted about 30 times) and the VI executes in milliseconds, not the 2-3 seconds you're experiencing. How large is the file that you are attempting to process? 

[Yamaeda · Accepted solution]

This seems to come rather close, and is quite fast. 🙂

Since the output string must go through all characters and inject the specials i do the opposite, i rebuild a new string and remove specials.

/Y

[jerry12345]

Try running "Escape string evaluator.vi".  The control "Input string" is an actual data string returned from my instrument.  There are 4 subVI's that are versions of my attempts to decode the string.  "KATCP - Escape String.vi" was my origional fast code that uses "Search and Replace string" which executes in about 2ms on my PC.  "KATCP - Escape String_one at a time.vi" was my best attempt at using LabVIEW string primatives and takes about 230ms.  "KATCP - Escape String_read number array_no struct.vi" converts the string to a byte array and processes everything as an array and takes 330ms.  Finally "KATCP - Escape String_read number array3.vi" (which is disabled) was my attempt to use an in-place struture which takes about a minute!

Hum...I just ran this same code on a newer PC that has 4 cores instead of 2 and I'm getting fthe following executation times:

KATCP - Escape String.vi: 2ms

KATCP - Escape String_one at a time.vi: 125ms

KATCP - Escape String_read number array_no struct.vi: 86ms

KATCP - Escape String_read number array3.vi: 26720ms

Maybe the array fucntions make better use of the additional cores than the string functions? 

[jerry12345]

Yamaeda,

Sigh...Your solution proves that sometimes the simplest solution is best solution.  My code was trying to be smart by searching and replacing values very similar to the native LabVIEW "search and replace string".  I was thinking this elegant solution would be much faster than brute force iterating through each and every element of the input array.  However, the overhead of manipulating the original data is much greater than just reading the input data and generating a new output data array.  Kudos to you and thanks!  I only had to make one minor modification to make your code work properly.  The while loop was executing one too many times because the while loop iteration counter is 0 indexed.  I just had to decrement the input array size by one.  Attached is my final subVI to encode and decode an escaped string.  Thanks again!

[Yamaeda]

Actually in a philosophical view it's a bit tricky, since the mass replace will cause unions of information sets, or what the term is. That's the beauty of the xml-solution, you cant get this issue with mass replace.

It's like the joke about people not working at all during a year, since from the 365 days you reduce 2 weekdays each week, so 261 days, then you have a month of vacation, so 230 days, some holidays and so on, resulting in 50-100 workdays in a year. The inclusive subset ofcourse being you counting weekends lots of times. 🙂

Glad to help.

/Y