Random Serial Read Overflow vs variable multi-line reply DUT

[Mikejj]

Hi All,

I am dealing with an interesting Serial, linux based device that takes different commands, which then will spit back different amounts multi-line data.  There is no terminating character except for the multi-line new line characters.  So basically, the amount of data that the device returns depends on the command I issue to the device.

The solution I came up with is to come up with a terminating string "endtest" since I can echo it back from the device and use serial read to read back 1 byte at a time with a string shift register as a buffer and check for the string "endtest".

Read Until

However, the issue I found is that I will random get a overflow error.  I am suspecting my loop is not reading the data fast enough due to the "post-processing" I am doing such as the termination string search, and the logic OR boolean calculation.  My previous version of this code had a constant polling of a FGV, which I removed suspecting speed issues.  I am still not sure if my new version without the FGV is fast enough and want to reach out to the community and see if there are some alternatives I can try.

Best Regards,

Mike

[deceased]

Mike, 

I don't have 2018 so cant tell what exactly is in the snippet, what is the value in the Termination Regex?

How long are the longest multi line strings?

A few things that might get the loop to run a bit faster:

Using the  "bytes at port" property node, then reading that number of bytes into your concatenated string might give some benefit over looping more for single bytes. (I haven't benchmarked it so it may actually perform worse).

Also might be worth keeping track of the concatenated string length and starting the regex search from there (using the offset input), save it searching through the whole string each time.

0xDEAD

[mcduff]

I use something like the snippet below when I do not have a terminating character. It basically polls the bytes at port until the value stops changing, then reads the reply. Depending on how fast your serial port is, you may have to change the waits. (2015 vi attached)

mcduff

[Mikejj]

Hi Deceased:

My longest multiline string is undetermined...as mentioned, depending on the commands the user will send, the replies from the DUT changes.

Hi mcduff,

Depending on the timing, won't this still cause possible buffer overflow issues if my data comes in faster or fills up the buffer faster than then time it takes for bytes at port to settle? 

Regards,

Mike

[mcduff]

---

@Mikejj wrote:

Hi Deceased:

My longest multiline string is undetermined...as mentioned, depending on the commands the user will send, the replies from the DUT changes.

Hi mcduff,

Depending on the timing, won't this still cause possible buffer overflow issues if my data comes in faster or fills up the buffer faster than then time it takes for bytes at port to settle? 

Regards,

Mike

---

Why don’t you try it. Since you have a special case it’s hard to predict.

mcduff

[RolfO]

May it help to increase the buffer size for the connection? 

Do you have any data after your terminator string you need to be kept in the buffer?
If not you could set the timeout value for the connection to 0 and read a bigger chunk of data at once instead of 1 byte at a time.

If you need the original timeout value afterwards, just read it before you set it to 0 and when you are done set it back to the original value after you have read your string..

[deceased]

---

@Mikejj wrote:

Hi Deceased:

My longest multiline string is undetermined...as mentioned, depending on the commands the user will send, the replies from the DUT changes.

Hi mcduff,

Depending on the timing, won't this still cause possible buffer overflow issues if my data comes in faster or fills up the buffer faster than then time it takes for bytes at port to settle? 

Regards,

Mike

---

You must have some rough estimate though. are talking hundreds or millions of bytes. Array preallocation and replacing elements rather than concatenation might help if its a crazy number of bytes.

What is the regex you intend to use, some can be terrible, there might be a more efficient one.

The bytes at port get the number in the buffer, you then read these as a block freeing the entire buffer (less the few bytes that might arrive as you read, you'll catch them in the next block).

Out of interest just how fast is your BAUD rate?

0xDEAD

[daveTW]

Hi,

how much data do you expect in the biggest message. I think, the serial buffer is set by default to 32kB, does your message exceed this amount?

To what baudrate is the interface set?

Edit: OK, deceased was quicker..

But, another hint: with every Byte you read you send the "match pattern" through all the (crazy big) message again. So what about just taking the last few bytes and comparing them with the pattern. Should be faster...

[Mikejj]

Hi All,

Thanks for the suggestions, sorry for some reason I thought I already included the baud rate...I am running at 115200.  I believe the default serial port size is 4096 bytes.

My regex (or more precisely, the match pattern) search string is just the same as the termination string "endtest".  Basically, as mentioned, I am basically sending an echo command to the DUT, so I can actually set the termination string to be whatever I want.

From scanning through the forums, there seems to be a big avoidance on using "Byte At Port" since it is not an accurate representation of what is exactly is available.  My issue with also using this is that the DUT is not guaranteed to be streaming data out all the time.  Depending on the command, the DUT might be processing some internal test/commands and will pause before putting data on the serial line.  I am afraid I will get "Byte At Port" reads = 0 intermittently.

If I choose to only look at the last few bytes, won't there be a chance that I will miss the whole string.   Although I think this might be one of the big issues as mentioned, I am searching through the ever increasing giant string again and again.

Regards,

Mike

[mcduff]

---

@Mikejj wrote:

My regex (or more precisely, the match pattern) search string is just the same as the termination string "endtest".  Basically, as mentioned, I am basically sending an echo command to the DUT, so I can actually set the termination string to be whatever I want.

---

If that is the case, can you use a character that is NEVER is your response as a termination character, such as a tab or !, etc. Then you can read until the character is in your response. You can set the termination character with a property node.

---

@Mikejj wrote:

From scanning through the forums, there seems to be a big avoidance on using "Byte At Port" since it is not an accurate representation of what is exactly is available.  My issue with also using this is that the DUT is not guaranteed to be streaming data out all the time.  Depending on the command, the DUT might be processing some internal test/commands and will pause before putting data on the serial line.  I am afraid I will get "Byte At Port" reads = 0 intermittently.

---

Bytes at Port should be avoided IF you have a termination character, which you do not have. In the VI I posted, it will read bytes at port until a timeout is reached or two consecutive readings give the same value.

Since a command may take some time, that is where you want to adjust and add a time delay before reading.

---

@Mikejj wrote:

If I choose to only look at the last few bytes, won't there be a chance that I will miss the whole string.   Although I think this might be one of the big issues as mentioned, I am searching through the ever increasing giant string again and again.

---

Regex is typically slow.

Lastly, have you tried any of the suggestions listed here? It is easier to give advice when you try something and show how it fails rather than list reasons not to try something.

mcduff