Com library, buffers, callbacks, and the nature of life

[ElectroLund]

I'm embarrassed to say that I've been using the CVI RS232 library for some time now, across multiple projects, but only now getting a fuller picture of how the thing works.  And I have some fundamental misunderstandings of the beast that I'm hoping all you wonderful people can clear up for me.

I have a new project with a device that talks at 115200 baud.  It has one particular mode which spits out data dumps continuously.  It's up to me to turn off this mode after a sufficient number of samples have been taken (~400).  This number of samples equates to 18,800 bytes.  No problem -- on my Win 7 machine and serial port hardware, I have plenty of space in my queue.

So I set up an InstallComCallback with this buffer size in mind.  My thinking is, capture the whole block first, then read it out of the queue at a leisure pace offline.

The ComCallback fires as expected.  Immediately, I do a 

qBytes = GetInQLen(port);

Consistently, I get qBytes = 19840 (not sure why it's more than expected, but whatevs.  Then inside the ComCallback, I execute a series of ComRdByte calls.

Here is where I was getting -99 timeout errors.  So Right before the ComRdByte, I did another GetInQLen.  I get a different value now, considerably less.  

I guess my misunderstanding is this: how robust is the com buffer?   Can I not depend on that data being there for later when I need it?

[ElectroLund]

Oddly, the more I experiment, the more I'm confused about how the library works.  Here's what I know now:

If I don't shut off my serial device quickly enough, the data inside the queue passes me by, despite setting up the InstallComCallback to trigger after N bytes.

The key challange for this app is the speed at which the device is outputting data.  I don't believe I can capture it fast enough.  The data starts outputting when I command it.  It outputs at a 5ms rate.  The blocks of data look like this:

08000503  <- header
09  <- ack
0A  <- ack
0000001F000000000000000000000000000000000000  <- data
0B  <- ack
0000001F000000000000000000000000000000000000  <- data
09  <- ack
0A  <- ack
0000001C000000000000000000000000000000000000  <- data
0B  <- ack
0000001F000000000000000000000000000000000000  <- data

After the 4-byte header, there are repeated 47-byte blocks of data.  The blocks are prefaced with a 0x09 ACK, then two sub-data blocks each prefaced with 0x0A and 0x0B ACKs and 22-byte data blocks.  These 47-byte blocks repeat until I command it to stop.

What I'm seeing is that no matter how I attempt to callback this, I get off in my data frames and can't get back in sync.  Is there some faster method I should be trying to capture?

The way I captured the above was using a Com port tool called RealTerm, and capturing in hex directly to a file, since it dumps so fast to the port.

[Gloorious_Leader]

I think I might be missing something. In your first post it sounded like you aren't able to acquire any of your data, but in the second post it sounds more like you are able to read some of the data (just out of sync). My first thought was that somehow the queue is being cleared before you read from it, but the more I think about it, the less sense that makes. It seems more like the data is getting overwritten before you have a chance to read it. I know that's probably not very helpful, but I figured I'd throw my two cents in anyway.

Have you tried any of the other COM Read functions (ComToFile, ComRd, ComRdTerm)? Do you get the same timeout with them?

Honestly you might want to try the Instrument Control board as well. Someone there might have a little bit more of an understanding of how best to approach this task.

[ElectroLund]

Thanks for the reply, Gloorious.  Yeah, it's confusing to me too.  I had been getting timeout errors.  I found that if I used breakpoints in various spots, I could get data.

After much more experimentation with all the other ComRd functions, I concluded that the debugger was lying to me.  For instance, using breakpoints, I found that FlushInQ was not flushing all of the queue!  When my ComCallback function was triggered with at least 18800 bytes, I do a ComRd with that block size, followed by a FlushInQ.  Yet, immediately following, the input buffer still measured about 4000 bytes!  If I then did another FlushInQ, it finally measureed 0.

So then I removed all breakpoints and instead used DebugPrintf calls with the buffer size.  This gave me more believable results.  FlushInQ was indeed working and the buffer is now reporting 0 afterwards.

So, I can see data after all.  However, I am still capturing data too late.  My ComRd never catches from the beginning of the data dump.  It's somewhere in the middle, even when I stop the device from dumping almost immediately after it begins.

[Gloorious_Leader]

Can you tell whether the data is being overwritten too fast or is the first part just not being captured at all?

[ElectroLund]

Here's an update on this.  As it turns out, this was mostly developer error.  The data was not so fast that the CVI RS232 driver couldn't catch it all.  I am capturing all the data.  Due to some very layered and multiple logic errors on my part, I was fracturing the data frames.  That's fixed now.

The part that wasn't so much my problem, the part that lost me hours of work, was the CVI 2013 debugger, which I'm still very disappointed with.  See this thread for more information.  It's a long, sordid tale that I won't cross post about here.

Suffice it to say that what saved me with this com port issue was not relying on the debugger to give me honest information about the com port buffer!  Instead, I used DebugPrintf statements, sprinkled about like chicken scratch, to give me lots of pleasant clucking noises about the size of my input queue.  That's what shone the light of truth on my aforementioned data frame errors.

Thanks for your help...

[OVR_CZ]

hi,

It seems that you fixed that, but i like to add some tips/thoughts there:

1)When application communicate through buffered interface(like Serial port or Tcp/IP), and you break the application by debuger, incomming data is still received to connection buffer inside operation system. So when you flush the buffer and break/stop application just after flush(or just execute Flush call by "Step over"). The OS-internal buffer is still filled by incomming data even if application is stopped.So when your run/continue the application, you can see tha buffer is not empty. (data can be received to buffer just after flush even if you fo not break the program, Operation system buffer is independent to state of your application)

2)Every I/O call has overhead, it is much better to read 1000bytes at once and cykle through them, than make 1000 reads of 1byte. So if you looking for performance boost, this is easy way.

3)DebugPrintf is great tool. But it can be slow if you debugging high performance task. It is better to store data in your own memory/buffer and then write them to debug console in one call. Or just store data to memory/buffer and use debuger to see them when all required data is collected.

I hope this help

P.S. I am sory for my bad english.