How to manage tcp buffer with array

[electric550]

I am having trouble parsing data from the tcpread buffer. I can parse the data from the buffer I was just wondering if the following is possible...Is there a way to have each tcp packet that is received be put into an array element instead of having the buffer append the in received data onto itself. It would be much easier for me if the buffer was divided in to packets that way I could just parse each packet instead of having to find each packet then parse each packet individually? I was just wondering if  there was an example or something, or do I need a vi myself to handle this. I noticed that the TCPREAD vi block has 4 options for the buffer...this is not one of them...but perhaps someone has done something like this before? Thanks for any help.

[CoastalMaineBird]

The TCP layer isolates you from the "packets" per se - you should not need to know nor care about the "packets" as used by the transport layer.

I almost always (>99.9%) use the BUFFERED mode for reception, with a 0 TIMEOUT value - I ask for X bytes, and let the system do the buffering.  I either get the X bytes (in which case I process them), I get a TIMEOUT error (in which case I tend to other business), or some other error (which I deal with). 

The value for X is completely under my control and depends on the message protocol.  Whether my message arrives in one packet or a hundred is irrelevant to me.

I have a couple of blog posts that might be relevant:

Beginners Guide to TCP/IP 

[electric550]

Well I don't know if this is the right place for this but perhaps that would be a suggestion for future versions of labview...to add the option of having the TCPREAD generate an indexed stack array of input packets instead of just lumping them all together, basically the newest packet is put on top of the stack and the rest are bumped down.

Or perhaps have an interrupt available for when a packet arrives...I have a bunch o binary data that is easy to parse, but when two of them get lumped together do to buffer it is hard to pick out the data because there are different message lengths depending on what is being sent. I just bring this up because I am aware that every datapacket is separate when it arrives, and it seems like that would be a lot easier to deal with if they were sitting in an array and I could read them whenever I got the chance. 

[electric550]

I do like the examples that you linked to, they are nice. The four while loops make the parallel part really easy...Unfortunately the project I have has a variable number of connections so it could be one or it could be up to 65535.

Message Edited by electric550 on 01-05-2010 05:02 AM

[CoastalMaineBird]

to add the option of having the TCPREAD generate an indexed stack array of input packets instead of just lumping them all together, basically the newest packet is put on top of the stack and the rest are bumped down.

Again, you are in the application layer of TCP, the packets are in the transport layer. I have worked with LabVIEW TCP for 15+ years and never needed to know about packets. 

Packets are a way of the transport layer (possibly) breaking up a message and getting it from here to there, possibly by different routes, possibly in different order from what was sent.

TCP guarantees that they will be re-assembled in the correct order and presented to the receiver in the correct order.

But all that is under-the-hood, you don't need to know it when using TCP. 

Perhaps if you explained what you are trying to do, it would help us to solve your problem. 

[CoastalMaineBird]

Or perhaps have an interrupt available for when a packet arrives..

You're thinking in assembly-language terms for a high-level language.

There are no "interrupts" in G, there are "events".

And how exactly would that work?  You would have to have a connID and a byte count in the event parameters.

For example, I have a program where the host PC and a PXI box exchange variable-length messages from a PXI box.

Each message is two parts: a 3-byte header and a variable-length payload.  Two of the header bytes make up a U16 which describes the payload size.

I read a header (ask for 3 bytes), then turn those 3 bytes into a header cluster.

I use the U16 value from that in a second TCP READ to read the payload.  The payload might be zero bytes, or it might be 64k Bytes.

Whether TCP breaks that 64k up into one packet or a hundred is completely irrelevant to me, what's important is that I asked for 64k bytes and I get 64k bytes.

And I never wait on it, I always use TIMEOUT = 0, and if the 64k is not there, I do other stuff.  Let the OS buffer it. 

That scheme would never work on an "event" basis, but it doesn't have to. 

[CoastalMaineBird]

Unfortunately the project I have has a variable number of connections so it could be one or it could be up to 65535.

I've been thinking of a server-type example for my blog.

I've written a web server in LabVIEW before (1995 or so).  Basically you separate the listener from the service functions.

The listener listens for connections and puts CONN IDs into a pool (queue).  The pool contains all the connections that are active at the moment, along with some sort of status flag.

A service loop continuously extracts a connection cluster from the pool, handles it according to it's status, and either returns it to the pool, or terminates the connection and  removes it from the pool.

The idea is that the server is a sort of state machine which runs through all current connections and gives them some attention.

You could also do it by spawning a number of copies (one for each connection) of a single reentrant VI, but I don't know about the performance of that.  If you truly have a thousand connections at once, then you would have a thousand instances of this VI running in parallel.

That's a bit daunting to me, but then I've never tried it. Perhaps someone else could comment on that feasibility.  It would seem that the thread-management overhead might be a problem, but I don't know. 

[electric550]

I was just throwing that out there because I have have a hardware background so I like to interface at low levels because I feel it is more efficient, and I know that most ethernet chipsets can generally send out the data on interrupts so it  is not too hard to implement at a low level, but I understand that Labview designed to be an abstraction, plus it sits on top of a bunch of other software, so all of that does not need to be worried about so I will just drop it at this point....

Anyways Basically there are some simple low level data acquisition and  control boards that are on a network and configured to talk to a server(that I am trying to put together). I have been asked to make the program scalable so that they can add more or remove sensors if necessary. So basically the sensors are designed to connect to a server using the port and IP address, this works fine and I can read all of the data and send data back to a single board, but I now need to be able to handle a bunch of them at once. Each sensor knows the IP and port number of the server,but the sensors have dynamic IP's so they are responsible for connecting to the server because the server has no real way of finding them...There is also A data base but I we will worry about that later.

So I could just plop 50 little vi icons in a while loop to handle 50 controllers, but I would like for the program to just spawn the vi's when necessary and close the vis when not necessary. The multiserver vi example that is included with labview is nice but I do not think that it is multithreaded, it just keeps track of the number of clients.

Anyways that is what I have been asked to do so I am just trying to put something together.

So basically the bunch of while loops will work but that would not be very efficient and it would not scale itself...thanks again for your help. 

[electric550]

Theoretically all of this could be done in one thread, I just need to be able to open and close a bunch of connections ID's then Just lump all of the data together and parse it that way. Each controller has a distinct number built into its data packet...So I can tell which controller is which that way. I really wanted the array thing so that parsing all the data would be easier...I will just try to do this in one thread right now because I just tried to run two of the vi's at the same time, and it sounded like my computer was about to blowup.

[electric550]

I think I just repeated what you said but I am just talking...I really like how simple the mutliconnectclient.vi is. That is basically how I would want to make it...I am just not sure if it is a good idea to hard code thousands of little while loops on there...although it would probably work...I think the tcpopenconnection would provide the mechanism for controlling which programs were running and which programs were not running. Thanks again for the help.