What's the best way to handle all my data?

[russelldav]

I have a black box system that connects directly to a PC and sends 60 words of data at 10Hz (worse case scenario). The black box continuously transmits these words, which contain a large amount of data that is continuously updated from up to 50 participants (again worst case scenario) 

i.e. 60words * 16bits * 10Hz * 50participants = 480Kbps.  All of this is via a UDP Ethernet connection.

I have LabVIEW reading the data without any problem. I now want to manipulate this data and then distribute it to other PCs on a network via TCP/IP.

My question is what is the best way of storing my data locally on the interface PC so that I can then have clients request the information they require via TCP/IP. Each message that comes in via the Ethernet will relate to one of the participants, so I need to be able to check if I already have data about that participant - if I do then I can just update it, if I don't I need to create a record for the participant, and if I havn't heard from one for a while I will need to delete it. I don't want to create unnecessary network traffic. I also want to avoid global variables if possible - especially considering that I may have up to 3000 variables to play with.

I'm not after a solution, just some ideas about how to tackle this problem... I thought I could perhaps create a database and have labview update a table with the data, adding a record for each participant. Alternatively is there a better way of storing all the data in memory besides global variables?

Thanks in advance.

[andre.buurman@carya]

An array of clusters, with each cluster containing info on a participant would be a solution. Its easy in use and easy to create and maintain.

André

[GerdW]

Hi russelldav,

one note on your data handling:
When  each of the 50 participants send the same 60 "words" you don't need 3000 global variables to store them!
You can reorganize those data into a cluster for each participant, and using an array of cluster to keep all the data in one "block".
You can initialize this array at the start of the program for the max number of participants, no need to (dynamically) add or delete elements from this array...

Edited:
When all "words" have the same representation (I16 ?) you can make a 2D array instead of an array of cluster...

Message Edited by GerdW on 10-26-2007 03:51 PM

[russelldav]

Thanks for the quick replies guys!

Although I am limited to receiving data from 50 participants, they won't necessarily always be the same 50. e.g. one minute you might be receiving data from participants 1 through 50, then 40 through 50 could drop out and 55 through 60 and 65 through 70 might join i.e. only the number of participants I want to deal with is limited to 50.

I've not used arrays or clusters in LV much, but this method sounds like it is ideal. If I create an array would the pseudo-process just be....

1. Data arrives from the black box

2. Process data and determine ID of participant

3. Is this the first instance of this participant? If so add them to the array

4. If not search for the element in the array which matches this participant and update the data

5. Loop

[andre.buurman@carya]

Yes.

BTW you could also look into using a variant data type as data store (intermediate level). See following nugget.

Message Edited by andre.buurman@carya on 10-26-2007 04:08 PM

[KostasB]

Hi russeldav

Just to sum up some of the comments:

1) Initialize the array to accomodate for the max number of users.Avoid deleting or adding elements to the array dynamically as it will continuously reallocate memory

space for the new array which will take up time and memory resources.

2) Don't use global variable as you may run into race conditions problems, e.g. you may want to write to the array the same time someone else it trying to read from that.

3) Use the TCP broadcast function to broadcast your data over the network.

Thanks,

KostasB

Applications Engineer NI UK

[russelldav]

I can't see a "TCP Broadcast" function? I thought TCP had to know the address of a client that it's talking to?

i.e. Instead of having the Interface PC spitting out the data is it best to have the clients ask for it?

Pseudo-Process

1. Interface PC constructs data array to distribute

2. Interface PC listens for connections

3. Client requests data

4. Interface PC responds

???

How would this behave if I had lots of clients requesting to view the data at the same time?

Edit: I just found the "Multiple Connections" Server and Client example VIs. I guess this is what I need?

Message Edited by russelldav on 10-30-2007 04:15 PM

[KostasB]

Hi russeldav,

your server model would have two loops:

The first loop will be a TCP listener that will listen for connections on a particular port. If you don't want to loose any of the connections, you should add these connections into a

queue.

The second loop will process the queue elements and and do all the data conversions ( TCP uses strings to handle data).

http://digital.ni.com/public.nsf/allkb/DD7DBD9B10E3E537862565BC006CC2E4

Thanks,

KostasB

Applications Engineer NI