Publish data to multiple users efficiently and flexibly

[Matt321]

Hi,

I'm looking for some suggestions for general approaches to the following problem:

I would like to make a system that performs one or several operations on some data. These operations include:

 - analysing the data and writing the result to disk

 - writing the raw data to disk

 - displaying a subset of the data for diagnostics purposes

Different combinations of these operations will be performed depending upon the situation. For example, raw data will not usually be written to disk, but this will happen during testing. I would also like to keep the flexibility add new kinds of operations at a later data (a different analysis method, for example).

This sounds a bit like the ideal case to use a publish/subscribe or observer design pattern, but there's an additional constraint: Because of the high quantity and speed that data is generated I would like if at all possible to avoid making copies of the data. Some of the potential operations don't even look at all of the data (for example, displaying a subset of the data for diagnostics purposes only needs to see 1% or so of the data so copying all the data for this operation would be inefficient).

Any suggestions for general approaches to this problem would be very welcome.

Cheers,

Matt

[mcduff]

I typically use multiple JKI State machines(see VI Package Manager) attached via user events. User events are extremely efficient and versatile. If a loop does not need data, then there should not be a memory copy. If you are worried about data copies you can always pass a DVR reference around instead of the actual data set.

cheers,

mcduff