I have an application with a high channel count (> 60,000) , and users running the front-end want to be able to plot any channel on demand. More importantly, they want to be able to pull up any given channel and already have the last X minutes of data immediately available for graphing, to view trends.. In order to accomplish this, we have been globally buffering all data from the moment the user launches the GUI, so the data will be available on demand. And we're doing that live, in RAM.
As you can imagine, it chews up a lot of memory. We've talked about other better ways to do this--the most obvious would be to store the data in a local database, and once a user queries a given channel, preload the graph with data from the database, and then start buffering data for the channel directly in memory moving forward. The problem is, we are dealing with (at times, anyway) very high data rates. I have my doubts that the native Database tools would be fast enough to keep up with the demand.
I have been toying with the idea of streaming the data to disk using TDMS format for the same purpose. Save everything that comes in to your GUI (in this case, the GUI really is just a front end...it's not a traditional NI DAQ system) to TDMS files, and if a user opts to pull up a plot of device X, preload the last, say, 30 minutes from file first, then start buffering.
Has anyone out there tried such a design pattern? Any thoughts/suggestions/caveats? More importantly, any benchmarks? Again, to emphasize, we are talking about 60,000 unique devices, at varying rates from 0.1Hz to 50Hz.
Or has anyone done any database wizardry to accomplish the same basic idea? What DB protocols did you use? Any advice?
