Plotting millions of points in a .Net WaveformGraph/WaveformPlot

[dpstark]

Hello,

I have an application I have written in C# using Measurement Studio and the WaveformGraph. The application deals with multiple data sets containing varying amounts of data. However, millions or tens of millions of points per data set are not uncommon. While I understand that displaying 10's of millions of points is kind of pointless (yes, that's a pun) since you can't really see them, I don't currently have a way around it. It is a requirement, however ridiculous, to plot some representation of all the data in a set. The amount of time required to plot so much data is very frustrating to my users.

I noticed this post addressing the issue to some extent. However, it is relatively old and I tried the implementation, which resulted in a 2-4 time improvement in display times (isolating the time it takes to plot and display the data only). This is good, but it's not a 10x improvement. I reimplemented the sub-sampling algorithm in the example to grab the minimum and maximum values within each pixel bin as well, which resulted in a much better plot (the example was definitely causing points to drop off the display that our users need to see).

This approach has several drawbacks. First, it is code I need to maintain that I feel like should be duplicating code that is somewhere in the WaveformPlot/WaveformGraph classes. Second, it obscures functionality built into the controls. For example, this only works with plotting lines. If the user wants to see points, this is very easy to do with the built in classes. It is not possible with this custom implementation (at least, not without a lot of work on my part). It also causes some weirdness with things like cursors since the plot line is drawn manually and there aren't actually any points being displayed by the plot. And finally, the 2-4x speed-up isn't really enough to justify the loss of features as described above.

So that's a lot of information, but I think it's necessary. My questions are:

1. It seems like this sort of thing should be done in the classes already. It must be done in some way. Am I just duplicating code?

2. Is there anyway to instruct the WaveformGraph or WaveformPlot to reduce it's fidelity or something in order to get faster plots?

3. Are there any better suggestions for handling such large amounts of data?

I have attached the modified example. It prints the amount of time required to plot (display only) each type of graph, allows changing the number of points displayed, the oversampling factor, and resizing the display. Thanks go out to the original poster for doing much of the work on this.

[K_Joy]

Hi dpstark,

There is a Decimate 1D array function built in to the NI analysis library:

http://zone.ni.com/reference/en-XX/help/372636F-01/mstudiowebhelp/html/50cc778f/

I’m not aware of any built-in functions that have some of the custom things you have implemented, with the min and max.

There are also some suggestions for increasing the plotting speed in the help documentation that is worth looking over:

http://zone.ni.com/reference/en-XX/help/372636F-01/mstudiowebhelp/html/optimizingnetscatterwaveformg...

Kelsey Johnson

Applications Engineer

[dpstark]

Thank you for the response. The approach taken by the example I pointed out and modified is not going to work. There are problems rendering the line when you change the line width. For some reason I can't pinpoint, lines that are rendered as vertical get "tails" on either end that stick out beyond the actual bounds of the points. 

I checked the information on optimizing the display, and I am doing most of that. Even doing all of it makes only a small difference. And things like grid lines- well, unfortunately, customers like to see those... I can let them turn it off if they're really impatient, but they do want that feature.

Can anyone from NI comment on what the control is doing in the background with regard to subsampling or decimating the data? I will keep working this issue, but in general, I don't see anyway to meet customer expections with this control any more.

[K_Joy]

dpstark,

Unfortunately, as far as I am aware, there is nothing being done in the background to that effect. It seems like, with your requirements, there might not be any way around the slow drawing and updating of the graph. The best bet is definitely to do some sort of subsampling yourself, but you said that obscured data your customers needed to see. If you are missing spikes of data, then doing some sort of running average instead of picking and choosing data points to plot might help avoid hiding important data. 

Kelsey Johnson

[dpstark]

As you can see from the example I posted (based on the previous example on this same sort of question), I wrote a routine to slice the data being plotted into bins for each pixel of the graph. From each bin, I selected the min and max values and plot those in the order they occur in the data. A decent graph required at least a 2x oversampling for each pixel (meaning two min's and max's for each pixel). This route seems to capture all data for display (compared to the original example, which clearly dropped important data).

Aside from the fact that drawing the lines myself leads to the loss of features normally built into the control, it also caused some strange behavior in the actual drawing of the plot line. This behavior is limited to the "DrawLines" function that is used to manually draw the plot lines, but essentially whenever it renders a vertical line with greater than 1 pixel width, it creates artifacts that make the values look larger than they are. In the attached example, I provide a sawtooth pattern that ranges from 0 to 15 in Y. The example fixes the Y axis to (-5,20). The "custom plot" will show an overshoot of (-2,17). Zoom in on the Y axis to about (0,0.005) to see the pattern. At a sufficient level of zoom (to about (0,.001), the graph will revert to the default WavefromPlot and show the correct pattern. The difference can easily be seen at the default full view when switching between the default and custom plots.

I do not know enough about graphics to solve this issue myself, and the end result would probably not work out for me anyway. At this point, I have to look for another solution or settle for providing a subset of data to the control and letting it draw the subset. This also isn't ideal because I have measurement and cursor functions that operate on the data and I would have to rework all of that to point to a data set in memory instead of the data held by the plot- quite a lot of rework. And, doing so would result in irregularly sampled data since I'm grabbing only local minima and maxima to represent the data.

Anyway- unless there is new information, there is no need to reply and I was just posting this in case anyone has similar problems and can learn from this.

[andreacer1]

I can suggest you the workaroud i have used. I record millions points and the user looking for spikes (10 points over 10 millions) . I record all the point over a off line array and capture the changexrange event, so replot only the requested points.

All the other function still work.

[justwakingup]

I just posted a similar answer to another post here http://forums.ni.com/t5/Measurement-Studio-for-NET/Measurement-Studio-vs-LabView/td-p/2792490