Load large .txt-file into graphic

[Cheese11]

Hi everyone,

I'm busy with an application where I want to load a large block of data into a graphic.

The file that I want to load into the graph is a .txt-file.

I did put one line of data into the .txt-file and copied that, just to have a lot of data into the file.

The problem is, that when the file is above approximately 10MB and up, the application is having a lot of effort to do this without problems.

Most of the time the VI will hang and nothing is possible anymore.

In the attachments you can see how the file is build up.

I also attached the VI with Sub-VI's.

Another question, if I zoom in on the graph, is it possible to see the zoom factor?

Because maybe it is possible to leave some data out of the file.

I am logging data for every 5 minutes.

Maybe it is possible to, when I didn't zoom in, one sample per hour is shown in the graphic.

And when zooming in, there will be shown more points of the graph.

But then I have to know if I can read out the zoom factor.

Regards,

Kenny

[smercurio_fc]

I honestly don't understand what you are trying to do. Why are you loading a text file into a graphic? That makes no sense to me. What is in this text file? I saw the sample file you provided, but I don't understand how I'm supposed to interpret it.

You code has a few issues (such as NEVER HARDCODING PATHS), but I'd rather try to undestand the concept of what you are doing before commenting on the implementation.

[vt92]

You will probably have to change your file storage strategy.  Take a look at this article:

http://zone.ni.com/devzone/cda/tut/p/id/3625

Is it possible to reduce the amount of data in each file?  For example, if you know the delta - t, you could leave out the timestamp.   Instead of one large file, could you use several smaller files?  You also need to be mindful of the arrays you are using and how much data is staying in memory. 

[Cheese11]

Dear Smercurio_fc,

I have a sampler that is writing every 5 minutes some measured data into a txt file.

The variable names of the values that were sampled are shown on top of the txt file.

The reason that these are shown, is that they can be selected (one or more) in the "#plots" box which is also shown.

After that, the variables are shown after each other, with a tab between them, and under them the sampled values.

If you want to see it good, you can copy the tx file into excel, than it's easier to read and understanding it.

The reason why I'm using a hardcoding path is that this is a test VI.

If it's ready and working, I will build it into my main VI.

I hope it is a little bit clearer for you know.

Regards,

Kenny

[smercurio_fc]

OK, so when you said "graphic" you meant graph. Because a graphic is a picture, so that's why I was confused.

So the file you are trying to read is being updated continuously by some other program, which I assume you cannot change. In that case, you want to be able to show all the data on the graph. This means you basically have two issues. The first is to parse an ever increasing file, and the second is dealing with ever increasing data in your graph. Here's the things I can tell you:

• Your code is memory heavy for this reason, as well as all the extra array indicators which are creating copies of data. These are not needed. If you want to see the data, use probes.
• You can also simplify the parsing to make it faster. For example, in the "Build Graph and Table" subVI you are using a shift register to build up an array for a table. You do not need to do this, since the data is already available from the other loop. All you're doing is causing LabVIEW to perform more memory thrashing in trying to allocate memory for an increasing array.
• Do not create a timestamp only to convert it to an integer. You are losing the precision of the timestamp datatype. Besides, the graph will accept timestamps, so stick with that. Just set the X-Axis to absolute time.
• The parsing of the 2 columns for the timestamps can be simplified by using just 2 Scan From String functions.
• Do not use Delete From Array when you should really be using Array Subset. If you're not using the other outputs, it means you should be using Array Subset.
• Why did you create a special Read From Spreadsheet File VI? The one that comes with LabVIEW is polymorphic, so select the version that returns the data as string.

I've attached modifications of the two core subVIs as well as the top-level VI to show an improved performance.See if they make sense. I made a small artificial change to the timestamps since the timestamps in your sample data are all the same. I do not know if this is a real file, or whether you just copied the same line over and over again (I am guessing the latter).

However, this does not solve the main problem, as that requires a lot more work. What I would do is to keep track of how much of the file you've read. Thus, in the loop, do not keep reading the whole file. Only read the additional lines. This way, you have a lot less data to parse. As for the graph data, I'd suggest pre-allocating a large buffer for the data and simply using Replace Array Subset to fill in the buffer as you read lines from the datafile. If you need more, then allocate a larger buffer and use that instead, throwing away the previous buffer.

Good luck.