Storing 30MB file opening and displaying on table taking long time (> 30 sec)

[NavneethK]

I have to store a huge amount of data to be stored and displayed, approximately 30 MB files.

When I open the file, it takes a long time to open (around 15 to 30 seconds or more sometimes). Even in TDMS, the time taken is that high.

Actually I have to display it in a table and is blank for that much time.

I tried to break the data into multiple files, but then it became very complicated to access it (because of other parameters to be taken care of) and I am stuck.

The hardware configuration of my target machine (PC) is as follows:
VIA Eden 667 MHz processor
128 MB RAM

The operating system is Windows XP (Service Pack 2).

The CVI version which I am using is CVI 8.1.

[Maciek]

Try to hide the Table Control (or even all Panel) before You fill it with data and after that display table again.

[RobertoBozzolo]

It's not clear to me if the longest operation of yours is loading the table or opening the file, and you should try checking it.

In the first case Maciek gave you the best suggestion of all, the only addition to it being that the table must not be the active control in the moment you are hiding and filling it (if it is, use SetActiveCtrl to move the focus to another control during this operation). On the other hand, if the problem is actually opening the file (with 30 MB of data it could take several seconds!) you should rethink your idea of segmenting data in several files, even with the difficulties you mentioned about the complexity of this operation.

[NavneethK]

Actually the problem is with the time taken to open the file. I tried two options:
1) storing it in own format and using OpenFile to open the file but it took several seconds.
2) stored it in TDMS format. This time the file size increased because of the format I suppose but it also took several seconds to open.

So finally I tried to segment it in files of 1 MB size but then it became difficult for me to do other operations (specific to my application) which could have been easily done if it had been a single file.

Basically the application is like this:
I have to store data continuously for 72 hours at every 5 seconds (the file size amounts to 30 MB with my format) in a cyclic fashion, that is, the file should store for the past 72 hours and then display it whenever required by the user.

Is there some basic which I am missing? Maybe there is some other way around.

Thanks for the reply. It would be good if you can help me out.

[msaxon]

Table issues aside, consider this:

• You are running Windows XP SP2
• And a CVI application
• And you are trying to load a 30MB file into RAM
• And put all that data into a CVI table control

In a system that only has 128MB RAM total! You shouldn't be expecting any sort of decent performance with that configuration.

The first thing you should do is to increase the RAM to 512MB. If that is impossible, then you will need to manage the amount of data loaded into RAM at any one time very carefully. In fact, you should only have loaded the data corresponding to what is actually displayed on the screen - anything more would be too much. Assuming your data file has fixed length records, random access into the file to pull out the required data on demand should be pretty easy (but admittedly harder than loading all of it).

[Maciek]

Personally, I don't believe that this could be a problem with open the file. I use applications which store data (in *.wav format) up to 1GB and more. There is no problem with reading file, not even with displaying it in plots. On the other hand in next application I use files about 10MB size (simple database) and I noticed that it is no problem to load all data into memory and perform some operations, the problem appears when I try to display results in table control.

If You use table only to display data maybe TextBox control instead of Table could help.
If not or if it is a hardware problem (maybe it is a stupid idea but...) You can try to update Table simultaneously with storing data into file.

(Sorry for my english)

[NavneethK]

As I understand:

30 MB single file size might not be an issue
but trying to load fully might be

Actually I have to display only some few records at a time. So I will try to only read the required records.

I will stop concentrating on making multiple files for the time being and concentrate on the above said approach.

I will come back within a few days.

I hope my understanding is right.

[NavneethK]

I will also look into the Table thing also. Thanks Maciek.