storing large amounts of data

[Sirin]

I have a bit of a problem, and I wanted to see if anyone had a better idea on how to solve it than I did. I am using LabVIEW Base 6.1, so don't have access to any toolkits or the like.

I have a program that generates reports on a customer's process; these reports are generated somewhere around every 15 minutes. I need to store these reports to disk so that they can be viewed at a later date. In the past, when I have created programs that do these sort of things I've run into issues with having many files in one directory slowing down my program as it accesses files. (The process generally runs 24 hours a day, 7 days a week, so that is 384 reports a day.

To solve this, I have been saving the files into subfolders based off of date. However, this makes it more difficult to load the report files as the user needs to be able to pull up a report based off of an ID number, not a File Dialog box. One solution I had been considering was making a "config" file that keys each ID number to the path of the report; I am worried about this file becoming too large and slowing down the system though.

Does anyone have a better solution?

[Henrik_Volkers]

If you store the files in subfolders based on their IDs, where is the problem to open them from that subfolder based on the ID?

If your first try to load the file in that subfolder fails, the user entered a not valid ID and then you can do a file dialog.

[falkpl]

You can write to a new file at durations which yield a suitable amount of data per file (lets say every 15 minutes), and place groups of files in folders of a larger size, such as each day, and all the days goes in to a data directory, with one additional text (or ini) which serves as a table of contents, so you write a file and append an entry to the table text file.  I have done this and had little trouble managing data, I used the system clock to name the file, so each file got a unique name since no two files are written at the same time (although there could be an issue when the counter overflows in 20-30 years).  To retrieve data look up the location of the data based on your table of contents, and load the data file in manageable chunks.  You can also add information about the data file in this table similar to a database.  I have recently been implementing this table in a database since my data files have 10-20 bits of summary information, this way I can search the data using SQL.  The size of the files and load times will determine how often to write a new file.  I have done this for data collection before, not for 24-7 operation but for large data collection where single data files can bog down the system by being opened in memory.

Paul

[EVS]

Sometimes I also use the index files. So I can store aquired data at any place. I have to append corresponding information into index file. As a rule there is not only path or location information but also some additions like timestamps.

The other way is historical database like Citadel Database. National Instruments LabVIEW DSC https://www.ni.com/en-us/shop/product/labview-datalogging-and-supervisory-control-module.html provides historical database. For example we have developed succesfully working system for wind turbine power plant monitoring using LabVIEW DSC.

[DFGray]

Citadel is probably the method to use if you want secure data storage. If you want to do this with no extra cost, consider HDF5. The toolkit is pretty low level, but it is a hierarchical file system designed to store terabytes of arbitrary scientific data. I haven't found much it can't do. Your file size limit is the size of your disk (64-bit file pointers - 128-bit if you compile it right). You can get full information at the HDF5 website. You can download a toolkit to use it from LabVIEW by looking at this article - Can I Edit and Create Hierarchical Data Format (HDF5) files in LabVIEW?. Ignore the top level VIs, which have been obsoleted by NI-HWS, and go for the lower-level HDF5 ones. Make sure you read the documentation on the HDF5 website before attempting this. Flush to disk when you finish each write so that you don't lose anything in case of power failure.

A word of warning. HDF5 is not thread-safe under Windows. Make sure you do not call the library from more than one place at once (very easy to do in LabVIEW).

One final comment. HDF5 is very low level and requires a bit of time to learn. If you choose to go this route, allow yourself at least a week to get your initial functionality. If you continue to work with it, you should be comfortable in about a month. It's worth the effort.

[Sirin]

Thanks for the information, I will take a look at HDF5 and see if I can make it work for my application.

[mikeporter]

The best solution: Save your data in a real relational database. There are excellent Open Source database management systems available (or you can use the one built into Windows) and you don't need any toolkits to make the connection. If this sounds interesting I can provide more information.

Mike...

[Sirin]

I was assuming that what I really needed was a database; I've never really had any training with databases or the like however. Unfortunately, there is not a lot of time left in the project to figure out how to use one at this time. It may be a feature that I try to add at some point in the future; but for now I think what I am going to do is what someone above suggested:

I'm going to still sort the reports into folders by date, but instead of having one master file I will have multiple master files based off of the first few digits of the report's identification number. Hopefully this will make sure that I don't end up with one huge file, as well as making sure I don't end up with a folder with a million files in it. (The ID numbers are not sequential.)

[Santiago D]

Although it seems you already have a solution to your problem, if you are looking for a way of doing your reporting and logging automatically in the future you should take a look at DIAdem (www.ni.com/diadem/).  It is an excellent program for analyzing, reporting and logging huge amounts of data.  Furthermore, it gives you the option of doing whatever you would do manually programmatically.  Check it out!

Santiago D