Create histogram from multiple files

[Mr. Spark Plug]

I have a bit of data, almost 1 gb worth in CSV format, that I am trying to create a histogram with.  I would like to know the most effective method to do this.  I am writing a script to pull in all the data, combine it into one channel, and then create the histogram.  Is this the best method, or should I create the histogram from each file, then combine it later?  Is there another method. 

Thanks. 

[Eric_L]

Hello Mr. Spark Plug,

If you are able to load the whole dataset into DIAdem, you will hove more options in creating your histogram compared running smaller batches. If you run into memory issues or the operation takes too long when running the whole dataset, I would then batch process the data.

Eric

[Brad_Turpin]

Hi Mr. Spark Plug,

Are you pulling in one scalar from each CSV file to form the big array that you calculate the histogram from?  Or are you pulling in an N element array from each CSV file and needing to concatenate those channels before running the histogram?  How many values will you have in your one big channel in DIAdem that you run the histogram on?  Do you know the X axis binning plan ahead of time, or would you prefer to automatically bin based on all the values in the big channel?  Is the VBScript you speak of a DataPlugin or a VBScrip that you run in the SCRIPT panel?

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Mr. Spark Plug]

Thanks for the reply.  I am registering just the channels I need from each CSV and using chnconcat command to stitch each one together in a new channel.  I create the histogram from this new channel.  It takes about an hour with this method, which is not too terrible, but its probably just a limitation of my hardware.  I am succesful in getting what I need out of the data, but as a learning experience, if you have any suggestions that might be more efficient, i'm all ears. 

Thanks. 

[Eric_L]

Hello Mr. Spark Plug,

One question that Brad posed related to the X Axis binning. If the X Axis binning was predefined, the creation of the histogram may complete more quickly. From your description it appears that your bins are based on the values in the big channel, if this is the case, you may be able to decrease the processing time by using predefined bins. Brad may be able to provide more insight to your question.

Eric