Delete a lot of data

[Mr. Spark Plug]

I have a LOT of data.  At least 10,000,000 data points across 32 channels (well, it's lot to me).  I need to delete 100,000 data points toward the beginning of the data file due to a data collection error.  I'm using the DataBlDel command. So far, I'm at 5 hours waiting for this command to complete.  Anyone know of a faster way to remove this data? 

[Walter_Rick]

Hello Mr Spark Plug,

I can’t imagine what happens in this case. I’ve done the following test:

• Just loading the “Noise data” channel group (each channel contains 325 000 values)
• Copy this channels 7 times (that’s 35 channels with 325 000 values each)
• Use the command “DataBlDel” to delete the first 100 000 values of each channel

I was not able to measure the command duration with the VBS timer. I’ve tested with DIAdem 2011 SP1. To get an answer for your 5 hour command duration we need more information of the environment where DIAdem is running.

Greetings

Walter

[Mr. Spark Plug]

Hello Walter, the computer will handle channel lengths of 325,000 data points without an issue.  The channel length I'm dealing with is 10,600,000 and I think that's part of the problem. I think the data is getting deleted easily, as I can see the progress bar increment quickly. After that, the computer sits for quite a while at high CPU usage.  I guess this would be when the rest of the data AFTER what I have deleted is bieng moved to fill in the deleted data.  That's my guess. 

Anyway, the CPU specs:

Windows 7 32-bit

2.8ghz dual core CPU

4 gig memory

80Gig Hard drive about 75% full. 

So, if my guess is really what is happeninge, is there a more efficient way to remove the data? 

[Walter_Rick]

Hi Mr. Spark Plug,

Yes, 32 channels with round 10 000 000 values each (and 8 byte per value) is all in all more than 2.5 GByte. A 32 bit application is only able to manage round 2 GByte minus the memory for the application itself. In DIAdem such data’s are managed then virtual on the hard drive. The speed then depends on the PC performance, fracmentation of the drive etc.

To solve that problem I would split the data in let’s say 3 pieces by loading a reduced number of channels, delete the certain area and combine the data again by loading the three value reduced pieces.

Greetings
Walter

[Mr. Spark Plug]

Thanks for the Reply Walter.  Would you recommend using 64-bit for Diadem, then?  Are there any downsides to using 64-bit for Diadem? 

[Walter_Rick]

A 64 bit W7 can help, it gives a 32 bit application the whole 2 GByte memory. But in the end it is – in your case – not possible to load the whole dataset into the memory.

Greetings

Walter

[Walter_Rick]

Here is another option. You can use the command DataFileLoadRed to just load the data you are interested in by giving a starting point for the import like:

DataFileLoadRed("32with10mValues.tdm","TDM","","IntervalWidth",1,eInterFirstValue, 100000, 10000000)

Greetings

Walter

[Mr. Spark Plug]

Hello Walter, thanks again for the information.  I assume by your message above, that Diadem is not available as a 64-bit application.  Can you confirm if this is the case? 

[Brad_Turpin]

Hi Mr. Spark Plug,

DIAdem 2011, the current version, is only available as a 32 bit application.

Brad Turpin

DIAdem Product Support Engineer

National Instruments