07-07-2015 09:29 AM
I ran your file with a 430MB file and these are the returned load and save times.
This save time is a lot faster than my original method but as you can see it still takes over half an hour.
Here's some further information.
DIAdem: 14.0.1Service Pack (32-bit)
USI: 14.0.2
As for the File information, the channel number and length varies from file to file. Usually there are anywhere from 10 to 30 channels.
07-07-2015 10:59 AM
I tried to reproduce locally with a NAS and watching your numbers I strongly
believe that it is related to the up and downstream of your datashare connection.
20000 kbit/s
1390 kbit/s
Be aware that I slightly changed the example because the
DataFileMove method works different than I expected.
I tried to create an script to reproduce.
In my system with a NAS and a 400MB file.
Channels: 30
Size : 400,02 MB
Save : 14,9813 s
Load : 13,1988 s
Option Explicit 'Forces the explicit declaration of all the variables in a script.
dim targetPath : targetPath = "\\192.168.2.2\public\out.tdms"
dim filesizeinmb : filesizeinmb = 400
dim nrOfChannels : nrOfChannels = 30
dim nrOfValues : nrOfValues = filesizeinmb * 1024 * 1024 / 8
dim chLength : chLength = CLng(nrOfValues / nrOfChannels)
MsgLineDisp "Generate data"
LoopInit
data.root.clear
dim grp : set grp = data.Root.ChannelGroups.Add("group")
dim i : for i = 1 to nrOfChannels
loopinc i * 100/nrOfChannels
Call ChnGeoGen("/ch" & i,i,i*100,chLength,"")
Next
loopdeinit
MsgLineDisp "Delete fie if already exists"
if FileExist(targetPath) then
DataFileDelete targetPath
end if
MsgLineDisp "Save to target file"
stopwatchreset 1
SaveTdmsLocalAndMove targetPath
dim dT1 : dT1 = stopwatch(1)
dim realFileSizeInMb : realFileSizeInMb = DataFileSize(targetPath)/1024/1024
MsgLineDisp "Load file"
data.Root.Clear
stopwatchreset 1
LoadTdmsFileImediately targetPath
dim dT2 : dT2 = stopwatch(1)
if FileExist(targetPath) then
DataFileDelete targetPath
end if
MsgBox "Channels: " & nrOfChannels & VBCRLF &_
"Size : " & FormatNumber(realFileSizeInMb,2) & " MB" & VBCRLF &_
"Save : " & FormatNumber(dT1, 4) & " s" & VBCRLF &_
"Load : " & FormatNumber(dT2, 4) & " s"
sub SaveTdmsLocalAndMove(byVal targetPath)
dim fileType : fileType = UCase(right(targetPath, len(targetPath) - instrRev(targetPath, ".")))
dim targetPathStr : targetPathStr = left(targetPath, instrRev(targetPath, "\"))
dim targetFileName : targetFileName = right(targetPath, len(targetPath) - instrRev(targetPath, "\"))
dim tmpTargetPath : tmpTargetPath = TmpDrv & targetFileName
call DataFileSave(tmpTargetPath, fileType)
call DataFileMove(tmpTargetPath, targetPathStr)
end sub
function LoadTdmsFileImediately(byval srcFile)
dim oldBulkDataLoadingMode : oldBulkDataLoadingMode = BulkDataLoadingMode
BulkDataLoadingMode = eBulkDataLoadingImmediately
on error resume next
DataFileLoad srcFile
LoadTdmsFileImediately = err.number
on error goto 0
BulkDataLoadingMode = oldBulkDataLoadingMode
end function
07-07-2015 01:04 PM
Running that exact code here are my reults,
If I am correct this would imply that my issues are related to the up and downstream of the connection.
07-08-2015 01:13 AM
It is hard to believe that you can save 400MB in 1.6 s
to a network share. So I am not sure what we are seeing here.
The read througput looks like 100 mbit/s network.
Would it be possible to share one of the tdms files.
Maybe upload it zipped to ftp://ftp.ni.com/incoming/
07-08-2015 01:00 PM
The 1,654s is 28 mins for the 400 MB file. I'm using american notation for my , and . in the numbers. sorry if there's any confustion there.
Also unfortunately I can't share the data due to company policies.
07-09-2015 12:58 AM
Yes the number imply that the saving time is just related to the upstream.
The DataFileMove method uses basic Windows functionallity.
So copy the file with Explorer should use the same amount of time.
What changes do apply to the file?
07-09-2015 07:57 AM
All I do in this step is changing the existing file name. However in the future it will be adding channels, properties, etc. for data analysis.
07-09-2015 09:07 AM
Rename can be done using
DataFileRename
potentially without having to move the file completely.
If you just want to edit/add properties.
CreateDataFileHeaderAccess
might help you without having to read the file completely.
But if it comes to add new channels ... you have to move the file and writing will be slow.
07-09-2015 09:09 AM
ok, thank you so much for all of your help!