File Synch problem with timestamp

[CoastalMaineBird]

Win2k, LabVIEW 7.0

I'm trying to do file synchronization between the data files I produced on the local disk, and a remote server. I am doing a LIST FILES on the local folder, and a LIST FILES on the remote folder. For each file, I do a GET FILE INFO, and look at the LAST MOD data (a TIMESTAMP value). For each file in the SOURCE folder, I match it up with the corresponding entry in the DESTINATION folder. I then compare the times, and if the SOURCE timestamp is greater than the DEST timestamp, then I delete the DEST file, and COPY the source file to the DEST.

There are a few files that fail this process, though, and I am puzzled as to why.

If I look (via Windows explorer) at the properties of one of these files, it says the LAST MOD date is 6:09:20. That is the case for BOTH the local copy and the remote copy.

However, the SYNCH logic above insists on copying this file every time it runs. When I add logging stuff to the synch process, it tells me that the local file has a date of 6:09:21, and the remote file has a date of 6:09:20, so it wants to copy it.

But after the copy, things are exactly the same - Windows shows the times identical, but LabVIEW shows them ONE second apart, and copies them AGAIN the next time it runs. This is true for about 6 files out of a folder of 50-60. I suppose I could allow for that one second, if I knew why it was there.

What's biting me here? If the file really had a MOD DATE resolution of 1 mSec say, then I could understand that maybe the PROPERTIES window truncates the value, and the TIMESTAMP does not. But even then, it ought to copy ONCE and set them the same...

The "Server" I am testing on is running OSX, 10.4.6, if that matters.

What am I missing?

[rolfk]

---

@CoastalMaineBird wrote:
Win2k, LabVIEW 7.0

I'm trying to do file synchronization between the data files I produced on the local disk, and a remote server. I am doing a LIST FILES on the local folder, and a LIST FILES on the remote folder. For each file, I do a GET FILE INFO, and look at the LAST MOD data (a TIMESTAMP value). For each file in the SOURCE folder, I match it up with the corresponding entry in the DESTINATION folder. I then compare the times, and if the SOURCE timestamp is greater than the DEST timestamp, then I delete the DEST file, and COPY the source file to the DEST.

There are a few files that fail this process, though, and I am puzzled as to why.

If I look (via Windows explorer) at the properties of one of these files, it says the LAST MOD date is 6:09:20. That is the case for BOTH the local copy and the remote copy.

However, the SYNCH logic above insists on copying this file every time it runs. When I add logging stuff to the synch process, it tells me that the local file has a date of 6:09:21, and the remote file has a date of 6:09:20, so it wants to copy it.

But after the copy, things are exactly the same - Windows shows the times identical, but LabVIEW shows them ONE second apart, and copies them AGAIN the next time it runs. This is true for about 6 files out of a folder of 50-60. I suppose I could allow for that one second, if I knew why it was there.

What's biting me here? If the file really had a MOD DATE resolution of 1 mSec say, then I could understand that maybe the PROPERTIES window truncates the value, and the TIMESTAMP does not. But even then, it ought to copy ONCE and set them the same...

The "Server" I am testing on is running OSX, 10.4.6, if that matters.

What am I missing?

---

It is a file system issue. NTFS uses something like a 100nSec resolution but historically the FAT file times have had a two second resolution. Apparently the Property dialog does convert the timestamp to a DOS time all the time before displaying it.
Your MacOSX system uses a completely different resolution (at least 1 sec but probably better). The network protocol does allow for high resolution transmissions and so the exact time gets transmitted but somehow your local system has a DOS limitation for the harddisk volume in question so that LAbVIEW returns the DOS time too for the local volume. The Explorer property dialog will always show DOS time only it seems, I had similar issues in the past but didn't really dig deeper, just accepted the 1 second difference.

Rolf Kalbermatter

[CoastalMaineBird]

Thanks for your answer, it certainly sounds plausible. But I see some evidence that it's not the whole picture.

historically the FAT file times have had a two second resolution. Apparently the Property dialog does convert the timestamp to a DOS time all the time before displaying it.

Looking at the logs, there is one failing case where the source time, as reported by LabVIEW, is 6:09:16 and the dest time is 6:09:15. If the dest time were always being converted to DOS time, that couldn't be true, could it?

In this case, the PROPERTIES dialog shows 6:09:15 for both files, which also implies that it's NOT converting to DOS time there - else the time displayed would always be an EVEN number of seconds.

ALL the failing cases are off by exactly one second. So I suppose I can check for SOURCE - DEST > 1, instead of SOURCE > DEST. In my case, it's not possible for the source to be legitimately modified only one second after the destination.

Still, I'd like to know why...

[CoastalMaineBird]

I should also report that it's the SAME 5-6 files that fail every time.

If I erase the dest folder, and run the synch operation, every file is copied. If I run the synch again, it's the SAME 5-6 files that are needlessly recopied.

So it's some property of the file itself.

[rolfk]

---

@CoastalMaineBird wrote:
Thanks for your answer, it certainly sounds plausible. But I see some evidence that it's not the whole picture.

historically the FAT file times have had a two second resolution. Apparently the Property dialog does convert the timestamp to a DOS time all the time before displaying it.

Looking at the logs, there is one failing case where the source time, as reported by LabVIEW, is 6:09:16 and the dest time is 6:09:15. If the dest time were always being converted to DOS time, that couldn't be true, could it?

---

Hmm, I guess this is something only someone with the ability to look both into the Windows and Explorer source code could really explain. The whole thing is a layered system of drivers, API layers and applications and each of them does do a few conversions. So it is possible that depending on the file type or location different code paths are executed with one having a DOS time conversion somewehre while others have not this problem.

And in my explanation I missed the fact that the remote file time seems always even seconds with the exception of above snipped so it seems that the network layer usually imposes the DOS file time while the local time is reported in real seconds at least by LabVIEW, which would indicate that you have probably NTFS on your HD.

I wouldn't really bother much more. Just be aware that the timestamp returned by LabVIEW for local files might also contain partial seconds so (local time - remote time) is not always < =1 but really < 2 or maybe even more complicated since I'm not sure conversion to DOS time is always truncated. It could also be rounded.

Rolf Kalbermatter

[CoastalMaineBird]

I missed the fact that the remote file time seems always even seconds with the exception of above snipped so it seems that the network layer usually imposes the DOS file time while the local time is reported in real seconds

Well, the second case above is not an extraordinary exception - In looking at the files, about 50% of them have odd seconds. I count 12 failures out of the 145 files (Each of 70 files is backed up to TWO places on the net). Of the failures, 8 have even seconds on the destination, 4 have odd seconds. So I don't think it's truncating or even rounding to the 2-sec DOS time.

I tried converting the TIMESTAMP from the FILE INFO to a DBL, and also to a U32 before comparing them - no difference at all.

I will build in some safety margin, since I know that my source can't change within 10 seconds, but I still don't know why....

[rolfk]

---

@CoastalMaineBird wrote:

I tried converting the TIMESTAMP from the FILE INFO to a DBL, and also to a U32 before comparing them - no difference at all.

---

This may highly depend on the LabVIEW version. While traditionally all timestamps have been uInt32 in LabVIEW they slowly changed that to DBL and then later to Timestamps. That didn't mean they did convert all the underlying functions at the same time, sometimes incurring conversions from the old uInt32 to the newer types internal in the LabVIEW nodes. But with every version more internal functions seem to have been overhauled and support for the higher resolution even in low level manager functions was added.
Especially with the complete renewal of all file functions in LabVIEW 8.0 there is a good change that file timestamps were also improved to make use of the new capabilities. On Windows NTFS and better file timestamps can have 100 ns resolution relative to 1.1.1601 and that is quite similar to what the NI timestamp has, other than that it references to 1.1.1904. So I wouldn't be surprised if at some point  they got or will get rid of the intermediate uInt32 timestamp conversion before providing the value to the user and then you can't assume that the timestamp will always be whole seconds.

Rolf Kalbermatter

[CoastalMaineBird]

So I wouldn't be surprised if at some point they got or will get rid of the intermediate uInt32 timestamp conversion before providing the value to the user and then you can't assume that the timestamp will always be whole seconds.

I understand that. I have followed the transition from U32 to DBL to TimeStamp myself.
Nonetheless, if I copy a file from here to there, I would expect it to have the same MOD DATE there as it does here, and it simply does not.

[jasonhill]

How about an alternate approach?  If the MOD time difference is within a small range (less than 2 seconds, for example) use a MD5 comparison.  If they are the same, the files are the same. If they are different, copy the modified one over.  Not the most elegant solution, but you work with the file system(s) you have.

[rolfk]

---

@CoastalMaineBird wrote:
So I wouldn't be surprised if at some point they got or will get rid of the intermediate uInt32 timestamp conversion before providing the value to the user and then you can't assume that the timestamp will always be whole seconds.

I understand that. I have followed the transition from U32 to DBL to TimeStamp myself.
Nonetheless, if I copy a file from here to there, I would expect it to have the same MOD DATE there as it does here, and it simply does not.

---

No, unfortunately that is not true. While LabVIEW queries the timestamp it does use a certain WinAPI call to do this. This call will at some point route through the files system driver that could be a real filesystem or a virutal one like a network. The precision of what this driver does provide is highly implementation specific.

When moving a file with the Move/Copy file function LabVIEW does not even deal with the timestamp at all but just tells Windows with a single API call to do the move. What attributes get transfered in a Move operation and to what precision is completely up to Windows and may use a different route or protocol element than the attribute query had used.

Rolf Kalbermatter