DSC 8.6.1 wrong timestamps for logged data with Intel dual core

[FermWorks]

Problem Description :

Our LV/DCS 8.6.1 application uses shared variables to log data to Citadel. It is running on many similar computers at many companies just fine, but on one particular Intel Dual Core computer, the data in the Citadel db has strange shifting timestamps. Changing bios to startup using single cpu fixes the problem. Could possibly set only certain NI process(es) to single-cpu instead (but which?). The old DSCEngine.exe in LV/DSC 7 had to be run single-cpu... hadn't these kind of issues been fixed by LV 8.6.1 yet?? What about LV 2009, anybody know?? Or is it a problem in the OS or hardware, below the NI line??

This seems similar to an old issue with time synch server problems for AMD processors (Knowledge Base Document ID 4BFBEIQA):

http://digital.ni.com/public.nsf/allkb/1EFFBED34FFE66C2862573D30073C329

Computer info:

- Dell desktop

- Win XP Pro sp3

- 2 G RAM

- 1.58 GHz Core 2 Duo

- LV/DSC 8.6.1 (Pro dev)

- DAQmx, standard instrument control device drivers, serial i/o

(Nothing else installed; OS and LV/DSC were re-installed to try to fix the problem, no luck)

Details: 

A test logged data at 1 Hz, with these results: for 10-30 seconds or so, the timestamps were correct. Then, the timestamps were compressed/shifted, with multiple points each second. At perfectly regular 1-minute intervals, the timestamps would be correct again. This pattern repeats, and when the data is graphed, it looks like regular 1-sec interval points, then more dense points, then no points until the next minute (not ON the minute, e.g.12:35:00, but after a minute, e.g.12:35:24, 12:36:24, 12:37:24...). Occasionally (but rarely), restarting the PC would produce accurate timestamps for several minutes running, but then the pattern would reappear in the middle of logging, no changes made.

Test info: 

- shared variable configured with logging enabled

- data changing by much more than the deadband

- new value written by Datasocket Write at a steady 1 Hz

- historic data retrieved by Read Traces

- Distributed System Manager shows correct and changing values continuously as they are written

[Ben_S]

Meg K. B. , 

It sounds like you are experiencing Time Stamp Counter (TSC) Drift as mentioned in the KB's for the AMD Multi-Core processors. However, according to this wikipedia article on TSC's, the Intel Core 2 Duo's "time-stamp counter increments at a constant rate.......Constant TSC behavior ensures that the duration of each clock tick is uniform and supports the use of the TSC as a wall clock timer even if the processor core changes frequency." This seems to suggest that it would be not be the case that you are seeing the issue mentioned in the KBs.

Can you provide the exact modle of the Core 2 Duo processor that you are using?

[andrew_johnson]

I'm working on this with Meg. The specific CPU info is:

General system:
       Windows XP Pro  version 2002  SP-3

General processor:
       Dell Optiplex 780
       Intel Core 2 Duo CPU
       E7500 @ 2.93GHz
       1.58GHz, 1.96Gb RAM

       Processor driver:
               Microsoft
               date: 4-1-04
               version: 5.1.2600.0
               digital signer: Microsoft Windows Component Publisher

               Details:
                       device instance Id:  ACPI\GENUINEINTEL_-_X86_FAMILY_6_MODEL_23_\0

               Driver details:
                       5.1.2600.5512 (xpsp.080413-2111)

Yes, it seems very similar to the AMD bug, but we are getting the problem with an Intel CPU. Timestamps get clumped together for a while, then show correctly, then clumped, back and forth. Problem resolved with sledgehammer approach of making the BIOS see only one CPU. We might try setting processor affinity on particular NI processes, once we determine which are the culprits

[Ben_S]

Meg K. B.  and Andrew Johnson,

Luckily, we've been able to reproduce the issue you are seeing on a computer here at NI with the same E7500 Processor. The AE who you've been speaking with about this issue will contact you via e-mail as we work with our R&D department to further investigate the problem. Thanks for the processor info!

[Ben]

... and someone will keep us updated?

Ben

(the first Ben )

[andrew_johnson]

Good to hear it's confirmed at NI, and that we're not crazy! Ultimately, we'll prefer identification of the particular processes which are unreliable on multi-core CPUs (at least this E7500) so we can use the imagecfg.exe utility to set their affinity to single-CPu, without the sledgehammer approach of modifying Boot.ini to tell the BIOS it has only a single CPU. Of course, a patched .DLL would be nice, too, but we understand the constraints involved in DSC development.

[NAKeel]

We have done further testing and we this was reported to R&D (CAR #115022) for further investigation.  We have reproduced the problem in LabVIEW 2009 as well as LabVIEW 8.6.1, and we have only seen the issue on the Intel E7500 processor.  We no longer have an E7500 processor available for testing, but we have verified that the boot.ini file change does solve the problem.  The two applications that we suspect would benefit from changing processor affinity are tagsvr.exe and nicitdl5.exe.

Nick Keel 

Applications Engineering

National Instruments

[Matthew_Williams]

Is it reasonable to assume that this will be an issue for 8.5.1 as well?

Any reason to expect that the issue is limited to the E7500 processor?

Is there an easy way to demonstrate or test a specific computer?

Thanks.

Matt

[FermWorks]

My simple test was writing known data at a known rate (repeating counter 0..100 at 1 Hz, written to shared var via Datasocket Write), then reading it back (via Read Traces). The incorrect timestamps were obvious, although maybe another processor would have more subtle problems. I don't know if the problem happens with any other flavors of processors; I can only say that we haven't seen it with dozens of other computers that are running our DSC-based application, but we haven't yet canvassed their processor types nor run the targeted test on them.

[Matthew_Williams]

Thanks Meg.  I'll give it a whirl.

Matt