PXI-6683H is not syncing timestamps across multiple cards

[alright_meow]

In my system i have 3 PXI chassis that each are controlled by a PXIe-8880. Each PXI chassis has a PXI-6683H card. I have a Cisco IE 4000 setup as a PTP time source, which it connects all the 6683H cards together. Two of the chassis timestamps sync. One does not. They all show the same Grandmaster Clock source. They all show a time reference present. I am using MAX test panels to verify this information. I cannot get the third PXI chassis to sync the timestamp with the other two. Anyone have any ideas? Here are the three test panels open. You will notice the two on the right are synced, and the one on the left is not.

[KingWeasel11]

Assuming you've already tried turning it on and off again, maybe one port on the switch is configured differently from the other two? That's the only thing I can think of. I have a 6683H in the system I'm currently working on, but not the Cisco.

[alright_meow]

All the ports appear to be set the same on the switch. I have tried power cycling at different intervals and get the others to not sync either, but they always do, and the one is always off.

[KingWeasel11]

Ok. Have you figured out whether the problem follows the card or the chassis? That is, swapping the non-syncing card into a different chassis and seeing if it still shows the same problem.

[KingWeasel11]

Seems silly, but I've seen NI hardware do some silly things 

[KingWeasel11]

I would also try sniffing the network traffic via Wireshark. If I remember correctly, I think you have to have some sort of PTP client on the machine running Wireshark, or make the port you connect to on the Cisco promiscuous, otherwise no PTP traffic goes to that computer.

https://www.wireshark.org/#download

[alright_meow]

It appears that it is the card. I switched chassis, and now the chassis that was working is now not syncing.

[alright_meow]

Also have tried different cables, and different ports on the switch. It seems to be wherever the card goes, the problem happens. It is interesting to note every once in a while it will sync up after unplugging the ethernet cable and plugging it back in a few times. On a reboot of the chassis, it loses it.

[KingWeasel11]

That would drive me nuts. Seems like the path of least pain is to send it back and get a replacement, if that's possible. Wireshark might be able to tell you something, but probably only that it isn't syncing, which you're plenty aware of by now.

[alright_meow]

So we ended up having to move stuff around on our test rack, and was debugging another problem. We noticed after hooking up a monitor to our PXI chassis that it was getting an error "protocolelv.cpp." I didn't get an image of the error. Stuff got rewired on the daq cards and magically that error went away, and it seems that the card always syncs. I've manually changed the time to be totally different, and the card will always sync. Now i can't get it to do what is was doing before. So not really sure what the solution was.