TSN Synchronized Analog and Digital Tasks

[DHerron]

LabVIEW 2018 SP1

DAQmx 19.1

cRIO 9048/49

• AI: 9220, etc.
• DI: 9437

FD-11601

We have a system that consists, in its most simple form, of a cRIO with an AI module and DI module and an AI FieldDAQ.  In its most most complicated form, it's a full cRIO with AI and DI modules -> daisy chained to a 9049 cDAQ full of AI and DI modules -> daisy chained to 10+ FieldDAQs (both Temperature and Analog versions).

For this reason, we have multiple AI tasks that rely upon the FirstSampleClock.When property to synchronize all of our AI modules/FDAQs/tasks, and it does a pretty good job at getting everything started at the same time and maintaining synchronization between the sample clocks.  However, we're having some issues getting our DI task to start at the same time.  The FirstSampleClock.When property does not appear to be supported by DI tasks (Error -200452), so we've been using the DAQmx Start Trigger (Time) and feeding it the same time value that we give to the FirstSampleClock.When property for our AI tasks.  Unfortunately this doesn't give us as tight of a start time synchronization as it does between all of the AI tasks.

Is there an approach that would allow us to continue to use time based synchronization, but with some tighter tolerance between AI and DI?

[mdwebster]

I did a TSN system last year and did the same thing as you with FirstSampClk.When (and FirstSampClk.Timescale = HostTime), but didn't have to deal with any DI channels.

Hopefully someone from NI can weigh in, because looking at the "Start Time" variant of the trigger VI, I would swear it was just a wrapper for those two property nodes.

So you're saying the DI tasks on the the cRIO and the cDAQ both complain about accessing those property nodes, or do you have a single DI task with channels from both devices in it?  I wound up having to go with a task per device for some reason I can't recall at the moment.

I'm also curious if you get any synchronization issues.  I wound up having to ignore out-of-sync errors as this was a system left up 24/7 and I would go out-of-sync for ~1 minute at a time up to 4 or 5 times per day and mine was a much simpler configuration than yours (3x cDAQ chassis sitting within 3 feet of each other).  We would also get flat out DAQ task crashes once or twice a week and I had to write recovery code to restart the acquisitions.

Oh, one other thought, since they are on the same chassis as your AI channels, you could gang them to the AI SampleClock.  Then they would be in perfect sync with the AI channels on the same chassis.

[DHerron]

Very good questions.  Yes, they both complain about that property; regardless of whether it's on the cRIO chassis or the cDAQ chassis.  Anecdotally, we do encounter some synchronization issues sometime but since it's more critical that it continues running than periodically running outside of our ideal timing scenario, I just set SyncUnlockBehavior to Ignore Lost Sync Lock and log the warning through a separate process that monitors the synchronization health.

So I do specify the same sample clock for the AI as the DI, but I think the issue stems from the actual use of the FirstSampleClock.When property.  It's my understanding that even if you provide it a precise time, what it actually does is negotiate amongst all of the specific AI tasks and takes into account startup time for DSA, pre-filter time, etc. so that the time it actually starts is some beforehand-unknown period of time after (or before?) your truly desired time.  My guess is the DAQmx Sample Trigger (Time) doesn't take part in that negotiation and thus starts when you tell it to.

I find it interesting you had crashes over long term durations; we've encountered the same.  We had to do the same thing you did, which essentially restarts the failed tasks with start times equal to exact multiples (usually 2) of the sample size * dt + t0 of the still-working channels.  We lose a little bit in our synchronization, but it usually recovers well.

[mdwebster]

Yeah, I'm not entirely sold on the stability.  I like the idea, but sometimes I yearn for the relative simplicity of the old PCI DAQ cards.  Same thing on the SyncUnlockBehavior.  It's super weird though, it went the first several weeks without a single one, then several in a day.  Now it runs sometimes for a week without a single one, or 5 or 6 a day.  No apparent rhyme or reason to it.

What I was getting at with the sampleclock is that if you wire AI\SampleClock into the source terminal of the DAQmx Timing.vi of the DI task, you don't have to set up any actual timing of the DI (even the sample rate input just sets the dt if you're pulling off waveforms in the read).  The 'right' way to do it is to get the AI tasks SampClk.Term since the cDAQ can use more than one timing engine.

This will only work if you're using separate tasks per chassis as you need to wire the sampleclock for the AI task on chassis 1 to the sampleclock source of the DI task on that same chassis.  I never was able to get the everything in one master task thing to work.  The SampleClock sharing is kind of an older way of syncing various tasks together as 'slaves' to one master task.  The SampleClock only runs when the AI is actively acquiring samples, so start all the 'slave' tasks first, then the AI task and when the AI task starts acquiring all the slave tasks will follow suit.

The TSN synchronization you're talking about is more or less correct, but only applies to AI channels and only applies to when they all start. My understanding is all the TSN aware chassis have their internal clocks synchronized and the DSA pre-filter stuff just adds or subtracts from that time by the appropriate increment to make those channels start at the right point.  Which might actually be an issue, come to think of it, if the AI modules on your chassis has prefilter / DSA startup time to consider. 

Thinking back to my project with this, I think I had some DSA cards, but they were in yet another task from the SAR cards.  So if you can slave the DI tasks to a SAR-only AI task on the same chassis, I think it would work with no problem.