DAQmxGetWriteTotalSampPerChanGenerated returns wrong number of scans generated after 25 minutes

[Mike_TheKraken]

Some of it is exactly the 2^32 issue up till now and that is why TMW contacted you. I am at least 1 person who brought this up to them. The task which is shutting down is signal generation, D/A output from one channel. For what ever reason, it seems to be worse for running the system with a few plots. When the only thing that is running is the signal generation, the system seems to be able to run, if left alone, for 13+ hours.

PC is an i3 core running at 3.1GHz, 8GB of memory, radeon graphics card with 1GB of memory, Win 7 64bit, ML - 2012b 64b.

32 bit versions can't handle this without shutting down realatively quickly.

D/A output sample rate 1e6 (1MHz) creating a 1s queue of data and repeatedly generating it for an hour.

Two or three A/D channels input running at 500e3 (500KHz) recording a sample period of 2e-3 (2ms) single ended signals.

System seems to generate signals fine for approximately 2400 seconds, sorry I don't have the exact time period, but will next time.

I am confused as to why it is necessary to "count" these output samples at all? There is always a limit to the number of "fingers" and even a PC has only so many of fingers. It seems like they'd just be generated and forgotten in my simple picture.

This is the error which reports to the window (-200290, which is the same as reported before installing 9.6):

The generation has stopped to prevent the regeneration of old samples. Your application was unable to write samples to the background buffer fast enough to prevent old samples from being regenerated.

To avoid this error, you can do any of the following:

1. Increase the size of the background buffer by configuring the buffer.

2. Increase the number of samples you write each time you invoke a write operation.

3. Write samples more often.

4. Reduce the sample rate.

5. If your data transfer method is interrupts, try using DMA or USB Bulk.

6. Reduce the number of applications your computer is executing concurrently.

In addition, if you do not need to write every sample that is generated, you can configure the regeneration mode to allow regeneration, and then use the Position and Offset attributes to write the desired samples.

Task Name: _unnamedTask<A5D>

Status Code: -200290

and then:

Warning: Error occurred while executing callback:

Error using daq.internal.StateNeedOutputData/processHardwareStop

(line 130)

Internal Error: Unexpected operation processHardwareStop occurred

in state NeedOutputData.

> In Channel>Channel.onCustomEvent at 378

  In Channel>@(source,data)obj.onCustomEvent(data.Type,data.Data) at 304

*************************

Here is some memory information at the time of signal stoppage:

    Physical Memory (RAM):

        In Use:                             2136 MB (   8586e000)

        Free:                               6038 MB (  1796ae000)

        Total:                              8175 MB (  1fef1c000)

    Page File (Swap space):

        In Use:                             2426 MB (   97a0e000)

        Free:                               6771 MB (  1a733f000)

        Total:                              9197 MB (  23ed4d000)

    Virtual Memory (Address Space):

        In Use:                             3821 MB (   eed56000)

        Free:                            8384786 MB (7ff1128a000)

        Total:                           8388607 MB (7fffffe0000)

    Largest Contiguous Free Blocks:

         1. [at   18000b000]             8377939 MB (7fd65375000)

         2. [at 7fefffb1000]                4086 MB (   ff65f000)

         3. [at   13f74d000]                1032 MB (   408b3000)

         4. [at   102e10000]                 969 MB (   3c910000)

         5. [at 7feea40c000]                 198 MB (    c614000)

         6. [at    e8690000]                 129 MB (    8120000)

         7. [at 7fee5bf1000]                  36 MB (    243f000)

         8. [at    69732000]                  33 MB (    217e000)

         9. [at    7d3eb000]                  27 MB (    1bf5000)

        10. [at    761ca000]                  19 MB (    13e6000)

                                            ======= ==========

                                         8384472 MB (7fefd863000)

***************************

Thanks Dan, this has been extremely frustrating. I know NI has equipment which will generate samples forever, so this bugs me.

Can I also ask you if you think a PCI based version of the USB-6351 would eliminate this problem? At this point, from going through this thread, I don't think it would.... I'm desperate for a solution.

[Mcdan]

There was one other 32 bit roll over issue that I'm trying to remember... It had to do with using the offset property associated with the DAQmx Read or DAQmx write property node.  Do you use that property?

I am confused as to why it is necessary to "count" these output samples at all?

There are two reasons to do this.  1) People do use the DAQmxGetWriteTotalSampPerChanGenerated property to know exactly how many samples the DAC converted.  With buffered tasks, this really can't be inferred from then number of samples written (via DAQmx Write) because data can be regenerated from the buffer or in device memory when the task is stopped. 2) When dealing with the buffer, we need to allow for users to specify where to read/write.  To do this, we essentially use 'sample number' as an offset.  So if you have a buffer of size 10 kS, you can specify to write at offset 100 k, and we'll figure out how this maps back to the actual physical buffer, and when hardware has generated enough samples that we can overwrite the existing data with something new.

The CAR that this thread created, should have fixed the 32 bit roll over with respect to the first use case (counting the number of samples which actually got output by the DAC).  If you are using the offset properties, I will dig around a bit more and see what I can find with respect to the 2nd use case (including a summary of such an issue if found, and a look at which hardware exhibits this and which doesn't).

Hope that helps,

Dan

[Mcdan]

To follow up on my last post, here is the issue I was thinking of:

http://forums.ni.com/t5/Multifunction-DAQ/X-Series-PCIe-6321-crashes-DAQmx-program-M-Series-PCI-6221...

Short version: Offset properties in DAQmx are signed 32 bit properties.  This limits the usefulness of the offset properties (if you want to use offset relative to a fixed point in the buffer for long running acquisitions/generations).  There was an additional issue with PCIe/PXIe X Series which was causing roll overs before we reached 2^31.  This premature roll over was fixed in DAQmx 9.5.

Dan

[Mike_TheKraken]

I'm not sure what that offset property does. I'll have to ask TMW if that property is somehow being used. If it is a DC offset of a sinewave file, no, I'm not. All I do is generate a sinewave and output it.

I've tried many drivers, etc. This time a new graphics driver. The system ran till 2160.5 seconds before shutdown. I'll keep better track of the shutdown times, but they are all in that range.

As for the "count" reply. I'll have to let that soak in a bit, but thanks for the explanation.

cheers,

Mike

[Mcdan]

Mike,

Because you get the 'The generation has stopped to prevent the regeneration of old samples' error, it appears as though you're not just writing your waveform once and allowing DAQmx to continually regenerate that from the buffer.  This means that your waveform must be continually re-written.  If this is done using the offset property, and the offset is relative to the first sample I could see where you would get into a roll over at 2^31.  I'll go ahead an try to create such a scenario here, and see what happens in this situation (what error code, number of samples, etc).

Let me know what you hear back from TM, and I'll let you know what my experiment shows.

Dan

[Mike_TheKraken]

Dan, that is right. We have to be able to change frequency and amplitude often every couple of seconds, so the only way I found to do it was to change the generated waveform. The TM guy, Wael, is out sick right now and he's been the man for this... Tis the season.

Much thanks for your help. Will be great to see the results.

[Mcdan]

Mike,

Here's what I found (ran these tests using a USB 6366 with DAQmx 9.6.1, which should behave the same as your USB 6351):

Setup 1 - Rewrite data, set buffer size to 1 MS, set DAQmx.Write.RelativeTo property to 'First Sample'.  Each iteration of a while loop, write write data to i*Waveform size (in my case 100 kS).  On write 21,475 I received error -200287 ("Attempted to write to an invalid combination of position and offset. The position and offset specified a sample prior to the first sample generated (sample 0).").  This seems like expected behavior since writing a value bigger than 2^31 results in a large magnitude negative signed 32 bit number.  I did not receive error -200290 that you have seen.

Setup 2 - Rewrite data, set buffer size to 1 MS leave DAQmx.Write.Relative to property at it's default value (Current Write Position).  Each iteration of a while loop, write data (again 100 kS).  This has run all night and is still running.  I am currently over 60 GS output.

Neither of these attempts match the behavior your seeing, so we need to figure out what's different with your setup.  One thing we can try, is using NI I/O Trace.  This should log all of the properties that are set on DAQmx, as well as all calls into the API.  You can launch this from Measurement and Automation Explorer, under the Software listing.  If you want to log your application from start until failure, I'll peruse the log file and see if anything jumps out at me.

Dan

[Mike_TheKraken]

Dan,

I'll work on it.

thanks

[Mike_TheKraken]

Tried loading a file, but it seems to be too big. I'll have to do some screen shots.

[Mike_TheKraken]

Here are some screen shots of the system running. I've added a big GPU card and that seems to unload the system allowing it to run for 60 minutes and not stop, as opposed to a 40 minute run time. Not sure why the system loads up, having trouble finding the offending parameter when watching the CPU/GPU usages to see which is over burdened. These lines are all calls to the driver and replies from the driver for the DAQ? I think about 17:47:05 the DAQ stopped producing samples, but I don't see any red conflict messages, if my interpretation is correct.

I still am contemplating buying an equivalent version of the USB-6351 in a PCI bus configuration, hoping this provides extra bandwidth. I assume my code won't change, it'll just have to be pointed to the new card.

I have a couple more screen captures to add.