6561 TClk Synchronization Problem

[bmann2000]

Hi George,

I've spent about 16-hours this week messing around with different permutations of the code.  The only functioning code I have as a baseline is the NI Shipped Example Continuous Acquisition - Stream to Memory.vi.  This VI will function correctly with one 6561 card.  The main differentiating feature of this example is that number of samples are not configured.

Do you know why the LabVIEW 8.5.1 and 2009 shipped example are different ? see attached images. 

[bmann2000]

I've looked at different ways to Fetch and stream to TDMS, when moving from one card to six I noticed you extended the shipped example for 1 card and indexed through each hardware reference, fetching for each instance.  I've sucessfully fetched data from 6-cards when configured for a Finite Acquisition, but when I move over to a Continuous Acquisition I encounter problems with either the acquisition not starting, or the Fetch backlog Property Node returning 1000 max.

Do you have access to any tested examples for more than one card, continuous acquisition ?

I've attached my latest code containing some of the combinations tried out.  I've just been interleaving the 6-cards to TDMS file to keep things simple, they're easy to de-interleave when reading back to strip out the zeros post-processing.  The main foucs is just to get all six cards streaming.

Your posted code also attached with results.

[gt3000]

Hello Brian,

I have been looking through the documented changelogs between versions of example code that is shipped with LabVIEW and HSDIO drivers.  

The reason the VIs are different is that as of the driver NI-HSDIO 1.6, the R&D team added a new attribute to the software which allowed a new 'continuous memory mode'.  This mode made the need for the older Software Reference Trigger redundant, which is why it was removed from the example.

This does highlight, however, that PXI system is currently on NI-HSDIO 1.5.3, which officially support up to LabVIEW 8.5.1.  This is something to keep in mind through this support issue, as I am developing in LabVIEW 2010 and NI-HSDIO 1.7.4.

At which point have you started working with LabVIEW 2009?  Has all of your testing been in LabVIEW 8.5.1 or 2009?  On the same PXI? 

Regards, 

[gt3000]

Hello bmann2000,

The example that was developed and posted was eventually tested by a colleague on a PXI system with two PXI HSDIO cards, and it did not return any immediate errors while logging to TDMS, as well.  I will gather that systems configuration as soon as possible for comparison.

Regarding your own testing, the default Front Panel values are meant to be adjusted to match your own requirements.  

Did you received errors at much lower rates?

Do you see the same errors whether it is two cards or six? 

Regards, 

[bmann2000]

Hi,

On the versions, all the development and testing is with the PXI development PC running LabVIEW 8.5.1 and HSDIO 1.5.3.  I also have LabVIEW2009 and 2010 on my desktop, so I had a look at the shipped examples on my desktop for comparison/clues on what might be the problem.  Is it also worth investigating what version of Firmware is installed on my 6561 Cards?

The VI posted 06-24-2011 01:22 PM UKSupportContinuous Acquisition - Stream to Memory.vi 93 KB are the results from the PXI development system.  It's possible some of the VIs I've previously posted have been saved as 2009, as I've moved the VI from the PXI as version 8.5.1, then reviewed on my desktop before posting, and unintentionally saved as 2009, I'll make sure in future I keep at 8.5.1 when posting.

It's encouraging that you've successfully achieved continuous acquisition on two HSDIO cards at your end.  The VI quoted above had the acquisition rate set at 2MHz, I'll experiment again today with 2 cards at much lower rates.

Regards,

Brian

[bmann2000]

Hi George,

Made progress, the example you posted does work for 2 cards and 6 cards, but only at low sample rates.  I understand that Fetching as data type U16 is the best to acheive good throughput, at the moment I'm using WFM, this is because my algorithm to strip out the unwanted zeros is coded using the WFM format, this code could be changed once I'm confident that Continuous Acquisition of U16 will acheive the required spec for the acquisition system.

Can you give me any insight into want's happening in the background with the Fetch>Queue>Write process?  how can we determine the optimum value for Fetch Size and what is the bottleneck that's preventing Read-Write at 2M(x6) to TDMS file?

VIs attached for reference.

[gt3000]

Hello Brian,

I am glad to hear that the example has worked.  Generally, for HSDIO applications, the physical bottlenecks are:

1. The onboard memory per channel.  Your card is the 16Mb/ch version.
2. The PXI backplane throughput.  This is limited by the PCI standard with a theoretical 132MB/s for one card. However, this implies only one PXI card only delivering data in one direction.  Multiple cards delivering data at the same time, PCI bridge multiplexing, and real world effects greatly reduce this to about 65MB/s in practice.   

Rough calculations equate to:

2MHz x 16bits = 32Mb/s/ch

32Mb/s/ch x 16ch = 512Mb/s/card = 64MB/s/card

64MB/s/card x 6cards = 384MB/s

The numbers above are quick math.  This does not mean your application is impossible.  

If the cards above also have the 16Mb/ch memory, which would mean that the data must be fetched from the channel at least twice a second before it fills up.  The above numbers are set for U16.  The Waveform datatype will be larger per record.

1. What is the maximum aquisition rate that the application actually requires?
2. What is the highest value for rate typed into the Front Panel before the application stops working?

Also, additional gains in efficiency can be made by writing to a Binary File, rather than a TDMS file, but that would change the architecture I had provided.  This can be a step for a later time.

Regards, 

[bmann2000]

Hi,

Being a Digital card, I'm assuming each U16 value Fetched gives you all 16-channels, xxxx xxxx xxxx xxxx = 16 bits (or 2 bytes).

For me, this makes the calculation 2Mx16bits= 32Mbits/s/card, 8 bits in a byte = 4Mbytes/s/card

Each card has memory quoted as 16Mbit/ch = 16Mbytes memory per card.

16/4 = 4 seconds of data can be stored on a card.  Therefore you require 0.25 Fetches per second to prevent data loss.

6cards acquiring 4Mbytes/s totals 24Mbytes/s to be streamed across the PXI bus.

The application requires 2M sampling as we're looking for pulse widths of 1uS.

As for the highest sample rate before errors, I've not experimented that much, it depends on the size of the Fetch.  I'll have another look this morning and get back to you.

Please confirm if you agree or disagree with the calculation above, 24Mbytes/s versus 384Mbytes/s will make a big difference.

[bmann2000]

If I sample at 2MHz, 500k samples per Fetch, 60M samples total, the VI runs until the total samples reaches around 22M, at which point a windows error box appears "Not enough memory".  If I open Task Manager I can watch the available memory decrease until the error message appears.  It's not clear to me why the controller memory is filling up as I'd imagine the memory usage should drop back down following each write to TDMS file.  This indicates that I am successfully streaming to TDMS at 2MHz, for 10 seconds.

When selecting much lower sample rates I experience a mis-match between Fetches and TDMS write operations, it appears that I've lost data somewhere in the queue/dequeue code.

Sample Rate 100k, samples per fetch 20k, total number of samples 500k, this works fine with Fetches required equal to number of TDMS writes, fetches required =150, TDMS write count=150.

If I move up to Sample Rate 200k, samples per fetch 40k, total number of samples 1M, I get a mis-match between Fetches and TDMS writes, fetches required =150 , TDMS write count=142-150, over several runs it might acheive 150 once.

I think these are two separate problems.  I'm still not clear on how to calculate the optimum number of samples per fetch.

[bmann2000]

Hi George,

We've now reached the point where the project are no longer prepared to wait on a possible solution that hinges on a software configuration issue, if that's what we have.  We need to know if the hardware we've purchased will meet the spec outlined in the first post of this thread (27th May). 

We can feed a GPS once per second pulse to a spare channel on each card to allow us to sync up the pulses post processing, however, if that's our way forward, to meet the test duration requirement, we would need to perform Continuous-Acquisition of Multiple-Records on six cards simultaneously , we do not have example code to achieve Continuous-Acquisition of Multiple-Records.

The alternative solution is what we've been discussing on 1st July, that is Continuous-Acquisition and file-write at 2M from 6-cards Simultaneously.  We have example code for this but it will not achieve the 24Mbytes/s required (2M sample rate on all six cards).  It's also not clear if we need to stream 24Mbytes/s or 384Mbytes/s, one is well below the known maximum transfer rate of the PXI bus, the other is well above.

Please aim to provide a Yes/No answer within the next few days, if we need to purchase an alternative DAQ system to meet the requirement we need to start the procurement process immediately.

Regards,

Brian