Choosing a PXIe controller for streaming 200 MBps

[Topher C]

Warning:  This is a long post with several questions.  My appologies in advance.

I am a physics professor at a small liberal-arts college, and will be replacing a very old multi-channel analyzer for doing basic gamma-ray spectroscopy.  I would like to get a complete PXI system for maximum flexability.  Hopefully this configuration could be used for a lot of other experiments such as pulsed NMR.  But the most demanding role of the equipment would be gamma-ray spectroscopy, so I'll focus on that.

For this, I will need to be measuring either the maximum height of an electrical pulse, or (more often) the integrated voltage of the pulse.  Pulses are typically 500 ns wide (at half maximum), and between roughly 2-200 mV without a preamp and up to 10V after the preamp.  With the PXI-5122 I don't think I'll need a preamp (better timing information and simpler pedagogy).  A 100 MHz sampling rate would give me at least 50 samples over the main portion of the peak, and about 300 samples over the entire range of integration.  This should be plenty if not a bit of overkill.

My main questions are related to finding a long-term solution, and keeping up with the high data rate.  I'm mostly convinced that I want the NI PXIe-5122 digitizer board, and the cheapest (8-slot) PXIe chassis.  But I don't know what controller to use, or software environment (LabView / LabWindows / homebrew C++).  This system will likely run about $15,000, which is more than my department's yearly budget.  I have special funds to accomplish this now, but I want to minimize any future expenses in maintenance and updates.

The pulses to be measured arrive at random intervals, so performance will be best when I can still measure the heights or areas of pulses arriving in short succession.  Obviously if two pulses overlap, I have to get clever and probably ignore them both.  But I want to minimize dead time - the time after one pulse arrives that I become receptive to the next one.  Dead times of less than 2 or 3 microseconds would be nice.

I can imagine two general approaches.  One is to trigger on a pulse and have about a 3 us (or longer) readout window.  There could be a little bit of pileup inspection to tell if I happen to be seeing the beginning of a second pulse after the one responsible for the trigger.  Then I probably have to wait for some kind of re-arming time of the digitizer before it's ready to trigger on another pulse.  Hopefully this time is short, 1 or 2 us.  Is it?  I don't see this in the spec sheet unless it's equivalent to minimum holdoff (2 us).  For experiments with low rates of pulses, this seems like the easiest approach.

The other possibility is to stream data to the host computer, and somehow process the data as it rolls in.  For high rate experiments, this would be a better mode of operation if the computer can keep up.  For several minutes of continuous data collection, I cannot rely on buffering the entire sample in memory.  I could stream to a RAID, but it's too expensive and I want to get feedback in real time as pulses are collected.

With this in mind, what would you recommend for a controller?  The three choices that seem most reasonable to me are getting an embedded controller running Windows (or Linux?), an embedded controller running Labview real-time OS, or a fast interface card like the PCIe8371 and a powerful desktop PC.  If all options are workable, which one would give me the lowest cost of upgrades over the next decade or so?  I like the idea of a real-time embedded controller because I believe any run-of-the-mill desktop PC (whatever IT gives us) could connect and run the user interface including data display and higher-level analysis.  Is that correct?  But I am unsure of the life-span of an embedded controller, and am a little wary of the increased cost and need for periodic updates.  How are real-time OS upgrades handled?  Are they necessary?  Real-time sounds nice and all that, but in reality I do not need to process the data stream in a real-time environment.  It's just the computer and the digitizer board (not a control system), and both should buffer data very nicely.  Is there a raw performance difference between the two OSes available for embedded controllers?

As for live processing of the streaming data, is this even possible?  I'm not thinking very precisely about this (would really have to just try and find out), but it seems like it could possibly work on a a 2 GHz dual-core system.  It would have to handle 200 MBps, but the data processing is extremely simple.  For example one thread could mark the beginnings and ends of pulses, and do simple pile-up inspection.  Another thread could integrate the pulses (no curve fitting or interpolation necessary, just simple addition) and store results in a table or list.  Naievely, I'd have not quite 20 clock cycles per sample.  It would be tight.  Maybe just getting the data into the CPU cache is prohibitively slow.  I'm not really even knowledgeable enough to make a reasonable guess.  If it were possible, I would imagine that I would need to code it in LabWindows CVI and not LabView.  That's not a big problem, but does anyone else have a good read on this?  I have experience with C/C++, and some with LabView, but not LabWindows (yet).

What are my options if this system doesn't work out?  The return policy is somewhat unfriendly, as 30 days may pass quickly as I struggle with the system while teaching full time.  I'll have some student help and eventually a few long days over the summer.  An alternative system could be built around XIA's Pixie-4 digitizer, which should mostly just work out of the box.  I prefer somewhat the NI PXI-5122 solution because it's cheaper, better performance, has much more flexability, and suffers less from vendor lock-in.  XIA's software is proprietary and very costly.  If support ends or XIA gets bought out, I could be left with yet another legacy system.  Bad.

The Pixie-4 does the peak detection and integration in hardware (FPGAs I think) so computing requirements are minimal.  But again I prefer the flexibility of the NI digitizers.  I would, however, be very interested if data from something as fast as the 5122 could be streamed into an FPGA-based DSP module.  I haven't been able to find such a module yet.  Any suggestions?

Otherwise, am I on the right track in general on this kind of system, or badly mistaken about some issue?  Just want some reassurance before taking the plunge.

[ryannorthington]

Topher C,

Welcome to the NI forums.  That was quite a post and I think I have some answers for you. 

For your controller, the 8105 or 8106 would be a good choice.  As far as reliability is concerned, treat the controller as a computer.  Since it has many parts there can be multiple points of concern, but computer components are very reliable for the most part. 

We also a have Real Time (RT) version of the 8106.  The big difference between Real Time and Windows is that Real Time will be more deterministic, meaning your code will take priority when executing.  RT is not necessarily quicker, but if you need certain things in your code to execute at a certain time or frequency then you may want to consider a RT system.  You are correct in saying that you can control a front panel on your development machine and have the code running on the RT system.  However, if you compile your code into an executable and deploy it to the RT system then you will not be able to do this without putting in a little more effort.

Judging from the requirements you mentioned:

Sample Rate:  100 MS/s

Number of Samples:  300 samples

I think Windows will do just fine.  The way the Digitizer works is that it acquires the data and stores it on its on-board memory, it does not transfer it to computer memory until you tell it too.  When you call a Fetch method in LabVIEW (or CVI) it will then grab the data from the board and load it into memory.  You can also perform Direct Memory Access (DMA) with our scopes which will speed up the process. 

For your software environment LabVIEW is a good choice, because of its tight integration with the other NI products, such as Digitizers.  Again since the code

The rearm time on the 5122 is 3us and I found that spec on page 17 of the 5122 Specification Sheet.  This is a little slower than you mentioned in your previous post, so you will need to make sure this will be acceptable for your application. 

In addition to these recommendations I would call into NI Support and have one of the Engineers work through your system with you.  They would be able to answer your questions and work through some different options. 

I hope this helps!

[drnikitin]

Dear Ryan,

I have similar problem. I want to measure form of pulses each of 3 microseconds width, entering digitizer at rate 1000 per second. I have USB-5133 digitizer and want to use it. But I didn't find in its specification the value of rearm time. Please answer me the following two questions:

1) What is the value of rearm time for USB-5133?

2) Can I use USB-5133 to stream these pulses continuously into PC via USB (with 10 Ms/s at least)?

 

Thank you,

Dr. Nikitin

[Topher C]

Thanks Ryan for your answers!  I have called a sales rep and am getting a quote on this equipment.  I am still interested in whether or not it will be possible to continuously stream data from one channel to a controller and process the data without saving it to disk or buffering it all.  While not essential, such a capability would be nice, and if one controller could keep up with the task while another one could not, then that would be the one for me to buy.  I believe that a field engineer will be called for these questions.  I'll let you know what I end up with, and whether or not it works!

[ESD]

Topher C

I am trying to do pretty much the same thing as you.  Though I will be using proportional counters to look at protons and neutrons.  I also rejected the XIA system because of the same reasons you mention, though that may have been a mistake since my sytem does not yet perform satisfactorily.

I have tried to set up my acquisition to acquire pulses on a trigger.  I eventually want to look at coincidences between seperate detectors and it is easier for me to use a coincidence NIM module to trigger on multiple coincidences than to search long multi-pulse waveforms for timing coincidences between channels and deal with streaming all of that data to the PC.

LabView and the NiScope driver store each triggered waveform in what they call a record. You can make the records any number of samples, and store multiple records in the onboard memory.  Supposedly, you can transfer multiple records at once instead of setting up a new DMA transfer for each record(which takes time).  But this has not been my experience so far.  I am finding transfers of many short records at once to be very long while transfers of long records happen at the PCI bus speed, around 100MB/s (I don't have a PCIe system). For our application it seems natural to make a separate triggered record for each pulse that is a couple hundred samples long, but be aware that things may not work as advertised for multiple record transfers.  Anyway, as it stands I have maximum count rate capabilities of about 5kHz.  You can see a post of mine on the subject here.

Ryan N., the link you gave for more information on setting up DMA transfers just leads to the 5122 specifications.  I'd be very interested in this information if you could repost the link.  Thanks.

[Topher C]

Thanks for sharing your experience, ESD!

Yes, the time it takes to read out records (multiple or single) would certainly be an issue for me as well, as I would also like to be able to handle some 200,000 records per second.  If I'm able to stream data continuously, this wouldn't be an issue, but I am even more skeptical of streaming successfully, especially not for two channels simultaneously.  I would really want triggered acquisitions to work out, and fast.

I wonder if the source of slow-down you're observing is a hardware limitation, or software?  I have so little in-depth knowledge of LabView ... Is a LabView VI pre-compiled before execution?  Or is it run through an interpreter?  Or both?  I am wondering if a different program, either a re-worked VI or LabWindows or something else, might work?  My only previous experiences are with relatively slow digitizers running at about 48 kS/sec, and at most 100 records per second.

I tried to look at your example program, Timestamps.vi.  I guess I only have Labview version 8.0.1, and you created that with 8.5, so I cannot see it.  It's strange that LabView has such limited version compatability.  I can't think of any other programming language that prevents one from even reading the code without having the latest runtime software.

[ESD]

Topher C,

I'm not sure if what I'm seeing is hardware related or software, but I have a feeling it is software because you should be able to set up a DMA transfer and transfer many records in one large chunk at the speed of the PCI bus. It would really be nice to see what is going on in the NiScope driver, but this is the problem with closed source software.  I am hoping a NI engineer decides to chime in on this isssue.

I also can't give you much info on how LabView compiles the programs you write since I am new too.  But you probably can't load Timestamps.vi because it uses the NiScope driver and you don't have the driver installed.

Another company to check out besides XIA is ZTEC instruments.  They have some PXI digitizers that let you do some measurements onboard with an fpga.
But 200kHz count rate is really pushing the limits of any system, is it not? Let's say you have records that are 100 samples long, at 2 bytes per sample that's 200 bytes plus the extra 480 bytes that come with every record, rounded to the nearest 128 bytes is 768 bytes per record. That's 153 MB/s and you can transfer at that speed with PCIe, but I doubt you can process it that fast or write it to disk.  So you'll likely have to acquire in bursts.

[ryannorthington]

drnikitin,

The reason you did not find the spec for the rearm time for the 5133 is because the USB-5133 is not capable of multi-record acquisition.  The rearm time is a spec for the reference trigger, and that trigger is used when fetching the next record.  So every time you want to do another fetch you will have to stop and restart your task.  To grab a lot of data increase your minimum record size.  Keep in mind that you have 4MB of on board memory per channel. 

Since you will only be able to fetch 1 record at a time, there really isn’t a way to use streaming.  When you call fetch, it will transfer the amount of data you specify to PC memory through the USB port (up to 12 MB/s for USB 2.0 – Idealy).

Topher C,

We do have a Digitizer that has onboard signal processing (OSP), which would be quicker than performing post processing.  It is the NI 5142.  It is essentially a 5122 but with built in OSP.  It may be a little out of your price range, but it may be worth a look. 

When dealing with different LabVIEW versions it is important to note that previous versions will be compatible with new versions; such as going from 8.0 to 8.5.  Keep in mind that if you go too far back then LabVIEW may complain, but you still may be able to run your VI.  If you have a newer version going to an older version then we do have options in LabVIEW to save your VI for older versions.  It’s usually just 1 version back, but in LabVIEW 8.5 you can save for LabVIEW 8.2 and 8.0.

ESD,

 DMA is actually done every time you call a fetch or read function in LabVIEW or CVI (through NI-SCOPE).   

Topher C and ESD,

LabVIEW is a combination of a compiled language and an interpreted language.  Whenever you make a change to the block diagram LabVIEW compiles itself.  This way when you hit run, it is ready to execute.  During execution LabVIEW uses the run-time engine to reference shared libraries (such as dll’s).  

I hope all of this information helps!

[ESD]

Ryan,

That link finally expalins the problems I'm having with the line "When fetching data from a multi record acquisition the DMA operation is started and stopped for each individual record.".  I really wish the salesman and engineer, who were working with me when I bought this system a little over a month ago, had mentioned this.  I geuss they never really understood what I was planning to use the digitizer for.  And I almost certainly wouldn't have bought it.

Is there any way around this limitation?  Are the records stored sequentially in memory?   If so, why not transfer them all with one DMA transfer?  Can I do it myself in C++?

[Topher C]

Thank you very much for all the info!  This is extremely helpful.

The problem that ESD is having would also be a show-stopper for me.  I am curious -- is there a spec indicating the maximum transfer rate of triggered records (of, say, 500 samples each) is to a controller?  This may be software-specific, but there still should be useful numbers for certain configurations.  Does the PXIe-5122 outperform similar modules by a significant margin?

I believe that the NI 5142 is still (barely) in my price range, and it does bring in some useful functionality.  I really like the option of anti-aliasing for lower sample rates, and it sounds like it can also do FFTs as data comes in.  I could use anti-aliasing to downsample to about 25 MHz.  The FFT is irrelevant for my primary application of gamma-ray spectroscopy, but I have another potential application in (primitive) radio astronomy which could put this capability to good use.  So this module is good to know about, and I hadn't considered it before.  Thanks!

The OSP of the 5142 is not flexible enough for the signal processing I need to do.  I think I would need a digitizer, or better yet a separate module, that could expose the sample data to an FPGA that is clocked at the sampling rate of 100 MHz.  If the FPGA could put some data in registers that could then be read out on the PCI bus, my problems would all be solveable.  AFAIK this technology is not yet a commodity (although the individual components are).  Other possible benefits of the FPGA-based DSP could be to stream data through a FIR filter, and maybe even do sample compression before streaming to memory or disk.

Another workable option would be to trigger and transfer records at a high rate of speed.  Again, it seems this will not likely work.  And finally, the third option is continuous streaming.  But I do not see anything that suggests I could process the data in memory and clear it fast enough to keep up with the stream over a long period of time.  Fast enough or not, It doesn't seem like I can even clear part of an actively streaming record and free up the memory.  Is this true?

Again, I really appreciate your input on these issues.  This has been much more helpful than engineers over the phone!