NI-DAQmx on Ubuntu 22.04 with kernel 6.8 not working

[pieleric]

Hi,

Not really a question, but more a note, for those who are also facing the same issue.

Ubuntu 22.04 just got an updated HWE kernel (beginning of August), from 6.5 to 6.8. That's the also the kernel which is used on Ubuntu 24.04 LTS.

However, with our systems, using a PCIe 636x, although the NI-DAQmx drivers do compile, they fail to work properly.

We see such error:

  File "/usr/local/lib/python3.10/dist-packages/nidaqmx/stream_writers.py", line 398, in write_int16
    return self._interpreter.write_binary_i16(
  File "/usr/local/lib/python3.10/dist-packages/nidaqmx/_library_interpreter.py", line 5968, in write_binary_i16
    self.check_for_error(error_code, samps_per_chan_written=samps_per_chan_written.value)
  File "/usr/local/lib/python3.10/dist-packages/nidaqmx/_library_interpreter.py", line 6410, in check_for_error
    raise DaqWriteError(extended_error_info, error_code, samps_per_chan_written)
nidaqmx.errors.DaqWriteError: A hardware failure has occurred. The operation could not be completed as specified.

Task Name: AO

The kernel log shows:

kernel: [  218.402117] DMAR: DRHD: handling fault status reg 3
kernel: [  218.402131] DMAR: [DMA Read NO_PASID] Request device [01:00.0] fault addr 0x148df5000 [fault reason 0x06] PTE Read access is not set

The NI "platform roadmap" states that only kernels 5.15 and 6.2 are supported (for now), so that issue is not really surprising (and maybe the surprising part is that it did work on v6.5). https://www.ni.com/en/shop/software-portfolio/platform-roadmap.html

The roadmap states that the Ubuntu 24.04 LTS kernel is 6.2, though that's actually 6.8, so probably it will be something looked into soon, for Q4?

I guess, for our side, the lesson is that really, with the NI-DAQmx drivers, only the LTS kernel can be used on production systems.

Cheers,

Éric

[GabeJ · Accepted solution]

Ubuntu 24.04 has begun enabling DMA remapping by default.  Many NI drivers do not support DMA remapping.  "DMA is not working with NI Drivers on IOMMU-enabled systems" on the 2024 Q1 Known Issues page has more information and a potential workaround.

[pieleric]

Thanks a lot GabeJ, that did work.

We'll switch the production systems the LTS kernels anyway, at the ones where the hardware is supported because apparently the kernel updates can break things a little bit too easily...

[nikongen]

Just wanted to mention I have the same Issue with Ubuntu 24.04.1 LTS (Noble Numbat) and Kernel 6.8.0-47-generic.

I have a python script, that creates this error:

A hardware failure has occurred. The operation could not be completed as specified.
Task Name: voltage-sweep
Status Code: -5015

Before executing the python script I run the following command to check for the kernel message

sudo journalctl --system -f 

I can relate this message to this message

kernel: niraptnonusb 0000:04:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0x1133cd000 flags=0x0000]
kernel: niraptnonusb 0000:04:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000c address=0x1133db000 flags=0x0020]​

Your message is different, thats why I did not find the error before via grep.

Are You using a Intel CPU? Maybe thats the reason for the different messages

This guide solved the issue for me, too.

[pieleric]

Yes, here I have an Intel CPU. That could well be the reason for difference of logs.

Eventually, passing "iommu=pt" as kernel option (by editing /etc/default/grub) works fine.

[nikongen]

Yes 🙂

I created a ansible playbook to automate installing nidaqmx on ubuntu machines. Thought I share it here.

Unfortunately I cant create a new topic. Before I forget I just post it here:

---

# Attention! Spacing ist very important!
- hosts: labpc
  become: true
  tasks:

  - name: Install nidaqmx
    block:
      - name: Install signing key for ni repo
        ansible.builtin.template:
          src: templates/ni-driver/ni-software-2024-noble.asc
          dest: /usr/share/keyrings/ni-software-2024-noble.asc
          owner: root
          group: root
          mode: '0755'

      # Get the signing key and source list entry from existing installation
      - name: Install template for ni repo
        ansible.builtin.template:
          src: templates/ni-driver/ni-software-2024-noble.list
          dest: /etc/apt/sources.list.d/ni-software-2024-noble.list
          owner: root
          group: root
          mode: '0755'

      - name: Install ni-daq package
        ansible.builtin.package:
          name:
            - ni-daqmx  # ni driver
          state: latest
          update_cache: yes

      - name: NI Driver install 
        ansible.builtin.command: dkms autoinstall

  # Apply workaround https://www.ni.com/docs/en-US/bundle/ni-platform-on-linux-desktop/page/iommu-linux.html
  - name: Apply DMA workaround
    block:

    # https://stackoverflow.com/questions/55844981/ansible-insert-word-in-grub-cmdline
    - name: Enable IOMMU via grub
      ansible.builtin.lineinfile:
        path: /etc/default/grub
        regexp: '^GRUB_CMDLINE_LINUX_DEFAULT="((?:(?!iommu=pt).)*?)"$'
        line: 'GRUB_CMDLINE_LINUX_DEFAULT="\1 iommu=pt"'
        backup: true
        backrefs: true
      notify: update-grub

    - name: Update grub
      ansible.builtin.command: update-grub
      become: yes

    - name: Reboot machine
      ansible.builtin.reboot:
        reboot_timeout: 3600

[norgc]

Hey nikongen,

Do you mind sharing the python script you used to reproduce this error? I believe we are experiencing a similar issue, but it seems to happen intermittently and are struggling to reproduce the issue consistently. This has made it quite difficult to validate any changes that we make to our system.

Thank you!

[nikongen]

Hi,

sorry I just saw the last post rn.

Here is the code sample. Hope it still helps

from nidaqmx.task import Task
from nidaqmx import constants
import numpy as np
import yaml


class AOM():
    # write single value to analog output
    def AoWriteValue(self, device, value, impedance=1.0e6):
        aoTask = Task()
        aoTask.ao_channels.add_ao_voltage_chan(device)
        aoTask.ao_load_impedance = impedance
        aoTask.write(value, auto_start=True)
        aoTask.wait_until_done()
        aoTask.close()

    # write digital output, value has to be True/1 or False/0
    def DoWriteValue(self, device, value, impedance=1.0e4):
        doTask = Task()
        doTask.do_channels.add_do_chan(device)
        doTask.do_load_impedance = impedance
        if value:
            doTask.write(constants.LogicFamily.FIVE_V.value, auto_start=True)
        else:
            doTask.write(0)
        doTask.wait_until_done()
        doTask.close()

    # write analog values and meanwhile collect data
    def GetSyncedAiWhileAo(self, aoDevice, aiDevice, aoData, samplingRate):
        """This function is necessary for calibrating the AOM with the photodiode.
        It allows for simultaneous writing of analog output and reading of analog input.

        Args:
            aoDevice (str): AOM for calibration
            aiDevice (str): Photodiode for calibration
            aoData (np.array): e.g. linear ramp
            samplingRate (float): rate, at which the data is collected

        Returns:
            np.array: array of collected data
        """
        # configure aiTask
        aiTask = Task()

        if aoData.ndim > 1:
            samplesPerChannel = len(aoData[0])
        else:
            samplesPerChannel = len(aoData)

        aiTask.ai_channels.add_ai_current_chan(
            physical_channel=aiDevice,
            terminal_config=constants.TerminalConfiguration.RSE,
            shunt_resistor_loc=constants.CurrentShuntResistorLocation.EXTERNAL,
            ext_shunt_resistor_val=50,
            units=constants.CurrentUnits.AMPS
            )
        aiTask.timing.cfg_samp_clk_timing(
            rate=samplingRate, sample_mode=constants.AcquisitionType.FINITE,
            samps_per_chan=samplesPerChannel
            )

        # turn on AOM
        self.DoWriteValue("Dev1/port0", True, impedance=1e4)

        # conffigure aoTask
        aoTask = Task()
        aoTask.ao_channels.add_ao_voltage_chan(aoDevice)
        aoTask.timing.cfg_samp_clk_timing(
            rate=samplingRate,
            source="/Dev1/ai/SampleClock",  # have to look this up!
            sample_mode=constants.AcquisitionType.FINITE,
            samps_per_chan=samplesPerChannel
            )
        aoTask.write(aoData, auto_start=False)

        # ai task is master, ao task is slave
        aoTask.start()
        aiTask.start()
        # do NOT call ai_task.wait_until_done()
        # otherwise the register_every_n_samples_acquired_into_buffer_event is called only ONCE
        aiTask.wait_until_done(timeout=200)

        dataAi = aiTask.read(number_of_samples_per_channel=samplesPerChannel)

        aoTask.close()
        aiTask.close()

        # turn off AOM
        self.DoWriteValue("Dev1/port0", False, impedance=1e4)

        return dataAi