inconsistent install failures on crio targets (resolved)

smithed · ‎08-21-2018

Hi, I have a few cRIO targets I am trying to install to and I keep getting different errors when installing from systemlink. The main purpose of this is just to (a) help anyone running into the same problems and (b) share the fun error messages I've gotten in hopes they can be improved in some meaningful way.

First, in the system:

All cRIOs connected to the server
feeds are set up correctly
If i run opkg from putty to install my package it works correctly
On one particular system, locally connected to the systemlink server, I was able to get the package to install, but only after changing from blank default password to a real password, and even though it installed it still got marked as failed and when I looked at the list of installed software from systemlink it didn't show up
On a different system, connected through the corporate network, with the default blank password, the package installed, but still showed up as having failed, except once I refreshed the installed software list it was there and I was able to then uninstall it remotely (successfully)

You can see some of the various logs produced below. I have some where the install succeeded, but checking the status failed. I have others where the "Redacted" command failed, etc.

The problem, it turns out, was that when this machine was set up a while back its 2nd port was set up to the same network range as one on the corporate network. So I had x.x.x.80 (local) and x.x.x.30 (corporate). The systems were able to talk to each other some of the time, but not always. So yes, I realize this was a stupid set-up and its our own fault for that being the case, the half-working state was really hard to debug.

Failed: -2147467259

[
   [
      {
         "__kwarg__": true,
         "pkgs": [
            {
               "ntp-utils": "4.2.8p8-r0.1"
            },
            {
               "ntp": "4.2.8p8-r0.1"
            },
            {
               "ntpdate": "4.2.8p8-r0.1"
            }
         ],
         "refresh": true
      }
   ],
   [],
   [
      {
         "__kwarg__": true,
         "attr": [
            "description",
            "displayname",
            "displayversion",
            "group",
            "packager",
            "priority",
            "url",
            "uservisible",
            "version"
         ]
      }
   ]
]


[
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 468, in install
    rs_result = _get_restartcheck_result(errors)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 78, in _get_restartcheck_result
    rs_result = __salt__['restartcheck.restartcheck'](verbose=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/restartcheck.py\", line 425, in restartcheck
    if not _kernel_modules_changed_nilrt(kernel):
  File \"/usr/lib/python2.7/site-packages/salt/modules/restartcheck.py\", line 372, in _kernel_modules_changed_nilrt
    retcode = __salt__['cmd.retcode']('md5sum -cs {0}'.format(depmodpath_md5sum), output_loglevel=\"quiet\")
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 1838, in retcode
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 575, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command 'REDACTED' with the context '{'timeout': None, 'with_communicate': True, 'shell': False, 'bg': False, 'stderr': -2, 'env': {'USERNAME': 'root', 'LC_NUMERIC': 'C', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'LINES': '24', 'LC_MEASUREMENT': 'C', 'SUDO_UID': '501', 'SUDO_GID': '500', 'LC_CTYPE': 'C', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/admin', 'LC_PAPER': 'C', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin', 'LC_TIME': 'C', '_': '/sbin/start-stop-daemon', 'SUDO_COMMAND': '/usr/local/natinst/bin/salt-register N9244489.the-lab.llnl.gov', 'TERM': 'linux', 'SHELL': '/bin/sh', 'LC_MESSAGES': 'C', 'SHLVL': '2', 'LC_COLLATE': 'C', 'LC_TELEPHONE': 'C', 'LC_MONETARY': 'C', 'PWD': '/', 'COLUMNS': '80', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'SUDO_USER': 'webserv'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'cwd': '/home/admin'}', reason: command not found
",
   false,
   null
]

Failed (-2147467259) 

[
   [
      {
         "__kwarg__": true,
         "pkgs": [
            {
               "demo-motor-controller": "1.0.0-2"
            }
         ],
         "refresh": true
      }
   ],
   [],
   [
      {
         "__kwarg__": true,
         "attr": [
            "description",
            "displayname",
            "displayversion",
            "group",
            "packager",
            "priority",
            "url",
            "uservisible",
            "version"
         ]
      }
   ]
]


[
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 440, in install
    new = list_pkgs()
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 871, in list_pkgs
    out = __salt__['cmd.run'](cmd, output_loglevel='trace', python_shell=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 980, in run
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 575, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command '['opkg', 'list-installed']' with the context '{'timeout': None, 'with_communicate': True, 'shell': False, 'bg': False, 'stderr': -2, 'env': {'USERNAME': 'root', 'LC_NUMERIC': 'C', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'LINES': '24', 'LC_MEASUREMENT': 'C', 'SUDO_UID': '501', 'SUDO_GID': '500', 'LC_CTYPE': 'C', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/admin', 'LC_PAPER': 'C', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin', 'LC_TIME': 'C', '_': '/sbin/start-stop-daemon', 'SUDO_COMMAND': '/usr/local/natinst/bin/salt-register N9244489.the-lab.llnl.gov', 'TERM': 'linux', 'SHELL': '/bin/sh', 'LC_MESSAGES': 'C', 'SHLVL': '2', 'LC_COLLATE': 'C', 'LC_TELEPHONE': 'C', 'LC_MONETARY': 'C', 'PWD': '/', 'COLUMNS': '80', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'SUDO_USER': 'webserv'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'cwd': '/home/admin'}', reason: command not found
",
   false,
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 1046, in info_installed
    python_shell=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 1675, in run_all
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 575, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command '['opkg', 'status']' with the context '{'timeout': None, 'with_communicate': True, 'shell': False, 'bg': False, 'stderr': -1, 'env': {'USERNAME': 'root', 'LC_NUMERIC': 'C', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'LINES': '24', 'LC_MEASUREMENT': 'C', 'SUDO_UID': '501', 'SUDO_GID': '500', 'LC_CTYPE': 'C', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/admin', 'LC_PAPER': 'C', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin', 'LC_TIME': 'C', '_': '/sbin/start-stop-daemon', 'SUDO_COMMAND': '/usr/local/natinst/bin/salt-register N9244489.the-lab.llnl.gov', 'TERM': 'linux', 'SHELL': '/bin/sh', 'LC_MESSAGES': 'C', 'SHLVL': '2', 'LC_COLLATE': 'C', 'LC_TELEPHONE': 'C', 'LC_MONETARY': 'C', 'PWD': '/', 'COLUMNS': '80', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'SUDO_USER': 'webserv'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'cwd': '/home/admin'}', reason: command not found
"
]

Failed (-2147467259)

[
   [
      {
         "__kwarg__": true,
         "pkgs": [
            {
               "ntp-utils": "4.2.8p8-r0.1"
            },
            {
               "ntp": "4.2.8p8-r0.1"
            },
            {
               "ntpdate": "4.2.8p8-r0.1"
            }
         ],
         "refresh": true
      }
   ],
   [],
   [
      {
         "__kwarg__": true,
         "attr": [
            "description",
            "displayname",
            "displayversion",
            "group",
            "packager",
            "priority",
            "url",
            "uservisible",
            "version"
         ]
      }
   ]
]


[
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 468, in install
    rs_result = _get_restartcheck_result(errors)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 78, in _get_restartcheck_result
    rs_result = __salt__['restartcheck.restartcheck'](verbose=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/restartcheck.py\", line 415, in restartcheck
    kernel_versions = _kernel_versions_nilrt()
  File \"/usr/lib/python2.7/site-packages/salt/modules/restartcheck.py\", line 325, in _kernel_versions_nilrt
    __salt__['cmd.run']('gunzip {0}'.format(compressed_kernel))
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 980, in run
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 555, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/utils/timed_subprocess.py\", line 42, in __init__
    self.process = subprocess.Popen(args, **kwargs)
  File \"/usr/lib/python2.7/subprocess.py\", line 390, in __init__
    errread, errwrite)
  File \"/usr/lib/python2.7/subprocess.py\", line 1000, in _execute_child
    data = _eintr_retry_call(os.read, errpipe_read, 1048576)
  File \"/usr/lib/python2.7/subprocess.py\", line 121, in _eintr_retry_call
    return func(*args)
MemoryError
",
   false,
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 1046, in info_installed
    python_shell=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 1675, in run_all
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 575, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command '['opkg', 'status']' with the context '{'timeout': None, 'with_communicate': True, 'shell': False, 'bg': False, 'stderr': -1, 'env': {'LC_NUMERIC': 'C', 'CONSOLE': '/dev/tty0', 'LC_ADDRESS': 'C', 'runlevel': '5', 'LC_CTYPE': 'C', 'LC_NAME': 'C', 'LC_TELEPHONE': 'C', 'HOME': '/', 'LC_PAPER': 'C', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin', 'threadsirqs': '1', '_': '/sbin/start-stop-daemon', 'previous': 'N', 'LC_IDENTIFICATION': 'C', 'TERM': 'linux', 'SHELL': '/bin/sh', 'VERBOSE': 'no', 'SHLVL': '2', 'LC_COLLATE': 'C', 'PREVLEVEL': 'N', 'LC_MESSAGES': 'C', 'LC_MONETARY': 'C', 'PWD': '/', 'RUNLEVEL': '5', 'LC_MEASUREMENT': 'C', 'LC_TIME': 'C', 'INIT_VERSION': 'sysvinit-2.88'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'cwd': '/'}', reason: command not found
"
]

Save system startup settings
Failed (-2147467259) 


Arguments
[
   [
      true
   ],
   []
]


Detailed Results
[
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1667, in _thread_multi_return
    func = minion_instance.functions[data['fun'][ind]]
  File \"/usr/lib/python2.7/site-packages/salt/loader.py\", line 1119, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File \"/usr/lib/python2.7/site-packages/salt/utils/lazy.py\", line 101, in __getitem__
    raise KeyError(key)
KeyError: 'startup.enable_labview_access'
",
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/var/lib/salt/minion/extmods/modules/nisysmgmt.py\", line 214, in grains_items
    _refresh_grains()
  File \"/var/lib/salt/minion/extmods/modules/nisysmgmt.py\", line 79, in _refresh_grains
    grains = salt.loader.grains(__opts__, force_refresh=True)
  File \"/usr/lib/python2.7/site-packages/salt/loader.py\", line 721, in grains
    ret = funcs[key]()
  File \"/usr/lib/python2.7/site-packages/salt/grains/core.py\", line 1898, in hwaddr_interfaces
    ifaces = _get_interfaces()
  File \"/usr/lib/python2.7/site-packages/salt/grains/core.py\", line 1219, in _get_interfaces
    _INTERFACES = salt.utils.network.interfaces()
  File \"/usr/lib/python2.7/site-packages/salt/utils/network.py\", line 895, in interfaces
    return linux_interfaces()
  File \"/usr/lib/python2.7/site-packages/salt/utils/network.py\", line 773, in linux_interfaces
    stderr=subprocess.STDOUT).communicate()[0]
  File \"/usr/lib/python2.7/subprocess.py\", line 390, in __init__
    errread, errwrite)
  File \"/usr/lib/python2.7/subprocess.py\", line 1000, in _execute_child
    data = _eintr_retry_call(os.read, errpipe_read, 1048576)
  File \"/usr/lib/python2.7/subprocess.py\", line 121, in _eintr_retry_call
    return func(*args)
MemoryError
"
]

Failed (-2147467259) 

[
   [
      {
         "__kwarg__": true,
         "pkgs": [
            {
               "ntp-utils": "4.2.8p8-r0.1"
            },
            {
               "ntp": "4.2.8p8-r0.1"
            },
            {
               "ntpdate": "4.2.8p8-r0.1"
            }
         ],
         "refresh": true
      }
   ],
   [],
   [
      {
         "__kwarg__": true,
         "attr": [
            "description",
            "displayname",
            "displayversion",
            "group",
            "packager",
            "priority",
            "url",
            "uservisible",
            "version"
         ]
      }
   ]
]

[
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 346, in install
    old = list_pkgs()
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 871, in list_pkgs
    out = __salt__['cmd.run'](cmd, output_loglevel='trace', python_shell=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 980, in run
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 575, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command '['opkg', 'list-installed']' with the context '{'timeout': None, 'with_communicate': True, 'shell': False, 'bg': False, 'stderr': -2, 'env': {'USERNAME': 'root', 'LC_NUMERIC': 'C', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'LINES': '24', 'LC_MEASUREMENT': 'C', 'SUDO_UID': '501', 'SUDO_GID': '500', 'LC_CTYPE': 'C', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/admin', 'LC_PAPER': 'C', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin', 'LC_TIME': 'C', '_': '/sbin/start-stop-daemon', 'SUDO_COMMAND': '/usr/local/natinst/bin/salt-register 172.17.3.1', 'TERM': 'linux', 'SHELL': '/bin/sh', 'LC_MESSAGES': 'C', 'SHLVL': '2', 'LC_COLLATE': 'C', 'LC_TELEPHONE': 'C', 'LC_MONETARY': 'C', 'PWD': '/', 'COLUMNS': '80', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'SUDO_USER': 'webserv'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'cwd': '/home/admin'}', reason: command not found
",
   false,
   "Traceback (most recent call last):
  File \"/usr/lib/python2.7/site-packages/salt/minion.py\", line 1674, in _thread_multi_return
    ret['return'][ind] = func(*args, **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/opkg.py\", line 1046, in info_installed
    python_shell=False)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 1675, in run_all
    **kwargs)
  File \"/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py\", line 575, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command '['opkg', 'status']' with the context '{'timeout': None, 'with_communicate': True, 'shell': False, 'bg': False, 'stderr': -1, 'env': {'USERNAME': 'root', 'LC_NUMERIC': 'C', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'LINES': '24', 'LC_MEASUREMENT': 'C', 'SUDO_UID': '501', 'SUDO_GID': '500', 'LC_CTYPE': 'C', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/admin', 'LC_PAPER': 'C', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin', 'LC_TIME': 'C', '_': '/sbin/start-stop-daemon', 'SUDO_COMMAND': '/usr/local/natinst/bin/salt-register 172.17.3.1', 'TERM': 'linux', 'SHELL': '/bin/sh', 'LC_MESSAGES': 'C', 'SHLVL': '2', 'LC_COLLATE': 'C', 'LC_TELEPHONE': 'C', 'LC_MONETARY': 'C', 'PWD': '/', 'COLUMNS': '80', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'SUDO_USER': 'webserv'}, 'stdout': -1, 'close_fds': True, 'stdin': None, 'cwd': '/home/admin'}', reason: command not found
"
]

SystemLink Forum

inconsistent install failures on crio targets (resolved)

inconsistent install failures on crio targets (resolved)