Strange python sys module behavior

[SeryDavid]

Hello, 

I noticed that the sys module is somehow modified in DIAdem.

The code 

import sys
print(sys.argv)
print(sys.stdout, sys.__stdout__)
print(sys.stderr, sys.__stderr__)

Produces output

[]

<DIAdem.Redirectstdout object at 0x00000000033C0E188> None

None None

Comments to that:

sys.argv should at least as the first item contain the script name - this is used even by some of the standard packages, e.g. matplotlib

Why is sys.stdout redirected? Now when I execute my script I have to wait until it finishes before I see the outputs. This is far from ideal - When I have a script running for several minutes, I do not know if anything/what is hapenning.

I wanted to revert the default sys.stdout, but the sys.__stdout__ is deleted altogether.

Why is sys.stderr None? Again, some libraries use that to print the error output. With this they throw an error.

P.S. Is there a way to stop a running program? E.g. for a situation when there is by mistake a neveending loop?

[Brad_Turpin]

Hi SeryDavid,

The big picture here is that the python engine is running embedded as a child process of the DIAdem application.  This is the only option if the python script is going to be able to "reach up" and access the "dd" variable from the parent DIAdem application, so that the python script can call DIAdem commands, use DIAdem objects and get/set DIAdem variables:

from DIAdem import Application as dd

This one huge advantage comes with a number of disadvantages.

1) DIAdem loads the python engine into internal memory at startup and uses that same python engine instance for the entire duration of that parent DIAdem instance, releasing it only when DIAdem terminates.

2) During the parent DIAdem instance's lifetime, any changes on disk to the python engine or it's modules do NOT reflect in the embedded python engine the DIAdem instance loaded at its startup.

3) DIAdem only resets the python engine once, at DIAdem startup and reuses that same python engine for each subsequent python script run from that DIAdem instance.

4) Any python module imported by scriptA.py will still be loaded in the python engine when scriptB.py runs a little later in that same DIAdem instance.

5) The embedded python engine running as a child process under the parent DIAdem application is NOT running in DOS stand-alone, the way most python programmers are used to.

6) There is no way to break out of a python script running in an embedded python engine under a parent DIAdem instance.  I you end up in an endless loop, you have to terminate the parent application DIAdem.

7) There are no DOS extensions/parameters you can use to parametrize the execution of the python script from outside it (the way it is called/invoked).

😎 Since the python script is not running in DOS, the usual output options are not available.  The print() command is rewired to output into the DIAdem logfile, which you can see at the bottom of your SCRIPT panel.

There is nothing standing in the way of using print() to show updates to the user in the DIAdem logfile during the python script execution.  Or you can create your own python logfiles and view those with external tools-- the python "logging" module helps with that.

Brad Turpin

Principal Technical Support Engineer

National Instruments

[Pesmith8]

Hi Brad and SeryDavid,

I very much appreciated the question, and the detailed response. This help my use, and explains the why it works that way!

I realize the there were some limitations in using python as a scripting language in DIAdem. but for me the benefits are much greater than the draw backs.

Questions some to mind:

1) If are running a worker, and have that worker/s running a python script.  Will it have a new python interpreter, and a fresh load of all call packages?

2)Comment:  In my use of Python in DIAdem. I work to develop python code in a IDE like Pycharm or Spyder or VS Code. Once this is fully working then I import it into DIAdem to call.

2) Comment:  In my development, I have multiple instances of DIAdem up at same time. So  the issue where can stop code that has ran away, does not bother to much.

3) Comment:  I have found the use of Redis for Windows in combination with DIAdem to be a useful combination. It lets me record log like data to Redis and share communication with other instances, in either python or DIAdem (Other languages as well)

Anyway. Would like to know what the improvements in this area are planned for DIAdem 2021 and beyond.

Paul

[DIAdemo]

Hi,

@Brad
this is all true, but Python does not run in a child process, it runs in the same DIAdem process to allow fast access to data and functions.

@SeryDavid
To the missing arguments there is a simple workaround, as I found out only now, you can create them yourself.
..
import sys
sys.argv=["scriptname"]
..

@Paul
if you run a DIAdem worker, it is a child process and it has its own fresh Python engine.

Greetings,

DIAdemo

[Pesmith8]

Hi DIAdemo,

Thanks for the fast answers.  

I especially liked that when start up workers they have a fresh python engine.

Paul

ps related question that is kind of strange from a strange guy.  If start up workers and the worker is python scripting based(I.e. the DIAdem.ddd file is configured to be python scripting).  Could each worker be setup to call the python.exe at a different path, that could be different for each worker.

[SeryDavid]

Hello, yes, I use similar workaround for the sys.argv. I also redirected sys.stderr, which was None. I am still confused why were those variables corrupted in the first place, though.

[DIAdemo]

Hi,

@Paul
yes, you can create every DIAdem worker with a different configuration.

..

ParallelProcessControl.Workers.Add(ID,true,"/dC:\myconfigs\config_1.ddd")

@SeryDavid
DIAdem currently does not require to save the script code to a file before starting it. Thats why there is no filname.

Greetings,

DIAdemo