Offline Results Processing Utility (ORPU) performance issues

[Corey.Rotunno]

Hello all,

  For some background information, in my test sequences I am saving a .tsr for every execution in a local folder. I then call the offline results processor to:

  1) Store results in a database

  2) Generate and store a report on a network directory

  3) Move .tsr file to network directory (I have it set up so this file is never deleted. This way we can always generate a new report using a different format template at a later date)

I have come across an issue with the offline results processing utility interface: As the number of .tsr files in the "output directory" increases my network activity increases and the responsiveness of the interface degrades. With 260 .tsr files in my output directory my network usage is ~3-5Mbps and the interface becomes unacceptably sluggish. Pausing the results processing profile does not help. It seems like the utility is constantly scanning my output directory no matter what.  Does anybody know of a work around or something in a .ini file I can change to fix this behavior? 

For the record....if any NI engineers happen across this post some option to disable or limit the output directory scanning should be implemented. In the "settings" dialog of the ORPU under the "delete processed files" section, the default used for maximum of historical files to keep is 5000! I don't even have 300 files and the ORPU has become borderline unusable.

Thank you,

[BoKnows]

Corey,

I'm taking a look at a few things.  The default value is 5000 and you're running into issues much before that.  I'd hope the default value has been tested.  That tells me something is different with your setup.  The first thing that sticks out is the network drive.  As a troubleshooting step, I'd be interested to see what happens if you use a local directory.  Do you see the same lagging?  I'm guessing the network usage would drop as a result.

If that's the case, the quickest workaround I could think of would rely on creating something else to move files around between your local PC and the network drive such that ORPU would be accessing them locally but you'd still be storing them on the network.  This is obviously not ideal.  But, it's the first thing I can think to try.

[Corey.Rotunno]

Jeff,

Thank you for the reply. So, I moved all my .tsr and .tsrinfo files onto a local directory and pointed the output of my processing profile to that directory. As expected my network activity went away and the interface became nice and responsive once again. However one thing did really suprise me; I expected to see the same level of disk activity as I had seen network activity but that is not the case. When I moved the output folder to my local drive, my drive is sitting there nice and quite.

I tried to amplify the issue I was seeing by creating 5 profiles in the ORPU all pointing to the same output directory. When using a network drive, the activity increased with each additional profile as I had observed previously. However, when I pointed all 5 profile to the same local directory, my disk activity remained very low.

Not sure what this all means. Your work around will work for me but I am not all that happy about it. I really want a network location that I can point a profile output to so that any test engineer in the organization can use the ORPU to re-process any results they wish. With the work around we can sort of do that, but as soon as anybody points an ORPU profile to the central network directory, the application is going to behave badly and their network activity is going to go crazy.

Thanks again though for the work-around. This makes me all the more curious as to what is going on. 

[Corey.Rotunno]

I just wanted to post a picture of what I am seeing. The provided screen shot was taken with 1 profile in the ORPU with 275 files in the network output directory. Just for clarification, the profile is not active, just there. If I copy/paste the profile network traffic increases in a largely linear fashion.

Thanks,

Corey

[WireWeaver]

There is a note in the TestStand Help about this.

Spoiler

Note  National Instruments recommends storing reports, offline result files, and databases locally instead of over a network for reliability and performance concerns. If you require a network storage solution, configure TestStand to store data locally and use a separate process or application to transfer files to and from the network.

-Trent

[Corey.Rotunno]

Trent,

  The help file you are referencing is for the "Offline Results File Generation Options Dialog Box" in TestStand itself. That's why I am using the ORPU, instead of the built in station resulting processing options described in the help file.

In the help you posted..."If you require a network storage solution, configure TestStand to store data locally and use a separate process or application to transfer files to and from the network. ". This is what I am doing, relying on a separate application (the ORPU) to post results to a database and store a report in a network location if and when the resources are available.

My issue relates to having the .tsr "output folder" of a ORPU profile pointing at a network folder. For my application I cannot simply let the .tsr files hang out on local machines indefinitely.

[WireWeaver]

ORPU uses pretty much the same process to generate reports as the standard plugins, so it will have the same limitations. 'separate process or application' is alluding to a script or similar app that runs automatically or manually to move the already generated files to a network location. Disk access is faster this way because the report generator is writing locally instead of to a network drive.

[Corey.Rotunno]

Trent,

  Yes, I see your point. I still think it's unacceptable that the utility made for processing, and more importantly to my application, re-processing past executions can't deal with a network directory as it's "output" folder. I am not processing the file from the network, I am using the ORPU to process local files and then simply move the .tsr (unprocessed file) to the "output folder" after the report has been generated and has been entered into the database. Even when the ORPU is not processing anything, it is a dog.... just because its' output folder is a network drive.

For a test engineer to re-process past .tsr's they must have a ORPU profile with its' output directory pointed to a network location, unless of course they first find the .tsr they want on the network and copy it to their computer.

I know I can work around it, but the fact of the matter is the is no way for me to process a past .tsr without the ORPU, absolutely none.

EDIT: I guess I'm not sure about that last thing. I have not learned very much about the report processing plug-ins and what they look like. Maybe I can build my own version of the ORPU, but it also might be over my head 😕 Guess I will look more into them and see if I can figure something out...just wish I didn't have to.

[Corey.Rotunno]

Jeff,

  Were you able to reproduce the issue I was seeing? If so, I am wondering if any of your colleagues have offered any input. That would be awesome if this issue was formally addressed by NI.

Thanks 

[BoKnows]

Corey,

I was out of town the past few days and didn't get a chance to reproduce the issue.  But, I'm also not sure there's much value in me doing so given the conversation that's taken place in my absence.  Trent works with TestStand and has a much better look at the inner workings than I do.  If he claims they use the same processes, I'd trust what he says.  As a result, the acknowledgement is already noted in the help documentation.  I could file a CAR.  But, I wouldn't expect it to be looked at as this is apparently expected behavior.  It sounds more like we're looking at a feature request where you're hoping to get the ORPU to use an entirely new API to handle the processing.  For feature requests, R&D tends to watch the Idea Exchange: http://forums.ni.com/t5/NI-TestStand-Idea-Exchange/idb-p/teststandideas  This helps weigh value to individuals and number of people this would affect so they can prioritize the features that have the most overall impact.

The workaround to this appears to be something like a VI/script that looks at the network folder, finds the files there, copies them to a local drive, and runs the ORPU from there.  When finished, a second VI/script deletes the files from the local drive (they're still stored on the network drive) and you handle all of your processing without networking issues.