time consuming FindRoute function

[Now_With_Underscores]

Hello Peter,

I reread my last post and there's an obvious error:

"Adding an option to use the first available route would drastically increase the search time, but might not find the best route."

This should be:

"Adding an option to use the first available route would drastically decrease the search time, but might not find the best route."

Hope this didn't cause any confusion.  

Message Edited by Knights Who Say NI on 12-28-2009 08:08 PM

[PePf_TS]

Hello John,

thanks for your reply. Well, we had a really long discussion here and yet there is no final decission done how we want to proceed.

Well, our feeling is, that Switch Executive is a really flexible tool with a lot of possibilities. But this great flexibility has the lack that it can not be as performant as it is required in our applications. We need a routing setup time of 1ms (max. 10ms) per routing, which is not possible with Switch Executive. At the moment we need 10 minutes (!!!) to setup the test system with all 400 connections. This is not a solution. We are searching for a faster solution to get the system setup within max. 8 seconds.

Best regards,

Peter

[Now_With_Underscores]

Hi Peter,

To achieve performance in the 10ms range in this large application, we're going to need to predefine each Route Group in Switch Executive; as we define each Route Group, Switch Executive determines the route path and stores this value as a string.  By specifically telling the driver which relays to connect, we forego the entire 'can connect?' search we would otherwise perform and therefore decrease our runtime switching times dramatically.  

[MSE_TS]

Hi everyone,

we managed to predefine the necessary routes in the excel Configuration file. Once this file is importe into NISE, the switching takes about 10..15ms for any given routing (not quite as fast as we originally intended the switchmatrix to be, but nevertheless a speed with which we can actually work)

However, we have massive problems with the actual import of the configuration file into NISE.

To give you an Idea, the first config with predefined routings we generated and tried to import had approx 30.000 routes specified (one specified route per route group) and had a filesize of 8.5mb.

The import was very slow and after approx 10minutes NICE quit itself with an Errormessage "internal error".

We cut down most of the configfile resulting in approx 10.000 trained routes (that is the bare minimum with which we can work at all), the filesize was 3.5mb.

During the import of this file, NISE took about an hour (!) while "not responding" (according to the taskmanager), it also went completely inert without reacting to anything. After that we somehow managed to  save the file which so far works quite fine. (I am using "managed" because NISE was extremely slow to be used...)

However, for another purpose we need a new configuration file that is bigger, which we cannot import due to the "internal error" error. We also have to use other configuration files, all pretty similar to the one that already took over an hour to import.  

So, basically you're right with the "predefining of routes", but the actual import of these is almost not possible. 

Now the question is, is there anyway to improve the performance of the import? We don't even know if the process is still doing something as no kind of progressbar is shown during the import...

(I also tried to import via a textfile, that is equally slow)

Is there a maximum of importable routes?

best regards,

Michael Seemann

ps: I am a colleague of Peter's by the way.. 

[Now_With_Underscores]

Hey Michael,

Please attach the 10,000 route group (3.5MB) file and we'll take a look to see if there's any optimization techniques.

[Now_With_Underscores]

Hey Michael,

One technique we could use in SwitchExecutive would be to not define the hardwires between each of the four modules.  Thus, we'd have 4 separate modules and many less routes to define.  We'd still physically hardwire the connections together between modules, but by not having these connections defined in software, we'd reduce the number of required predefined routes.  When we want to make a connection from module1c1 to module3c4, we could make the necessary internal connections on each module in order to arrive at the desired connections.  I understand we'd lose a lot of the flexibility NISE gives us by making this sacrifice, but given the number of routes we've defined, we might need to consider trading off ease of use for speed.  

Given your desired timing requirements of 1ms, another option we might consider is using the 3rd party IVI switch API; lower level APIs are much faster than SwitchExecutive, but don't give us many of the ease of use options contained within SwitchExecutive.  With most 3rd party APIs, the user must define each physical connection.

As was mentioned previously, the ultimate solution to this issue would be to add the option to use the first available route in SwitchExecutive(which might not be the 'best' route).  This functionality would drastically decrease the time required to dynamically create routes on the fly.  

Message Edited by Knights Who Say NI on 06-03-2010 10:57 AM

Message Edited by Knights Who Say NI on 06-03-2010 10:58 AM

[MSE_TS]

Hello John,

thank you for your reply & ideas.

However I think we talk about two different topics, one ("mine") is that it seems to be impossible to import large configurations into NISE to prepare routes beforehand. Concerning this issue we would like to know what actually happens during import to find out what's the problem. (I am pretty sure that no validation etc is happening during the import as we found several examples where the import itself run just fine and later on we had problems, e.g. the specification string of a given routing could not be imported and therefore was not written correctly into NISE, no error occured during import)

Therefore I ask you again if you could check what is happening during import and if there might be any conctraints (memory, number of predefined routes, the like...

The way I understand your proposals, all deal with the improvement of the findroute process itself, either by eliminating the hardwires, by using other software or by integrating the "first available route" option, please correct me if I'm wrong.

a propos Hardwires, we tried diffent things:

- elininate (in NISE) hardwires, then switch routes with the "test panel" that should "use" the same channels that previously where hardwired, this works fine (i.e. NISE says it does work, we haven't actually measured anything with the hardware itself but I am confident that a connection is made if NISE does not raise an error)

- figurring we maybe do not need hardwires at all (which I thought would not be possible..) we then used the same configuration file and eliminated all hardwires (in excel before importing the file). The import itself worked, but all specification strings where empty with an error that said that the endpoints where not available. (we did NOT use a big file as we can not import these, see above, but a pretty small configuration just to test the hardwire-related issues)

my question: what is the difference if i set up hardwires in excel, import these and then erase them in NISE compared to importing a configuration that has no hardwires in the firstplace? there seems to be a difference.

- we used the big configuration file with which we encountered the importing problem described above and erased all hardwires to test if that makes any difference during import. So far it looks as if the import might work, at least NISE has been running for close to 90 minutes now without crashing. (however, although that is very slow, if it worked it might a way for us to work, but as I described above, configurations withoug Hardwires do not seem to work at all....

Another Software is not an option at the moment 

We'd greatly appreciate if the "first route availabe" option was availabe as fast as possible, if it is performing well, but from what we heard that might take time until august or even longer. We can not wait that long, we need a solution really fast. 

[Now_With_Underscores]

Hello Michael,

I agree with you that we've discovered incorrect behavior with NISE with your application.  We are definitely looking into the root cause of this issue, but in the meantime we'll need to work around this limitation.  We strive to implement valuable customer feedback and I definitely agree that a "first available route" option will decrease the autoroute search time and therefore increase the usability of switch executive for your application.  

In regards to your validation concerns, we should always validate after making any modifications to the virtual device configuration and after deploying a virtual device.  The import process does not guarantee that the user-defined data is valid.  Validation is therefore recommended after importing.

My goal in removing the hardwires was to have fewer possible route combinations to import.  The tradeoff here is that we must treat each switch module separately, which requires logic to determine which routes interconnect between modules.  I agree with you that this is not an ideal solution.

You asked: what is the difference if i set up hardwires in excel, import these and then erase them in NISE compared to importing a configuration that has no hardwires in the firstplace? there seems to be a difference.

Can you elaborate on the difference you've observed?

Since you've indicated that we must use Switch Executive for this application, we could forgo the large predefined list and build the static routes on the fly within the programming API.  Instead of defining a particular predefined route, we can build the route at runtime.  A detailed description of the required syntax exists in the Switch Executive Help file, and is also available as an example in the NI Example Finder (Toolkits and Modules»NI Switch Executive»NI Switch Executive - Route Specification Syntax.vi).  I've reproduced the Full Route Specification syntax below:

 

With this syntax, we could build the routes dynamically on the fly.  You could, for example, keep track of which rows are currently in use, and determine how to build the route.  

One other possibility is that depending on how we access the routes, we could only load a small subset of the total number of routes at a time.  For example, suppose test1 only uses 5000 specific routes.  We could dynamically load these 5000 routes using our API, then run test1, then load the next 5000 routes for test2, etc.   Obviously, this won't work if the test routes aren't modular enough for subdivision.  Let me know if you'd like more information on how to use the Switch Executive API to accomplish this.

Again, thanks for bringing this particular limitation to our attention.  For the time being, we'll need to come up with some sort of workaround until we can implement a faster autoroute function. 

[MSE_TS]

Hello John

-validation:Thank you for the information, we are going to use the validation procedure regurlarly now. (an idea: you might set a popup-window or some other sort of information right after the import is done, to notify the user that there was no validation so far, or set an option to swtich on automatic validation after import)

-removing hardwires: so far we do not understand how the reduction of hardwires would lead to "fewer possible routes", the overall number of possible routes should remain the same. However we do not need more information regarding this topic at the moment (see below)

-differences hardwires def. in excel+ erase or no hardwires at all in config:

I'd have to find my notes on that topic to give you a detailed description. I'll see what i can do..

-using subsets:

not possible in our usecase, since we would have to change a lot in the software (client's software as well) to make changing of swmx configs during the test possible:

-route generation at runtime:

We changed quite a bit of our software to do what you proposed, i.e. generate the full route specification string(s) at runtime, then use niSE_Connect to switch the route. so far this looks promising. 

thank you for your quick response.

best regards,

Michael 

[MSE_TS]

Hello everyone,  

it is very likely that we are going to use the Switch Exec. in the near future for a very similar project like the last one (with which we had the problems with the slow routing)

I was just wondering if there have been any changes to NISE in the field discussed in this thread, such as:

- option to use the first route available

- overall performance increases in the routing find software (expecially within large complex setups)

- etc...

please let me know if the things discussed here lead to some changes that are relevant.

best regards

Michael