PPL Namespaced Dependencies - Strategy/Design Discussion - Development Issues

[DerrickB]

I've got a project (picoG) with plugins to provide support for supported target devices, using a PPL hierarchy to simplify the plugin distribution (at least for file count on disk). I'm curious if others have come across similar dependency and namespacing issues, if you've taken the same approach I have, and what other thoughts people may have. Perhaps come up with some idea exchange feature requests...

Here is a chart of the initial dependency hierarchy before I made some changes:

This is a problem I ran into while refactoring picoG to integrate my terminal emulator to be able to monitor app deployments to targets and interact with the targets. Initially, I started out by implementing everything in the picoG library stack (directly using VISA / TCP etc) and was going to create a terminal component specifically for picoG. Of course I had to go and make the Terminal an external reusable component based on my streams libraries so I can use it standalone and elsewhere. (And for practice and because I wanted to). All of the gray libraries are VIPM packages that live in vi.lib and the other colored libraries are part of the picoG application. Each of the libraries marked as PPL are from separate projects and built into PPLs. The yellow Platform libraries are the plugins that the App dynamically loads and adhere to the Platform definition from Core. Core also contains wrappers over top of the TCP and Serial libraries that adhere to Core's Resource definition to provide some unified configuration capabilities and a stricter API for the platforms and app to use.

The problem is that the Core PPL build pulls in its own non-exported copies of the lower level stream libraries that become namespaced to the PPL and incompatible with the Terminal library at the App layer.

For completeness here's how the build output looks like for a the fully built app including the RP2040 platform library:

Solution 1

My current solution is to force all the streams library dependencies through the Core PPL by making a wrapper/proxy for the Terminal library within Core:

This works and I'm not too upset about the extra levels of API created to namespace everything streams into the Core PPL. Of course this means even more footprint and possible confusion for others (and myself when I get sidetracked for a few months and come back) at the higher layers when there are more options of classes to use and moving in/out of wrappers however some of it is a little cleaner due to making some more specialized APIs in those wrappers. This also means that if the external libraries (streams/terminals) are updated then the entire PPL hierarchy needs to be rebuilt to pull those changes into the Core PPL.

Solution 2

Another option for this architecture is to do all the builds as source distributions instead of PPLs and keep naked VIs on disk that are compatible with the run-time engine but I'd prefer to avoid the explosion of files needed to be distributed on disk along with the inevitable refactor to handle the changed structure and plugin folder layout. This is attractive for the fact that it would eliminate just about every dependency issue I'd have to deal with since there'd be no more namespacing being done in exchange for some more effort in managing build specs.

Other Solutions ??

Has anyone approached this kind of problem any other ways?

Possible Feature Requests

While I don't foresee future roadblocks to my solution #1 that I'm taking, there are a few issues that make working like this a pain in the butt. Perhaps I'm missing something and someone will be able to point out build/project settings that alleviate some of these. Perhaps NI can be asked to change/add features to alleviate them.

For starters, the Streams libraries that get pulled into the Core PPL aren't exported. The only reason this isn't a show stopper is because the right-click menu entry to place class members provides access to those VIs. Unfortunately, sometimes that capability misses VIs for some reason. It also doesn't filter / overlay the list based on access scope which can be confusing. Working through this menu, particularly for classes with larger footprints, is quite tedious. 

Idea #1: Add option to export dependencies pulled into PPLs.

I can't just add the streams libraries into the Core library that gets built into the Core PPL as that would modify the vi.lib libraries and cause all sorts of wonkyness. I could also maintain a different source copy of those streams libraries where they are pulled into the Core library and exported but that sounds like a nightmare for maintenance.

 So you can still get to and place non-exported VIs, it would be nicer if they could actually be viewed in the PPL and placed from there instead of the unorganized class palette menu:

And the class palette is only working because I've created a new child class of the IG Terminal that's part of the Core PPL solely to enable this workflow and dependency management.

Idea #2: Allow telling LabVIEW to ignore/remap namespaces.

Another possible workaround would be something to tell LabVIEW to treat what becomes "picoG-Core.lvlibp:IG Stream.lvlib:Stream.lvclass" as just "IG Stream.lvlib:Stream.lvclass" which would enable not pulling the Terminal library into Core and the App layer can still use the Terminal library from vi.lib as-is and avoids the need to proxy classes in the Core library. Sure, this could lead to compatibility issues as items being remapped from the PPL may be out of date with what else is in the app but I don't see that as any different from PPL builds being out of date, built in the wrong order, or not built at all so it doesn't seem like a real concern to me. Or any other mismanagement of source files on disk that we've gotten support cases about a handful of times. There would need to be a bunch of cases to catch for this demonstrated by my application architecture: when running the App layer in development, IG Terminal would be expecting "IG Stream.lvlib:Stream.lvclass" but once App gets built into a PPL there would be yet another namespacing rename that would need to be handled and even IG Terminal itself would get namespaced to "picoG-App.lvlibp:IG Terminal.lvlib:Terminal.lvclass". So surely not an easy to implement or manage idea.

Idea #3: Allow control over namespacing during builds.

This could overcome the issues in #2 and probably introduce a whole bunch of other issues or be infeasible depending on how LabVIEW implements namespacing based on filenames. This would basically entail overriding the namespace that gets applied to items packed into a PPL. I tested this by renaming a PPL file after building it and items definitely get names built into themselves and don't just rely on the file name, but renaming the PPL file causes LabVIEW to not be able to match up dependencies built into the PPL as expected since the linker info points into the PPL file. Since the fully qualified name is written to the item inside the PPL, surely it wouldn't be *too* difficult to be able to tell it to write a different name and exclude the wrapping PPL portion of the name?

The End

I'm curious what thoughts y'all have about this. I may end up just biting the bullet and refactoring out the usage of PPLs as a long term solution.

[crossrulz]

It there a reason the IG libraries couldn't also be built into PPLs specific for this project?  Though, if you are going that far, you could just add it to the "core" PPL, which is your solution #1.

---

@DerrickB wrote:

This also means that if the external libraries (streams/terminals) are updated then the entire PPL hierarchy needs to be rebuilt to pull those changes into the Core PPL.

---

Not necessarily.  Unless you change how something is interfaced (ex change the connector pane of an exported VI), you will likely only need to rebuild the core PPL what whatever PPL directly calls the updated library.

[DerrickB]

In the solution to get them into the PPL, I'm mainly thinking about new APIs that would be needed at the app later, so a rebuild to get changes in and propagated.

The IG streams libraries are all separate libraries and packages. To get them into a PPL that works if a new stream type is added means all separated PPLs which means refactoring the projects to link the different layers to PPLs.

I guess all that really matters is the final executable build and platform plugins but I'm trying to avoid a situation where there's manual processes to modify and build other dependencies if other devs get involved, eg someone wants to make a plugin for a device with a different build tool chain. Ideally there shouldn't be a need to get things through the core ppl when someone wants to make a new platform plugin, including me!

[DerrickB]

I think the path I'll be taking is to not separate the Core and App layers. I'll pull the App layer into core and then there's no concern over things getting pulled into the Core PPL and being namespaced differently than the dev libraries from vi.lib. There may still be issues for developers down the road that want to implement different communication resources for their platforms but that's a lot smaller of a problem and kicking that can down the road is a much smaller can that *shouldn't* require any other drastic refactors if needed. Platform plugins get to stay disconnected and small which is my ultimate goal.

[Mark_Yedinak]

Have a look at the presentation I did at NI Week in 2019: Effective use of PPLs.  What we ended up doing was pretty much building all of our libraries into PPLs. It does mean you have to manage the build so things are built in the correct order.

[DerrickB]

That's specifically what I'm avoiding. I don't want to maintain PPL and non PPL versions of the terminal and stream libraries, hence the inclusion via inheritance into the picoG-Core to get it accessible and namespaced. Even though I wrote them I'm treating the streams and terminal libraries as something someone else wrote and pulling them from VIPM. (Because I'm depending on their package installation locations, they're not direct source dependencies). Even if I used the original source repositories as dependencies I'd still need to refactor them to be built as PPLs which would break how they get published to VIPM and I certainly don't want to manage different builds for different architectures; it defeats their purpose.

I'd rather refactor picoG to not use PPLs at all than go down the route of putting the communication libraries into PPLs. Build order is less of an issue with LabVIEW Solution Explorer for those unfamiliar with the code base.

I've got the other option as stated of just combining Core and App to eliminate the namespacing difference that get introduced because of App using Core as a separate PPL. I'll take that route if using the proxy child class in Core becomes tedious. But that doesn't mean there can't be some feature requests for NI to reduce roadblocks for using shared reuse libraries without jumping through hoops.