Handling of NIPKG files when using nipkg feed-add-pkg

[cbutcher]

For a 'local' (or indeed, I believe, networked and specified via UNC) path feed, the use of nipkg feed-add-pkg adds the package to the package feed, but doesn't move the package (blah.nipkg file) anywhere.

This results in sometimes curious relative paths in the MyFeedDir/Packages file (example given below for a well formed case, and then a less delightful example...)

Reasonable example (actually, the source of this post, and this doesn't work, explained below*)

Architecture: windows_all
CompatibilityVersion: 210000
... blah blah other fields ... 
Filename: ../pipelines/OIST_Error.lvlibp/NIPKGs/oist-oist-error-windows32-debug_0.2.0.4_windows_all.nipkg
Plugin: file

Less reasonable example

Architecture: windows_all
CompatibilityVersion: 220000
Filename: ../../../../../../C:///Users/Christian/temp-test-builds/NIPKGS/oist-oist-error-windows32-debug_0.2.0.4_windows_all.nipkg
Plugin: file

I wondered

1. Am I doing this in the completely wrong fashion?
2. Is this behaviour the same regardless of the type of value used with nipkg feed-add or feed-create?
3. Does feed-add-absolute-package do something different (that command takes a FEED_DIR, package specification and FEED_NAME, where the package specification seems to be searched for in the feed given by FEED_NAME)

With regards 3, I got the following after using it for the same package and same FEED_DIR:

Filename: \\C:\Users\Christian\temp-test-builds\NIPKGS\oist-oist-error-windows32-debug_0.2.0.4_windows_all.nipkg

which seems a bit more reasonable, but still useless (network drives typically won't have \\C:\ point to my harddrive, even assuming such a path exists).

I can currently imagine the following options, but I'm happy to learn about more:

• Manually copy to adjacent directory, provide paths to feed-add-pkg in such a way that the relative path is used and fairly short. This seems to work, but is an extra step (not terrible, but also not documented that I found) (this results in something like the 'good' example)
• Create a webservice that accepts the nipkg as an upload and carries out that step (storing on the 'server' as appropriate, wherever that might be, before using feed-add-pkg or feed-add-absolute-package). Is this how AWS-backed or SystemLink-like services work?

Actually, writing those, it seems more or less the same, and on closer inspection only feed-add (not anything relating to packages) specifies a URI or gives other options for specifying remote services...

So I guess perhaps nipkg feed-add-pkg or feed-add-absolute-package are intended only to be run server-side with access to reasonable local (or local-looking) directories?

* since I said I'd explain below, this path is inside a container and used a bind mount to the package feed directory, but didn't also copy the package out, leading to the feed having a path specified for a file in a container that was destroyed...

[cbutcher]

There's a nice post describing AWS hosting here: https://forums.ni.com/t5/NI-Package-Management/Create-a-Cloud-hosted-NIPM-Feed-on-Amazon-S3/ta-p/384... (Thanks Allen)

[Scott_Richardson · Accepted solution]

cbutcher -

The primary workflow for creating feeds are to publish them on a share where the feed is in a folder and its packages are either in the same folder or some relative path on that share, so all package paths in the feed manifest are relative. This is how NI publishes its feed externally and internally.

The nipkg CLI commands "feed-create" and "feed-add-pkg" add package references to a new or existing feed "packages file" and does not copy or move any package files. The design of the commands are to generate a relative path between the folders for the feed to that of the packages.

When attempting to use these commands with the feed and packages folders residing on two different UNC paths, C:\ and D:\, or C:\ and \\mynetworkshare\, the commands incorrectly try to generate a "relative" path between the two and generate paths like you see above. My initial testing suggests that this has been like this since our first release of NIPM around 2017, so these commands do not appear to support specifying "absolute" paths to the packages. I have created an internal work item to track this use case. It might be that once fixed, we will just generate absolute paths when relative is not possible. For now, these commands cannot be used.

I do not know the history of the "feed-add-absolute-package" command. Although generating an absolute path using this is possible, it is more difficult to use. The command takes a packageID and a feedname as parameters. For this to work, NIPM must have a registered feed that already includes the package, and the call would be something like: "nipkg.exe feed-add-absolute-package c:\myfeeddir\ ni-example-package:windows_x64=18.5.0.49153-0+f1 MyRegisteredFeedName". The command then finds the package in the registered feed, grabs the package details and then adds the package details to the target feed. In the end, not very useful.

So at this point, nipkg.exe does not really support configuring absolute paths in feed manifest files.

Technically, the "Packages" file in the "Packages.gz" file in a feed directory is the source of truth for the feed, and can be extracted, edited and replaced. I tested this by configuring a local feed on my system with a package path to a network share. It is still tricky because I had to copy the package locally from the network share location, generate the feed locally, edit the manifest to point to the remote location.

When specifying a package URL to a website, NIPM might also prevent the package from installing and display the error "Redirection to another URL is forbidden" because the feed path is not have the same base as the package path. This is one of the current limitations that prevents NIPM from supporting feeds hosted on GitHub, which we would like to support in the future.