LabVIEW
How to pass and set Variants in the DLL?
Dear colleagues, can you please help me with kind of my homework?
I can easily fire UserEvent from DLL something like that:
The code under the hood:
typedef struct {
int32_t Type;
LStrHandle Value;
} TD1;
__declspec(dllexport) MgErr SendEvent(TD1 *Data, LVUserEventRef *UserEventRef)
{
char value[256];
switch (Data->Type) {
case 1: //Integer
sprintf(value, "%d", 42); break;
case 2: //Boolean
strcpy(value, "True"); break;
case 3: //String
strcpy(value, "Hello, LabVIEW!"); break;
default: //and so on
strcpy(value, "undefined"); break;
}
size_t len = strlen(value);
MgErr ret = NumericArrayResize(uB, 1, (UHandle*)&(Data->Value), len);
if (noErr == ret) {
MoveBlock(value, LStrBuf(*(Data->Value)), len);
LStrLen(*(Data->Value)) = (int32)len;
ret = PostLVUserEvent(*UserEventRef, Data);
}
return ret;
}But such trivial serialization is not elegant. Now I would like to do the same, but with Variant:
The question is — how C Code may looks like?
typedef struct {
int32_t Type;
LvVariantPtr Value;
} TD1V;
__declspec(dllexport) MgErr SendEventV(TD1V *Data, LVUserEventRef *UserEventRef)
{
switch (Data->Type) {
case 1: //Integer
// Set(Data->LvVariantPtr, 42) - ?
break;
case 2: //Boolean
// Set(Data->LvVariantPtr, TRUE) - ??
break;
case 3: //String
// Set(Data->LvVariantPtr, "How to do this???")
break;
default:
break;
}
MgErr ret = PostLVUserEvent(*UserEventRef, Data);
return ret;
}
One of NI Engineer's comments from April 10, 2012, states, "We do not publicly document how Variants are structured in LabVIEW. Therefore, we cannot provide information about how the Variants are structured."
Before deep diving into reverse engineering, I would like to ask first, because maybe someone already did this exercise since 2012 and can kindly share a code snippet, which will be helpful for me and other participants? Rolf Kalbermatter, perhaps?
Thank you in advance!
Yes I have worked on that for an alternative OPC UA Toolkit implementation to be released soon and figured out most of it. But don't expect it to be trivial. There is absolutely no way to directly manipulate the data behind a variant. In essence it is a structure containing various pointers, one is the datatype description as a C++ class with inheritance for every datatype, whose class interface is undocumented and almost impossible to reverse engineer without massive investment. Another pointer is the actual non-flat LabVIEW native data but without ability to parse the data type descriptor it is simply impossible to implement a generic library to access variants. Then there is a reference counter and some other pointer stuff, including what I think to be yet another class interface to handle various variant related operations. Additional difficulty is that this structure representing a variant in memory has been changed at least once somewhere around 2009 and may change anytime again. It's definitely not something you can consider as a stable element, so accessing the data in that structure directly is simply not an option.
I'm still in discussion with NI about how to guarantee that the necessary and so far undocumented APIs that LabVIEW exports, will be guaranteed to work in future LabVIEW versions. So far I only received preliminary confirmation that those APIs are not expected to change, but no full commitment that they won't change, accidentally or by design.
Basically the only set of APIs that I could find that was useful enough for my purposes and also not relying on to much almost impossible to determine internal details in LabVIEW, is based on compatibility functions that work with the old 16-bit typedescriptor format. And working with that is a tedious job. Your C code needs to parse this for receiving Variants and generate it for writing a variant and also then be able to walk the according unflat memory tree for the actual data. Additional difficulty arises out of the fact that LabVIEW uses different alignment rules for 32-bit Windows (byte packing) than the other still supported platforms (8 byte alignment). If you wanted to support older platforms like VxWorks things can get even more complicated. This alignment needs to be carefully taken into account as you walk the unflat data tree or build it up. One byte off here is the difference between a smooth running interface and a constant crash generator (found out the hard way many times during my experiments to figure out the necessary APIs).
De data handling routines on the C side need to be able to handle every possible datatype you want to support and each with its own data handling, ending up in long case structures with individual code paths for each datatype. About half of the actual code in my OPC UA Toolkit is concerned about this. The rest is adapting the LabVIEW friendly C function interface into the underlaying OPC UA C library API and some resource tracking and managing.
I'm at the moment mainly busy with finalizing the first version of this toolkit and not able to provide more information or support other projects to use variants from C code. And while NI was pretty supportive about information to create this toolkit, they have not yet stated if they will be releasing this information as additional part of the External Code Reference Documentation in the future or how they really feel if someone would at least document that interface openly. I got the feeling that they are more open to make this information available but that it is a low priority thing that would require more resources than they are willing to invest at this point.
Rolf dropped a whole essay there looking pretty scary.
I understood... some of it.
Anyway, how devoted are you to passing a Variant directly? Or do you just want something that could be close?
I am wondering if the "Variant To Flattened String" and "Flattened String To Variant" are functions you would be willing to use in LabVIEW on either side of your DLL, because unlike the Variant itself, those formats are documented:
https://www.ni.com/docs/en-US/bundle/labview/page/flattened-data.html
@Kyle97330 wrote:
Rolf dropped a whole essay there looking pretty scary.
I understood... some of it.
Anyway, how devoted are you to passing a Variant directly? Or do you just want something that could be close?
I am wondering if the "Variant To Flattened String" and "Flattened String To Variant" are functions you would be willing to use in LabVIEW on either side of your DLL, because unlike the Variant itself, those formats are documented:
https://www.ni.com/docs/en-US/bundle/labview/page/flattened-data.html
That's what my colleague Albert-Jan Brouwer used in the Lua for LabVIEW Toolkit. It was back then the only way to deal with LabVIEW data on the C code side without strict typed function variants for every datatype, if you did not want to rely on so far undocumented functionality. And it still is until now. In terms of complexity of the according C code it is in parts similar to what you need to do with the undocumented Variant functions. Even the 16-bit typedescriptor is in fact exactly the same. The disadvantage of this approach is that Flattening and Unflattening is not a zero cost operation at all in terms of both memory and performance.
@Kyle97330 wrote:
Rolf dropped a whole essay there looking pretty scary.
I understood... some of it.
Anyway, how devoted are you to passing a Variant directly? Or do you just want something that could be close?
There is additional overhead, as Rolf mentioned above (thanks for the answer). If you take a look at the original NI Example for OPC UA client, you will see it:
The OPC UA data change event is set in the DLL (a reference is passed as a parameter to niopcua_client_createDataChangeSubscription(...) called from ni_opcua.dll). So, it is definitely possible, this is why I asked the question. This design has pros and cons, and many other solutions and workarounds to avoid variants are possible, for sure, from custom serialization to multiple events. I was just curious.
@Andrey_Dmitriev wrote:
The OPC UA data change event is set in the DLL (a reference is passed as a parameter to niopcua_client_createDataChangeSubscription(...) called from ni_opcua.dll). So, it is definitely possible, this is why I asked the question. This design has pros and cons, and many other solutions and workarounds to avoid variants are possible, for sure, from custom serialization to multiple events. I was just curious.
The OPC UA Toolkit uses a special form of Variant interface that is called FlexData or something along those lines. It is basically yet another flattened format that is most likely part of the original Logos framework that NI lifted from the Lookout software and for some part added to LabVIEW itself, probably related also to Datasockets and later Shared Network Variables, which are based on the same Logos infrastructure that Datasockets uses. I had investigated that interface too, and even started going down that path as I had not been able to identify a more direct access to the LabVIEW Variant with the existing APIs that LabVIEW itself exports. And going deeper into the manager core DLLs is simply an exercise that I want to avoid at pretty much any cost as that is for sure VERY LabVIEW version specific and not at all documented in any way by an inadvertent NI leak. I had managed to get it working for basic scalar datatypes but always felt unsatisfied by the fact that it is basically a multi step conversion (LabVIEW Variant<->FlexData flattened format<->whatever native binary format you want). By accident I then found an API that provided the missing link to go directly from LabVIEW Variant to native binary data, but it required to use the old 16-bit type descriptor. This is not ideal, but the advantage is that it is (almost) fully documented whereas the more direct 32-bit type descriptor and/or TDR class interface is simply so very much undocumented, that use of that is beyond what I was willing to put effort in.
@rolfk wrote:The OPC UA Toolkit uses a special form of Variant interface that is called FlexData or something along those lines. It is basically yet another flattened format that is most likely part of the original Logos framework that NI lifted from the Lookout ...
Thank you for this note. This explains why I have seen some strange differences when I ran my own library and OPC UA under the debugger. Interesting.
In theory, I can hook into internal variant structures, but this is not necessary because after some research, I've found a pair of functions:
MgErr LvVariantFlattenExp(const uintptr_t lvVar, LStrHandle *flatData, uint32_t lvVers);
MgErr LvVariantUnFlattenExp(const uintptr_t *lvVar, uint8_t *flatData, int32_t length, int32_t lvVers, int32_t lvContext);They are exported but undocumented (as well as the whole Variant itself). In general, they're the same as 'Variant to Flattened String' and 'Flattened String to Variant', but suitable for DLL use. I only need the second one; it is sufficient for me.
The 'flatData' parameter contains a sequence of bytes where type string and data string are joined together. The length should be written in the third parameter, and the last two can be zeroes (the 4th param is not fully clear to me, the last one is context, which needs to be investigated deeper).
Anyway, this flat data can be obtained with LvVariantFlattenExp(), just for better understanding:
And this is how it looks:
Type Descriptors are officially documented. For example, for Boolean and String:
#![feature(concat_bytes)]
use std::{ffi::c_void, ptr};
#[repr(C)]pub struct TD1Variant {
data_type: u16,
data_value: TVariant,
}
type TVariant = *mut *mut c_void;
const NO_ERR: i32 = 0;
type MgErr = i32;
unsafe extern "C" { //exported from LabVIEW.exe
fn PostLVUserEvent(user_event_ref: *mut c_void, data: *mut c_void) -> MgErr;
fn LvVariantUnFlattenExp(variant: TVariant, str: *const u8, size: i32, unknown: i32, context: i32) -> MgErr;
}
#[unsafe(no_mangle)]
pub extern "C" fn SendEvent_Variant(data: *mut TD1Variant, user_event_ref: *mut *mut c_void) -> MgErr {
unsafe {
let value: &[u8] = match (*data).data_type {
1=> &[0, 4, 0, 3, 0, 0, 0, 42], //Integer, 42
2=> &[0, 4, 0, 0x21, 0b1000_0000, 0x0], //Boolean, True
3=> concat_bytes!([0, 8, 0, 0x30, 0xFF, 0xFF, 0xFF, 0xFF, 0, 0, 0, 16], b"Hello, Variants!"),
_=> b"Unknown"
};
let value_out = prepend_length_header(value);
let ret = LvVariantUnFlattenExp(ptr::addr_of!((*data).data_value) as TVariant,
value_out.as_ptr(), value_out.len() as i32, 0, 0);
if ret != NO_ERR {return ret};
PostLVUserEvent(*user_event_ref, data as *mut c_void)
}
}
fn prepend_length_header(value: &[u8]) -> Vec<u8> {
let mut result = Vec::with_capacity(4 + value.len() + 4);
let total_length = (value.len() + 😎 as u32; // Include header in length
result.extend_from_slice(&total_length.to_be_bytes()); // Big-endian 4-byte header
result.extend_from_slice(value);
result.extend_from_slice(&[0u8; 4]);
result
}And here's how this is used and works:
Yes I did look at those APIs too. They are rather similar to the Flex functions, but the flattened format is different. The main difficulty I found was the fact to have to build the flattened data in memory. Manageable if you do it for a limited set of types like in your example but a major pain in the ass if you want to do it more generic. And the fact that you basically generate flattened data, to be converted into an variant is not exactly high performance either.
One point however: PostLVUserEvent() does not consume the data passed in. Since you create that data with every new call to LvVariantUnFlattenExp() call you also have to properly deallocate it after passing it to PostLVUserEvent() or you simply create a memory leak. Although in your specific use case the owner of that variant is basically the diagram, so it is most likely going ok., but your use of Rust kind of confuses me a bit.
As to your extra parameters for those functions:
lvVers: is the LabVIEW version number as 32-bit value for the flattened data format. While the flattened format has mostly stayed consistent over the course of LabVIEW versions, there are some changes. For the documented datatypes this is limited to booleans between LabVIEW 4 and 5 and another thing around LabVIEW 7.1. But for the undocumented types like variants, waveform data and more, there were actually more and sometimes subtle changes between LabVIEW versions. Passing in 0 tells LabVIEW to use the version of the current runtime.
lvContext: is pretty much the application reference. This is useful when trying to convert flattened data for a specific application context such as a real-time target. These targets have potentially different endianess (not really a concern anymore since LabVIEW 2021 since the vxWorks PPC cRIOs are not anymore supported) but also memory alignment rules, which are different between 32-bit Windows and all the other platforms. Potentially there are other target specific differences, but these two are the main factors. Passing in 0 again tells LabVIEW to use the current application context under which the C function is called.
