Flatten to JSON - Allow any data type on the "anything" input

[matt.baker]

Suggestion: Allow any data type to be connected to the "anything" terminal of the flatten to JSON function.

Basically the same as the flatten to XML function.

Also the reverse, unflatten from JSON, may need to be done as well.

[Tomi_Maila]

Darren,

Thanks for the update. I don't see this anymore as a major issue becasue there are third party tools that can provide the support missing from LabVIEW. I maintain one of those tools, the JKI JSON library, which supports many more data types than LabVIEW does. If anyone has suggestions to what is missing from the support, please submit an issue report the JKI JSON GitHub page.

Tomi

[drjdpowell]

>>I don't see this anymore as a major issue becasue there are third party tools that can provide the support missing from LabVIEW.

The problem is speed, Tomi.  Existing JSON addons like JKI's or LAVA's are orders of magnitude slower than the inbuilt primative.  This restricts their use to applications where speed doesn't matter (config files, for example).   

[Tomi_Maila]

djrdpowell,

That's true. The LabVIEW reflection API is way too slow for the third party solutions to be able to match with the native performance. The preferred solution to the problem would be for NI to make the reflection API orders of magnitude faster. That would help not only to this problem but a lot of other problems. Right now flattening to JSON can be relatively fast using the third party libraries, unflattening from JSON is quite slow compared to the built-in JSON.

Tomi

[AristosQueue (NI)]

Don't use a reflection API. Write a JSON serialization for your data type, and if you don't want to hand-code it, use scripting to generate it -- or regenerate it when it changes. Regeneration is generally the harder challenge anyway with non-primitive data structures. You quickly start needing version management control, nesting of data structures with independent change schedules, etc. It happens to pretty much every piece of software I've ever worked on that had any sort of serialization component, down to the smallest utility apps. The reflection API approach rarely helps with that.

This is the approach taken by the Character Lineator. I think it is overall a better way to go for any information that will be needed for more than just temporary string transfers.

[drjdpowell]

 > use scripting to generate it -- or regenerate it when it changes

Do you mean xnodes or similar here?  That may be a good solution for flattening clusters/objects, but I definately have code use cases where there is no statically-defined data type structure to (re)generate from.  For example, I have code that performs statistics on supplied JSON data, and applies User specified test conditions, without knowing at compile time the structure of the data.

[AristosQueue (NI)]

JDPowell: That's outside the use case of Flatten To JSON and Unflatten From JSON functions anyway. You want a DOM parser for JSON. This thread was discussing the Flatten and Unflatten nodes.

[drjdpowell]

 OK, but isn't the current NI JSON functions too inflexible to be used for many flattening operations?  In anything that requires a combined JSON structure coming from multiple sources (a config file for a multi-module project, say, or a JSON flat form of an object where multiple classes contribute) the current primatives are unusable.

[PBD_ctrl]

NI wants to play in the IOT but can't natively support JSON without using external tools or developing your own.  Doesn't line up to me.

[drjdpowell]

PBD_ctrl, FYI,

I've got a side project for a new JSON library, "JSONtext", that I hope to be much closer in performance to the inbuilt primatives, while being highly flexible:

https://lavag.org/topic/19830-what-do-you-use-json-for/?do=findComment&comment=120103

[Tomi_Maila]

AristosQueue I'm not sure I understand. How exactly do you create a reuse component that can serialize any JSON compatible LabVIEW type to JSON that would not need to query type information somehow from the reflection API? LabVIEW does not support type parameters or generic types so the only type that can accept anything would be a variant. Now to my understanding we need to either traverse the variant to figure out what it is that the variant contains or alternatively flatten the variant to flattened string and type descriptor and parse these for type information. Do you see another way to create reusable serializer that would work with a wide range of LabVIEW data. Emphasis here on reusable as writing a serializer for each use separately is way too time consuming. Ain't nobody got time for that!

Let's take an example use case. I want to create a generic logger that logs JSON events over the network to a log server over a TCP connection (This is a real world high performance use case I've had to implement). The application components can log whatever they want. Let's say there are about 50 application components each actors or actor like components managing different aspect of the application. Every 50 to 100 ms something happens in each of them and they need to call the logger to log the event. The way I implemented it was that they called a log event API method that took a variant as an input. This variant got enqueued to a log event queue together with timestampt, the name of the component and some other metadata. The log manager then processed the events in the queue and constructed a JSON message from the variant and the metadata (not knowing what the variant contained) and pushed the JSON message to the TCP socket. Worked well but the performance of the variant to JSON conversion could have been better. How would you AQ solve this exact real-world use case? The implementation goal has to be that it is dead simple for application components to call the Log Event method and that the application components can log anything. And that what they log can change from state to state and is not necessarily the same from one iteration of the event loop of the actor to the next.