Add data types: table, frame

[EngrStudent]

Languages like 'R', 'Python', and MatLab (yes I use the old name) have these.  They are useful.

One of the key ideas in LabVIEW is that the user needs minimal interventions to code a useful result.  As more information is encoded in a data type there is more opportunity to make "hands free" code that "just works".  I think these two data types can do that.

Data Frame:

• Primary data type in R
• It is Array-like, but NOT an array
  • each column contains one measurement (row) on one variable
  • It acts like a list of vectors, but the vectors have the same number of rows, and indexing allows a return of all or subset from all columns
  • column types are heterogenous - they can be different
  • column types can be set.  A column of 0 vs. 1 can be set as factors/binomial values or as continuous.

There are functions that data analysis folks do every day that are informed by variable type, so the function operating on the inputs doesn't need type specified because it is interior to the table.  This means you can say "plot(mydata)" and if your data is set up well, the graph parameters are already specified and useful.

Some references:

• http://tutorials.iq.harvard.edu/R/Rstatistics/Rstatistics.html
• http://www.r-tutor.com/r-introduction/data-frame
• http://stackoverflow.com/questions/2280276/vector-vs-data-frame-in-r
• https://stat.ethz.ch/R-manual/R-devel/library/base/html/data.frame.html

Data table:

This is from Hadley Wickham, a very famous person in 'R'.  He does great work, and his name has high brand value in data-analysis.

It:

• Uses the "data.table" package
• is able to be screaming-fast (think roller-coaster) especially when used with the "split-apply-combine" approach to data analysis, and SQL-like operations.
• is built for handling huge data (100GB tables) quickly and efficiently.

In many applications the same operation is not possible due to memory constraint or viable due to processor overhead can execute adequately (aka wonderfully) by using this data type on the same hardware.

References:

• https://cran.r-project.org/web/packages/data.table/index.html
• https://www.analyticsvidhya.com/blog/2016/05/data-table-data-frame-work-large-data-sets/
• http://datatable.r-forge.r-project.org/datatable-intro.pdf
• https://www.jstatsoft.org/article/view/v040i01
• https://github.com/hadley

[AristosQueue (NI)]

I flagged this for consideration by various people at NI. It's an interesting combination of a UI element for data entry and API for data processing. Kudos.

[aasenjo]

Kudos. As a user of the DataFrames in Python within NI, I can attest to its flexibility and speed.

The DataFrame type does take advantage of the more dynamic nature of these languages (duck-typing, specifically) so I'm curious if you have an opinion on whether we could have a more static version (fixed column types for the lifetime of the object, for example) without a major loss of flexibility, or whether it would require the language to become more dynamic.

I'm also curious how you'd rank this request against the need to have other simpler types such as dictionaries, trees, sets, etc.

Finally, libraries in these languages are built to take advantage of this type and it would help us if you had a list of favorite operations you think we'd need to support when adding this type to LabVIEW.

[Dmitry]

aasenjo wrote:

I'm also curious how you'd rank this request against the need to have other simpler types such as dictionaries, trees, sets, etc.

I would say this is a false choice - we need both. But given scarce LV R&D resources -Collections first - I think they address way more use cases than DataFrames ...

And since Collections need Generics, the latter should come first. Generics are very long overdue. Lack of Generics in G make us less productive. Many people are frustrated not having Generics in CG ...

[thols]

Really interesting about Data Frame. Didn't know about them. However, as Dmitry said: Generics first.