Data Acquisition Idea Exchange
The request is for NI to prioritize development of VIs that allow integrators’ CAN data to be measured and stored in standardized formats.
While TDMS and XNET are decent tools, it’s lack of modern development and open compatibility in industry makes it somewhat difficult to deal with. NI could probably expand and service its user base if it supported XNET and MDF4 better.
Currently our process is as follows:
- LabView XNET data comes from the cRIO FPGA (eg. NI-9853) as an array of 6 U32s
- LabView XNET default data structure is the typdef ‘XNET Frame CAN.ctl’
- See XNET help ‘XNET Convert (Byte Array to Frame CAN).vi’
- There are obscure (near obsolete) VIs or the end user must write code to convert from 6 U32 to XNET Frame CAN on poor documentation.
- As an aside, there doesn’t seem to be a clean standardized way to store either of these structures in the TDMS files
- There is a way to configure a TDMS channel for CAN frame logging, but it is not easy for signal post processing because it is only one channel of frames.
- We believe a much better way would be something like this if we stuck with TDMS.
However as to this feature request we have found that the industry standard MDF4 file format to be advantageous in most aspects.
Our tests found that a 1 hour 1.1GB raw CAN frames and analog data TDMS file could be stored as a 200 MB file in the compact MDF4 file ! When the CANbus data was expanded into actual signals, the MDF4 file was 900 MB (still less than the raw TDMS). Further this MDF4 file loaded into FAMOS for review in ~1 sec !
We would prefer to output directly to MDF4 rather than having to script this conversion process.
- One key detail is the storage of raw CAN frames such that other industry tools (eg. MATLAB, FAMOS, PYTHON, MDFvalidator, ASAMmdf GUI) can natively interpret and process with various DBC files (the standard includes tools to extract CAN signals).
- Secondly with high channel counts of fast sample rates, the file size quickly becomes enormous. MDF4 allows excellent compactness, further compression, and usability of data (including but not limited to CAN).
- However, NI’s DataPlugin needs work as it fails to read much of the structure, properties and data of MDF4 files (as seen from ‘Data File Viewer.vi’ utility below). To unlock the true power of MDF4, the CAN frame has to be written correctly. Currently this is impossible.
- We were surprised that NI as an ASAM member had not fully supported the MDF4 format.
LV vs. free ASAMmdf GUI
- Vector’s free MDFvalidator tool useful for seeing the file structure
APPENDIX: here are some links and resources to summarize importance of MDF4 file format.
- Link to the standard’s homepage ASAM MDF [asam.net]
- Link to the MDF4 format’s documentation asammdf "7.3.14" documentation [asammdf.readthedocs.io]
- Link to a founding company’s page Measurement Data Format MDF | Vector [vector.com]
- Link to NI MDF4 plugin showing features https://www.ni.com/pdf/manuals/asam-ev-dataplugin-mdf4-2023-q3.html (note that #10 ByteArray is not supported).
-
Why MDF4 is Popular for Testing:
- Efficient Storage: As a binary format, MDF4 allows for the efficient storage of large volumes of data, which is crucial in extensive testing scenarios.
- Standardization: Being an ASAM (Association for Standardization of Automation and Measuring Systems) standard ensures interoperability between different tools and systems used in the industry.
- Metadata Support: MDF4 allows for the inclusion of rich metadata, providing context and facilitating the interpretation of the raw measurement data. This includes information about the signals, units, and test conditions.
- High Performance: The format is optimized for fast read and write access, which speeds up the analysis of large datasets.
- Flexibility: MDF4 supports various data types and allows for extensions, making it adaptable to evolving testing needs.
