Ethernet data

[SJenkins9]

Hi everyone,

I am evaluating Labview for my company to see if this is a product that we can use.  I'm confident we can get this to work, I'm just not confident in my Labview abilities yet and I need to create a working app for management to review.

Here is an overview of what I'm trying to do:

I need to create an application that, for all intents and purposes, is a data viewer.  I need to gather data from a network, via UDP, and display the data in various output fields, graphs, charts, ADI's etc...  The main issue I see is that I need to bring in large amounts of data, up to 2 MB in some instances, parse this data out, run some algorithms, and then display the result.  The incoming data will be a mixture of char, short, int, floats, and doubles.  We will have multiple display pages within our VI (we will probably use tabs) where a single parameter can be displayed in multiple pages.

What I have done so far:

I can read a packet of data into my VI with the UDP read (awesome and quite easy tool to use).  I can take the data and parse it out into an unsigned byte array.  I have also experimented with the Unflatten from String function into an unsigned 1-dimensional array of shorts (I wanted this to be a 1-dimensional array of bytes but couldn't figure that out) and then convert that array to a named cluster to parse the data.  I can successfully see the data coming in and can manipulate each piece.

I am hoping that someone can kind of point me in the right direction on what tools would be best to use and where to start.  I have looked through the forumns and online to try and find some examples of best practise for dealing with large amounts of data but I haven't had any success.  I'm also not asking anyone to extensively write anything as that would take away my fun.  So, here are my questions:

1.  Is there any easier/more logical way of bringing in large amounts of data to manipulate?

2.  Is there a simple way to change the data to signed data, which would save me a lot of conversion routines?

3.  Is there a way to bring data in and dynamically (before runtime, so dynamically might not be the most appopriate word here) set that up to a known structure?  (I have an ASCII text file with the structure of the data which includes the parameter names and data types)  This way, I could change the structure of the data in the text file, and the VI would resize appropriately without having to realign the 350 or so parameters if we inserted something in the middle.  I would like to use something like the System.Collections.Generic Dictionary class in C# so that I can access a variable by name instead of by array index.

4.  Would it be better to attempt to write a .NET assembly to handle this, or possibly LabWindows?  (I'm still not exactly sure everything that LabWindows can do or should do)

Thanks in advance for your help.

Stephen Jenkins

[unclebump]

We can provide much better help if you post your code and some sample data.

[SJenkins9]

I am currently in the process of writing a .NET assembly to handle the UDP functionality and populate properties to pass into Labview.  Unfortunately this wil create a list, upwards of about 1000 parameters that labview can then access.  The labview code is really basic as of now so I don't see the value in posting it.  I could possibly post the .NET code after I have finished if someone is interested.

[Pcorcs]

Hello Stephen,

Might inquire on your selection of UDP for data transfer? Typically, for large amounts of data TCP/IP is preferred over UDP, as it employs 'handshaking' to ensure packets are properly sent and received. The TCP/IP protocol will ensure that you can achieve lossless data transfer, as well as ordered data transfer. UDP information is simply broadcast, which makes the data received entirely dependent on the client listening. Perhaps you have a specific use for UDP, but addressing question 1 TCP/IP would be preferred. For getting started with TCP/IP there is a reference library of VIs designed for data communications please see the link to: Simple Messaging Reference Library (STM).

With regard to question 2, I'm not sure what you are specifically referring to in changing to signed data? You later mention a ASCII file with multiple data types included, do they all have the ability to be signed data? Posting a sample data set would assist in providing a better assessment on the system. 

Question 3, is this ASCII formatted file a database? Would this system benefit from database management? How many different data types are being collected? Overall, how much data will processed and stored using this software?  

Question 4 will entirely depend on your comfort level with each language. If the goal of the project is to migrate to LabVIEW driven applications, it seems that building functions into .NET would work against driving the application with LabVIEW primitives. To truly evaluate LabVIEW as a language, I would encourage you to explore the STM library for the data communication. Or, develop a VI set to handle your communications, as opposed to evaluating LabVIEW through customized .NET functions in the constructor node.

I hope this helps move in the right direction. Example code no matter how small is a helpful picture of the system, especially in LabVIEW. 

 Cheers,  

[SJenkins9]

Thanks for the reply Patrick,

We have our business model set up in such a way that some initial Sun workstations process, store, and then pass the data onto another set of PC's in the form of ethernet data.  The ethernet data is packaged up at 10 sps, while the original data to the Sun stations has a much faster incoming data rate.  This being said, we aren't worried about extremely reliable data transfer to the PC's running Labview as all the data is stored in its raw format prior to Labview.  We use UDP because we have a variable amount PC's listening to this broadcast and don't want to worry about the overhead of TCP/IP.  It is quite possible that we could replace our Sun Stations with Labview, in time.

After becoming more familiar with Labview, I can see that question 2 about signed data was a little premature.  The signed data is no longer an issue.

The ASCII file just gives us the type and structure of the incoming data so we that know how to accurately parse and display that information.  The data is stored prior to Labview on the Sun workstations.  Labview will be used to run some algorithms and display the information real-time, in a human readable format.  From what I can tell, it will be easier for us to write a .NET assembly that will give us the ability to bring the data in through a constructor and property node.  The property node then has the named parameters, instead of an indexed array, which is easier to read and troubleshoot.  This gives us sort of the same functionality that a Systems.Collections.Generic Dictionary object would.  This is what we prefer as we will be bringing into the Labview application anywhere from 50 to 3,000 parameters at a time, which create rather large lists to deal with.

For question 4, I guess I feel comfortable writing .NET functions to handle certain things and not quite as comfortable in Labview yet.  Eventually, as I become more familiar with Labview, I will migrate everything over.  I ask this question as I am used to using .NET tools and was curious to see if such tools were also available for use in Labview, such as Dictionaries.

I can see quite clearly that we aren't using even 10% of Labview's power, but that should change as we become more comfortable with it and discover more of its potential.

I started this thread as I couldn't find any really good examples for handling large amounts of ethernet data in Labview and was hoping to be pointed in the right direction as to which tools would be best utilized and what would be best way to go about it.

[Ben]

---

SJenkins9 wrote:

...

I started this thread as I couldn't find any really good examples for handling large amounts of ethernet data in Labview and was hoping to be pointed in the right direction as to which tools would be best utilized and what would be best way to go about it.

---

Ethernet does not make much difference once it bubbles up through the stack.

See here for Dr. Grey's article on managing large Data sets in LV.

This Tag cloud links threads about memory usage.

This tag cloud links threads about performance.

Is that waht you wanted?

Ben

[Mark_Yedinak]

For the data sets I would recommend using LVOOP (LabVIEW Object Oriented programming) and plugins. You could swap the necessary plugins which would decode your data correctly. In addition, rather than use an array for your data you should look at using typedefed clusters. This allows you to access the specific elements you want, is very readable and is easy to update and maintain. Based on your description so far I see no reason to introduce the .Net layer to solve your problem. It can all be done using native LabVIEW.