Can LabVIEW handle fast and large amount of data?

[HB.]

Hello,

I want to develop server for GPS Tracking system. This system will be communication with about thousands of devices at a time. My question is that whether LabVIEW can b used to implement a server on LINUX platform to handle such fast and heavy transactions without having any hanging of software/system. I can use multithreaded processor, probably, Core-2-Quad processor, to build this application. 

Please let me know whether LabVIEW can be used to designed such type of application.

Hasan Baig

[altenbach]

Properly programmed, LabVIEW can handle anything that you could do in other programming languages and it can take full advantage of multiple CPUs. Writing such an high performance applications will probably take some experience and performance tuning, but that would not be different in any other programming language. Using multiple CPUs is definitely easier in LabVIEW, because it is inherently parallel.

It is not clear what kind of communications you are talking about. Is this serial as your "name" might imply? Serial is inherently not a very high performance protocol. What is involved in a "transaction"? Is it mostly communication? mostly computation? If it can be done at all with todays hardware, it can be done in LabVIEW.

LabVIEW certainly scales very well to highly complex problems. For example have a look at the giant telescope control as an example.

QUOTE from the link: "One such area is advanced, high-speed control.  LabVIEW is uniquely positioned to combine advanced algorithms with best-in-the-world I/O capabilities.  Combine these algorithms with the parallel/multicore capabilities of the LabVIEW programming language – and you’ve got some real power."

[HB.]

It is TCP communication and transaction contains the GPS frames that contains Latitude,Longitude,date.....etc information transaction frames may have maximum of 256 bytes of data. Do I need some special consideration in programming while handling such application?

[Ben]

Assuming your network hardware is not a bottleneck....

Design the application such that there are a minimal number of shared resources (TCP file etc) and make it modular to start. Nenchmark your code as you go so you don't run into latter when scaling it up.

Architecture will make the difference between a good and limping solution. if you don't know how to design the architecture, it would be a good idea to contract with someone that can. I'm not saying get someone else to write it, only design it.

Ben

[altenbach]

I agree with Ben that a careful design will be important for success. Does the communication need to be very deterministic?  What is the rate of packets? Regular intervals or bursts? Is it mostly incoming traffic or outgoing traffic? Is each transaction a new TCP connection or does each device maintain a connection for extended periods.  Do you need to guarantee that all communications succeed or is some loss OK?

A protocol that is not connection based (UDP) might be sufficient and could offer less overhead. Remember that each TCP connections needs several packets to setup, maintain and break down (Threeway handshake, ACKs, FIN, etc) while UDP packets can just be placed on the wire.

Are the connections over a wide area (worldwide) or confined to the local network?

How much control do you have over the protocol. Do the clients follow some industry standard protocol or do you have free choice, writing both sides (clients and server) in LabVIEW from scratch? 

Message Edited by altenbach on 03-25-2010 08:21 AM

[HB.]

Each computer is sending the string of 256 bytes each after 5secs on the same port of the server, and the number of computers are roughly around 1000. its Local area network

[HB.]

Protocols that I am following is MODBUS over TCP/IP

[GoCougs]

As discussed previously, you should be able to implement your application using LabVIEW, but you should take care that you design the code properly.  From what I understand, you need labview to be able to process ~256kBps.  LabVIEW will have no problem doing that, but the problem may lay in getting the information from ~1000 computer in 5 seconds.  This will all be determined by:

1. Your network hardware and set up.

2. The code used to pull the data from the network.

There is a library available for using MODBUS with LabVIEW, I suggest reading up on it and learn more about the API of the library.  You can find all that information here: https://forums.ni.com/t5/Reference-Design-Content/LabVIEW-Modbus-API/ta-p/3524019

If you would rather it be an easy, not too much time required, out of the box solution there is the DSC module that is available for LabVIEW.  This greatly simplifies your application and also makes data communication very effecient.  You can find more information about the DSC module here: https://www.ni.com/en-us/shop/product/labview-datalogging-and-supervisory-control-module.html

Just as a point of advice, the DSC module will most likely save you anywhere from 1 to 2 weeks of solid work, depending upon your experience with LabVIEW.

[Ben]

---

Scott W wrote:

As discussed previously, you should be able to implement your application using LabVIEW, but you should take care that you design the code properly.  From what I understand, you need labview to be able to process ~256kBps.  LabVIEW will have no problem doing that, but the problem may lay in getting the information from ~1000 computer in 5 seconds.  This will all be determined by:

1. Your network hardware and set up.

2. The code used to pull the data from the network.

There is a library available for using MODBUS with LabVIEW, I suggest reading up on it and learn more about the API of the library.  You can find all that information here: https://forums.ni.com/t5/Reference-Design-Content/LabVIEW-Modbus-API/ta-p/3524019

If you would rather it be an easy, not too much time required, out of the box solution there is the DSC module that is available for LabVIEW.  This greatly simplifies your application and also makes data communication very effecient.  You can find more information about the DSC module here: https://www.ni.com/en-us/shop/product/labview-datalogging-and-supervisory-control-module.html

Just as a point of advice, the DSC module will most likely save you anywhere from 1 to 2 weeks of solid work, depending upon your experience with LabVIEW.

---

A couple of notes on the previous (sorry Scott but people tell me they appreciate me being honest)

1) The Modbus Libraries at one time (I don't know it it has been fixed) was a CPU pig because there where no "zero ms waits" in any of the loops. It takes some time but it can be fixed.

2) Although DSC may save 1-2 weeks if we don't know LV, it ends up costing me 1-2 weeks of effort every time I go through a major upgrade ( I have been suppoting DSC apps since BridgeVIEW 2.1). PLUS since all of the DSC functions are hidden protected burried.... I have no option to fix DSC. I also believe the Modbus in DSC is going to come through the lookout protocol driver which is supported out of (?) Singapore and I am reduced to e-mail support. Does DSC run on Linux? The files system used to support DSC is complex hard to find out what does what and when last I checked, Security software steps on it functionality with no sign that logging has stopped until we go back to look for it and find out is its not working. It cost one of my customers almost $5K to pay me to trouble shoot why the DSC history was failing on five of thier machines. In the end all I could tell them was, "Unplug the 1st network cable before starting a test." And to the speed of DSC... I find it interesting that the White Paper comparing DSC with Datasocket reads has disappeared from the NI site.

So....

Code it yourself so you can fix it.

Ben

[HB.]

Thankyou Scott and Ben. I'll see what I can do. I'll start programming soon.