Improve data transfer performance

[xive]

I am looking for any ideas for how I could improve my data transfer performance.  I am running this on a pxi-8196 controller but I think the problem is inherently data transfer.  I need to transfer around 40-50 MB/s from an external device to the controller.  The problem being that this just seems to be too intensive for the controller.  My first approach was using a dedicated gigabit Ethernet connection and to use tcp to transfer the data.  The data is split onto four ports of 12MB/s each.  The controller experiences a load of 20% when receiving just one of these data channels.  To receive all the data is a 80% load.  If I combine all the data onto one port I still get around an 80% load.  The other problem is that the external device is sending an array of integers.  I now need to convert the binary string I get from the labview TCP functions into an array of integers in labview.  For each of the ports this puts an extra 12% load on the controller.  So if I were to get all the data and convert it that would take roughly 130% load, obviously a problem.  I have also tried to use UDP and write a simple receiver in c to make a dll but both approaches have had just as bad of results, granted my c application was probably not built the best.  I was wondering if any one had an idea that could maybe give me some performance increase.  Maybe a tcp vi that would read tcp data as an array of integers already, thereby skipping the data conversion overhead, or some other protocol I haven't thought of.  Perhaps a different physical transport system such as usb 2.0 or firewire.  Any help would much be appreciated.  thanks.

[unclebump]

Is the 40-50MB of data stored in a single file/multiple files?? Available in memory only?? Could you use some sort of zlib compression/decompression before/after data transfer?? You can multihome an ethernet card and assign it up to ten different ip addresses. What is the external device?? Can you modify the configuration and program on this device??

[unclebump]

Here are some possible solutions.

https://forums.ni.com/t5/LabVIEW/Optimizing-network-throughput-over-Ethernet-with-TCP-IP/m-p/117842?...

https://forums.ni.com/t5/Measurement-Studio-for-NET/NI-Datasockets-vs-TCP-vs-UDP/m-p/310107?query.id...

https://www.ni.com/docs/en-US/bundle/labview/page/using-labview-with-tcpip-and-udp.html

https://forums.ni.com/t5/Machine-Vision/Transfer-large-image-through-network/m-p/248624?query.id=459...

[unclebump]

Here is some interesting info. Jumbo frames sounds like a solution. Larger packet size.

http://sd.wareonearth.com/~phil/jumbo.html

http://www.psc.edu/~rreddy/networking/mtu.html

http://www.small-tree.com/jumbo_white_paper.pdf

Windows XP/2000 Server/Server 2003
The magical location for TCP settings in the registry editor is HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

We need to add a registry DWORD named TcpWindowSize, and enter a sufficiently large size. 131400 (make sure you click on 'decimal') should be enough. Tcp1323Opts should be set to 3. This enables both rfc1323 scaling and timestamps.

And, similarly to Unix, we want to increase the TCP buffer sizes:

ForwardBufferMemory 80000
NumForwardPackets 60000

One last important note for Windows XP users: If you've installed Service Pack 2, then there is another likely culprit for poor network performance. Explained in [knowledge base article 842264(http://support.microsoft.com/?kbid=842264), Microsoft says that disabling Internet Connection Sharing after an SP2 install should fix performance issues.

The above tweaks should enable your sufficiently fast server to attain much faster data rates over TCP. If your specific application makes significant use of UDP, then it will be worth looking into similar options relating to UDP datagram sizes. Remember, we obtained close to 900Mb/s with a very fast Pentium 4 machine, server-class motherboard, and quality Intel network card. Results may vary wildly, but adjusting the above settings are a necessary step toward realizing your server's capabilities.

Message Edited by unclebump on 01-17-2006 04:05 PM

[xive]

unclebump,

I greatly appreciate all of the work you have put into trying to help me work out this problem.  Just to explain a little more the labview code is running on a realtime device and the sender is a unix machine, I have access to the code running on both machines.  I will try and see if I can work out the jumbo size for the packets, hopefully that will improve performance.  LV 8.0 increased buffer size to 64KB significantly increase the performance already.  Also I found your idea of using compression interesting, I always threw the idea out since I assumed that it would take too much processing power, but perhaps the time saved by sending less data will be worth it.  I will try these two things tomorrow and let you know if they produce any gains.  Once again, thanks.