POST data consumes more download data bandwidth than upload

[ColinCR]

Hello all.

I'm using the POST.vi that works great for sending data via an API that consumes it and responds with a status message.

The issue that I've run into recently, when running this on a cell modem where data is expensive, is that the POST operation consumes FAR more data than the sum of the post and response, including headers. Something like 5-10x. More concerning is that when I use a bandwidth analyzer tool that monitors by application, I find that while the upload quantity is in line with what I'm estimating, I'm seeing 4-5x this in download. I see this on various networks, so it does not appear to be related to that.

Any ideas why this would be happening?

Thanks,
Colin

[Kyle97330 · Accepted solution]

Have you tried using Wireshark or similar to see the actual contents of what it's sending and getting back?

The built-in LabVIEW "POST" VI (the entire HTTP library in fact) is unfortunately password protected so there's not much that can be done if the problem isn't with the content that you're posting.

[ColinCR]

Hi Kyle,

That's a good suggestion. All the post is ssl, so I'll have to sort that, but that seems like a good place to look.

Colin

[ColinCR]

Figured it out. It's all TLS handshakes.There's a huge overhead for each transaction. The solution is either to go without encryption, or store up a log and send. I have it set to chunk data at a pretty low threshold, but this finding will encourage me to increase the chunk size to whatever is reasonable, to minimize the number of transactions.

[rolfk]

---

@Kyle97330 wrote:

Have you tried using Wireshark or similar to see the actual contents of what it's sending and getting back?

The built-in LabVIEW "POST" VI (the entire HTTP library in fact) is unfortunately password protected so there's not much that can be done if the problem isn't with the content that you're posting.

---

It's simply calling into a LabVIEW private DLL, which is a fairly thin wrapper around an NI build of libcurl. It would surprise me if the seen behavior would be caused by the NI wrapper but who knows. A test with a libcurl client such as the curl command line tool might shed some light into that.

Note: I see you already pinpointed the culprit to the TLS handshake protocol, which is even a layer beyond libcurl, which uses OpenSSL for that stuff.

[ColinCR]

Yes please see my above reply. There is just significant overhead with the TLS transactions. Please see attached a screenshot of the usage before and after going to the http endpoint of the API instead of https. 

[LLindenbauer · Accepted solution]

Do you open and close the connection every time you POST? Keeping the connection open might help with traffic.

[ColinCR]

Good question, and good idea. I modified the program to optionally create and destroy the handle and session. It makes a huge difference for https, but next to none for unencrypted sessions. For a test case, please see data below. All transfer values are measured, but map well to simply counting ascii characters for the uploaded body and headers. "Create" indicates whether the a handle was opened and closed for each post session. When this is false, the handle is persistent unless empty or there is a post error. Anyway, thanks for the help! Useful.

		Measured kB		kB	kB	MB	MB
Create	SSL	Upload	Download	total	per hour	per day	per month
Y	Y	1.9	5.1	7.0	420.0	10.1	302.4
N	Y	1.3	0.622	1.9	115.3	2.8	83.0
Y	N	1.2	0.411	1.6	96.7	2.3	69.6
N	N	1.2	0.411	1.6	96.7	2.3	69.6