How to know the buffer size before viRead ?

[gdargaud]

Hello all,

if I call viRead, I need to know the buffer size beforehand. Which is currently a problem because I'm reading buffers that are sometimes 500Mb and sometimes 1kb depending on config. I'd like to do dynamic allocations and not have tons of huge static buffers in my code.

I don't see any viGetBuf* functions in Visa.

The only thing I can think of is to do a blank shot, iterate viRead with a small temp buffer until the end is found, and return the sum of the sizes before calling it a second time, hoping the the size stays the same.

Any better idea ?

[rolfk · Accepted solution]

This is about device handling and understanding the protocol for a specific device.

It seems unlikely that you have one time a few bytes and then 500 MB for the same device. That makes little sense for any device I can think off. 

So you need to understand the protocol your device is using.

There are two main cases here:

1) The expected return data is terminated with a specific character. This is the default for most measurement devices and well supported by VISA. However it is EXTREMELY unlikely that a device would use that for multi mega byte data transfers for several reason. First such huge data is almost inherently binary, so using a specific byte value as termination character is usually not possible. Second it has the problem you encounter that in order to be able to find that termination character you have to either allocate huge buffers or iterate many times with a smaller buffer until you find that termination character. If your device uses this method for multi mega byte data, its developer should be punished hard.

2) Your protocol contains implicit or explicit transfer size information. Implicit means that the protocol elements are predefined and already knows based on the data requested, explicit means that there is some form of header that contains the size of the data to follow. In this case you read the short header information into a fixed size buffer and interpret it and then do a dynamic allocation of the buffer for the required size and read the remainder into this buffer.

If your device does not fall into these two categories, you have my sympathy and you should track down the developer of it and talk a serious word with him about his developer qualities.

[gdargaud]

Well, you are of course right on both points, but...

- it's a high-res scope that depending on configuration will return huge buffers... or tiny ones (for instance an histogram or even just a status)

- I know the protocol and indeed when you parse it you can compute (with some effort) the full buffer size from the metadata contained at the beginning.

But doing that means you can't have a clean architecture when you acquire or read from file the full buffer and then hand it to the parsing function. You need your parsing function to also be able to understand a partial buffer.

Hence my question about a hypothetical GetBufSize()... It would simplify coding.

[rolfk]

And how would you expect VISA to know how much of data will be there without it reading it and storing it somewhere in an intermediate buffer, potentially blowing up your computer if that buffer reaches your process memory limit?

VISA can’t perform magic and can even less know about your specific device than you. You “could” calculate the required buffer size from reading that header as you know what device you talk to, VISA can’t. As far as VISA is concerned it is some device returning some data. What data and how much data it can not even start to guess.

[gdargaud]

Well, I don't know, that's why I ask   😉

On Linux you can often read to a /dev/null buffer where the data is read and discarded, there might have been such a trick here too...

[rolfk]

How would that help? Don’t you want to use the data at all? After the data has been sent to /dev/null it’s gone, nada, futschi. With some effort you may count how much data was sent to /dev/null but even that comes not for free. /dev/null is literally a data black hole. Once data was sent to it it only exists in the n-dimensional possibilities of the universe, which even our limited quantum knowledge can’t access.

[gdargaud]

Well, one run to get the size, another one to allocate the buffer and get the data...

[rolfk]

Let’s get take your original extreme example! You are seriously proposing to download the 500MB your device provides twice just to save a little bit of effort in your code to use the actually available information in the header?

Sounds to me a bit like I’m to lazy to think a little bit upfront to make a shopping list and rather drive twice to the supermarket to get the items I forgot to buy the first time around!

[gdargaud]

Well, I still need to read and parse it the 1st time......... Hoooo I see it now. Instead of allocating the buffer outside the calling function, I read part of the header to get the size, allocate, read the rest and return the buffer. Yeah. I understand quickly, but you have to repeat many times as my gramp used to say.