05-09-2019 10:44 AM
Hi there,
this is my first post in this forum, as I looked to find a solution but reached a point where I really dont know what to try next. I attached three different files.
Setup description:
I use LabVIEW and a cDAQ device to capture some data. I also control various pieces of hardware through LabVIEW via serial connection. That all works fine via LabVIEW using a single main VI and multiple subVIs. Additionally I want to capture some data with some sensors powered and controlled by an Arduino. The Arduino then prints to the serial port and I use LabVIEW to read the data from the serial Port.
Problem description:
The Arduino prints to the serial port the data (basically every 56microseconds). Every time the loop finishes printing the data ("X1,Y1,Z1, ... , \r\n) a linefeed (\r\n) is send at the end. First I made a VI to just test the functionality of reading in the data independent of the main VI. This VI works perfectly fine (even though its running a bit slow (4ms execution time for the while loop) and I let it run for a few hours capturing data with no problems. I then ammended this VI to work in the main VI. This requiered the following changes:
When I start the main VI everything starts fine in the subVI (COM port is initialized and the VISA Read function starts reading in the data correctly and adding it to the queue, the loop also executes every 4ms). However, after 2-5 minutes (this seems to be random) an error occurs at the VISA Read function (Error -1073807339 VISA Read Timeout) and it stops reading in data. I tried to reinitilaze the serial port once this error occurs automatically... then it will work fine again for a couple of minutes, but I loose some data as the reinitilazation takes place and eventually the error happens again. I removed the queue in this version to see whether this has any effect but the failure occurs as well. I slowed down the loop to 100ms to investigate if the failure still occurs and it does (though maybe a bit later). I referred to the error description here: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019L3mSAE&l=en-GB but I could not find a solution as my connection works in the beginning and then the error occurs at some point randomly. But as I mentioned before, this only happens if I use it as a subVI. The rest of the main Vi and subVIs continues to work perfectly fine. I wonder what it could be, as this problem only occurs if running this as a subVI in combination with the main VI? Is this maybe a problem with not having enough memory (but windows Task manager shows only 20% of CPU is used)? Or could it be that a buffer overflows?
Also, this is another question, but is there a solution to let the LabVIEW while loop run faster so I could capture the data at a higher rate?
Kind regards and thank you for your help in advance
05-09-2019 11:04 AM
I looked only at the LabVIEW code as my text code knowledge is archaic and lacking.
There really are no substantive differences in the code I saw that could explain why the subvi version eventually results in VISA timeouts. Thus, I conclude (tenatively) that the problem lies in code I *didn't* see.
When you run as a subvi, your the VISA COM port resource name is passed in by wire. Does your main vi branch that wire off to do anything else with it anywhere? That "anything else" could be interfering...
-Kevin P
05-09-2019 11:17 AM - edited 05-09-2019 11:22 AM
Congratulations on a very nicely written and informative first post, complete with all of the useful things to help you.
Unfortunately I don't have LabVIEW 2018 installed, so I can't open your VIs, but here are my first thoughts...
My guess is that when you run the code as a subVI, it shares access to some I/O resources, possibly via the VISA driver, and so the loop is unable to keep up with the communication from the Arduino.
After a period of time that depends on how badly it's falling behind, the buffer overflows and then the Read Timeout occurs. It's not obvious to me that this is possible when my suggested hypothesis is that you're overflowing with data... so my next questions become: how large is your buffer, how many bytes are you attempting to read each iteration of the loop, and is it possible that a warning occurs regarding reading without emptying the buffer, but that this warning is overwritten by the error which occurs quickly afterwards? (These are all questions I could answer if I opened the code, so again my apologies)
It's certainly possible in principle to run loops faster than once every 4ms, but without seeing I can't guess what you might be able to change. The baud rate combined with message length make me feel like the time taken to receive the message should be something like maybe 70*8/115200 = 4.861111ms, which actually given I guessed the message length based on the components is pretty close to 4ms... Maybe it can't be faster.
Edit: This actually brings me to a separate question - how sure are you about the loop time on the Arduino, and are the sequential data sets independent (i.e it's not just spamming the same data?). I'd guess this uses the TCNT1 timer, but I'm going to have to google that now...
05-09-2019 11:39 AM
As a troubleshooting step, I'd suggest the following two possible modifications:
05-09-2019 11:57 AM
Hi cbutcher,
thank you very much for your answer.
please find attached some screenshots of the VIs.
Regarding: This actually brings me to a separate question - how sure are you about the loop time on the Arduino, and are the sequential data sets independent (i.e it's not just spamming the same data?). I'd guess this uses the TCNT1 timer, but I'm going to have to google that now...
--> I am not sure if I understand that correctly, but yes I use the TCNT1 timer and the delay of the loop is 56.
@cbutcher wrote:
As a troubleshooting step, I'd suggest the following two possible modifications:
- Check the Bytes at Port value periodically during execution. If this number is continuously increasing, you'll eventually have buffer overflow (checking this will slow down your loop, making the problem worse, so don't leave it in if there is a problem)
- Make the buffer larger (Setting the Serial Port Buffer Size for Receiving or Transmitting Data with VISA) and see if the time to fail is longer. If you prefer to save time, make it smaller and see if it fails faster.
I will implement these points. Also tomorrow I will have a look with the NI Serial Spy tool of whats happening and will post the results.
05-09-2019 11:59 AM
Hi Kevin_Price,
thank you for your answer....
When you run as a subvi, your the VISA COM port resource name is passed in by wire. Does your main vi branch that wire off to do anything else with it anywhere? That "anything else" could be interfering...
In the main VI version the COM Port is red in through an .init File, as all the others. But its not used anywhere else. Also I tried making it static and fixed it to a certain value (COM4 in this case) in the subVI but
05-09-2019 12:08 PM
@perry_tiff wrote:
Regarding: This actually brings me to a separate question - how sure are you about the loop time on the Arduino, and are the sequential data sets independent (i.e it's not just spamming the same data?). I'd guess this uses the TCNT1 timer, but I'm going to have to google that now...
--> I am not sure if I understand that correctly, but yes I use the TCNT1 timer and the delay of the loop is 56.
I meant that I imagine you're using the value of the TCNT1 timer, via timer1_counter and the Serial.print to determine the rate of the loop. Is the value being printed 56?
You're using a 1024 multiplier with the clock rate - did you account for this in your determination of the loop rate (you seem to be careful, but better I ask a silly question than ignore the possibility).
Thank you for the screenshots. I'll look through them more carefully but I also don't see any problems at a first glance. Any warnings created at the VISA Read (for example due to remaining values in the buffer and a non-terminated read) wouldn't appear though - it will wait for an error to cause a stop, and warnings don't read as True when evaluated for a stop condition.
05-09-2019 02:09 PM
You're attaching timestamps to each serial line as you receive it in LabVIEW. Any clues available from the time intervals, or any suspicious-looking serial text?
One plausible-seeming kind of interference would be that some other part of the main app starves the subvi from getting enough CPU to keep up with reading, but I wouldn't expect that to lead to a *timeout* error. But just to rule out the notion that "running as a subvi" is all by itself the problem (I'm very skeptical of this), have you tried the simplest possible main app that does nothing but call the subvi and then enter a loop that dequeues and discards everything the subvi puts into the queue?
-Kevin P,
grasping at straws
05-09-2019 08:43 PM - edited 05-09-2019 08:50 PM
So I have a new "grasping at straws" hypothesis - I'm still not convinced that you can actually get data out of the Arduino faster than about 1 iteration every ~4ms, but reading the documentation for Serial.print (unlike Serial.write, it seems) informs me that Serial.print(..) is asynchronous and will return before transmission. This would definitely (at least initially) allow the Arduino loop to run much faster, filling the Arduino buffer.
Unfortunately, I'm not sure what happens at that point. It seems the write buffer might be just 64 Bytes (which I'd have guessed is less than your message length, but I'm not sure - probably close) and then Serial.write would block. I can't find out what buffer Serial.print uses though - perhaps it's willing to continuously increase usage of the SRAM until it runs out of memory (perhaps this takes 2-5 minutes? I haven't tried to calculate, and would need to know which Arduino you're using).
You can slow down the Arduino loop by using Serial.flush() at the end. This will block until the write buffer is emptied.
Possibly alternatively, you could switch from .print(..) to .write(..), but then you're writing binary byte arrays rather than ASCII strings (unless you pass a string), which might (precision-dependent, but likely?) reduce your data size with some care (increasing max speed) but would require more coding (mostly on LabVIEW side)
There's also a Serial.availableForWrite() function that returns the number of bytes that can be added to the write buffer, but as I said I'm not sure the location or size of the buffer used by Serial.print(..).
Edit: I should point out the reason why I'm "grasping at straws" here and not particularly confident is I have no idea why it would work happily in one case and not in the other regarding subVIs and main VIs.
You can also try enabling "Synchronous" reads on the LabVIEW side by right clicking the Read node - this will possibly mess with other parts of your program but the LabVIEW help tells me it produces the fastest throughput for that node. I've never used that setting, YMMV.
05-10-2019 05:03 AM
Hi there,
I tried to address all the points mentioned above seperately for the standalone VI and the subVi Version to isolate the error:
--> So I figure its really a buffer problem as you guessed... but honestly my understanding here is not deep enough. Also its not clear to me why this is only happens in the SubVI version. I removed everything like a queue and so on to make them as similar as possible. The only substantial difference is the wired in error function.
Unfortunately, I'm not sure what happens at that point. It seems the write buffer might be just 64 Bytes (which I'd have guessed is less than your message length, but I'm not sure - probably close) and then Serial.write would block. I can't find out what buffer Serial.print uses though - perhaps it's willing to continuously increase usage of the SRAM until it runs out of memory (perhaps this takes 2-5 minutes? I haven't tried to calculate, and would need to know which Arduino you're using).
-> the message is 68bytes I think. I have an arduino UNO.
You can slow down the Arduino loop by using Serial.flush() at the end.
-> using Serial.flush()
--> reducing sent datasize to 5 bytes and keeping Serial.flush()
So thats what I tried so far ...