11-30-2014 08:30 AM
Hello.
I have a NI USB-6003 reviecing a signal continously. This signal is then converted to true/false with a simple "greater than"-function.
I would like to write this true/false value to a file or array every x millisecond, to achieve a proper timing with the signal sent I also want to delay the initial read with x/2 ms.
Example; I send a true signal for x ms, then a false for x ms. The first write should occur at x/2 ms after the first true value has been detected and then it should write at every x ms.
My attempts at simply using a wait for next ms-multiple inside a loop with a "write to file"-VI have failed, it seems like I can't get the values to pass through the loop.
Any ideas?
Thanks for your time.
11-30-2014 09:42 AM
@Reduts wrote:
Hello.
I have a NI USB-6003 reviecing a signal continously. This signal is then converted to true/false with a simple "greater than"-function.
I would like to write this true/false value to a file or array every x millisecond, to achieve a proper timing with the signal sent I also want to delay the initial read with x/2 ms.
Example; I send a true signal for x ms, then a false for x ms. The first write should occur at x/2 ms after the first true value has been detected and then it should write at every x ms.
My attempts at simply using a wait for next ms-multiple inside a loop with a "write to file"-VI have failed, it seems like I can't get the values to pass through the loop.
Any ideas?
Thanks for your time.
I have several questions. First, you are writing True or False values to a file. I'm unclear how time is being represented here. Your example gave an X ms "True" state followed by an X ms "False" state, but then what? Are all transitions X ms apart? Are you setting the sampling/writing to X ms? Is X a "variable", determined by the first False-to-True-to-False pulse width?
It seems logical to me that time must be part of your problem, but it wasn't clearly defined. To see why, assume you simply write "True" whenever the signal goes from False to True, and False when it goes from True to False. Your output file will simply be "True, False, True, False, ...", and the only significant information in the file will be its length, or the number of transitions you measured (simpler just to write out that one number).
On the other hand, if this is supposed to be a regularly-timed sampling and writing exercise, how do you sent the sampling interval? It would seem that you might set it to be X, determined by the width of the first True pulse. Are all of the pulse widths exact multiples of X? If so, you'll end up with a sequence something like True, False, False, True, True, True, False (for pulses of width 1, 2, 3, unknown). On the other hand, if the first pulse is not the "unit" of pulse widths, then you may well miss a transition.
Another way to handle this is to sample faster than X, say, every millisecond. Now, by looking at the sequence of True and False, you can reconstruct (to the nearest millisecond) the pulse train, with each pulse's width explicitly determined by the number of True and False elements. The output format has the advantage of being "simple", but could be very large, particularly if the pulses are long -- a more compact format might be to write a cluster whose first element is the pulse value (True or False) and second the pulse width (an integer number of milliseconds).
For the final part, why does the writing of the output file need to be "timed" to the gathering of the data? Once the file is written, it is "static", and any timing information has to exist in the file contents (as suggested above). This being the case, you should consider a Producer/Consumer pattern of two independent loops, connected by a Queue. The Producer loop gets the Sample data at regular intervals (determined by your sampling hardware) and puts the data on a Queue. The Consumer loop, a While loop, dequeues the data (as fast as it arrives) and "does something" with it, in this case, writes it to the file. The virtue of this structure is that the two loops run independently at their own speed, and as long as the Consumer loop runs, "on average", faster than the Producer, it can "keep up" and not miss any of the incoming data points. Note that the Consumer doesn't need to always be faster than the Producer -- for example, sometimes writes require a little extra time because directories get shuffled, or something like that, but the Queue has some "elasticity" that can absorb extra elements.
Bob Schor
11-30-2014 02:31 PM
Hey there Bob, good questions and thoughts.
I have several questions. First, you are writing True or False values to a file. I'm unclear how time is being represented here. Your example gave an X ms "True" state followed by an X ms "False" state, but then what? Are all transitions X ms apart? Are you setting the sampling/writing to X ms? Is X a "variable", determined by the first False-to-True-to-False pulse width?
All I need in the file is the sequence of 1's and 0's that the true/false values would represent. All signals are x ms long, but there can be more than one true/false value in a row, which is why I want to write at a specified interval. X in this case is a "constant", determined by how long I set each value to be sent.
It seems logical to me that time must be part of your problem, but it wasn't clearly defined. To see why, assume you simply write "True" whenever the signal goes from False to True, and False when it goes from True to False. Your output file will simply be "True, False, True, False, ...", and the only significant information in the file will be its length, or the number of transitions you measured (simpler just to write out that one number).
Agreed. But not what I'm after.
On the other hand, if this is supposed to be a regularly-timed sampling and writing exercise, how do you sent the sampling interval? It would seem that you might set it to be X, determined by the width of the first True pulse. Are all of the pulse widths exact multiples of X? If so, you'll end up with a sequence something like True, False, False, True, True, True, False (for pulses of width 1, 2, 3, unknown). On the other hand, if the first pulse is not the "unit" of pulse widths, then you may well miss a transition
Yes, all pulses are the same width/length, or atleast intented to be.
Another way to handle this is to sample faster than X, say, every millisecond. Now, by looking at the sequence of True and False, you can reconstruct (to the nearest millisecond) the pulse train, with each pulse's width explicitly determined by the number of True and False elements. The output format has the advantage of being "simple", but could be very large, particularly if the pulses are long -- a more compact format might be to write a cluster whose first element is the pulse value (True or False) and second the pulse width (an integer number of milliseconds).
This is more or less my idea, sample faster than x but only write "in the middle" of each pulse, as a way to eliminate small timing discrepancies. Thats why I wanted to delay the first write with x/2 ms and then write at every multiple of x. To write as a cluster with pulse width seems like a good idea, I could then use the pulse width to determine how many true/false values I've recieved. But sadly I don't know how. I'm quite new to both LabVIEW and programming in general.
For the final part, why does the writing of the output file need to be "timed" to the gathering of the data? Once the file is written, it is "static", and any timing information has to exist in the file contents (as suggested above). This being the case, you should consider a Producer/Consumer pattern of two independent loops, connected by a Queue. The Producer loop gets the Sample data at regular intervals (determined by your sampling hardware) and puts the data on a Queue. The Consumer loop, a While loop, dequeues the data (as fast as it arrives) and "does something" with it, in this case, writes it to the file. The virtue of this structure is that the two loops run independently at their own speed, and as long as the Consumer loop runs, "on average", faster than the Producer, it can "keep up" and not miss any of the incoming data points. Note that the Consumer doesn't need to always be faster than the Producer -- for example, sometimes writes require a little extra time because directories get shuffled, or something like that, but the Queue has some "elasticity" that can absorb extra elements.
The reason for having the write function work simultaneously is that I want to interpret the data in real time. Reading the file and outputing it's values. It might be an unneccesary step to write the values to a file, but is the only way I've concieved for making this possible.
This producer/consumer pattern seems interesting, but in my mind it should be the other way around from how you described it. The producer loop should output values at a faster rate than the consumer, which at a specified, slower interval should fetch this information and write it
The extra time writing takes has caused problems for me in the past, probably because I've only used pre-existing write VIs which contain alot of functionality I don't need in the same loop as the read function.
I hope this made any sense, thanks again for your time.
11-30-2014 05:08 PM
@Reduts wrote:
Hey there Bob, good questions and thoughts.
This producer/consumer pattern seems interesting, but in my mind it should be the other way around from how you described it. The producer loop should output values at a faster rate than the consumer, which at a specified, slower interval should fetch this information and write it
The extra time writing takes has caused problems for me in the past, probably because I've only used pre-existing write VIs which contain alot of functionality I don't need in the same loop as the read function.
I hope this made any sense, thanks again for your time.
The reason Bob recommends that, generally, the consumer is faster than the producer is because the elements are queued so that data is not lost. If the consumer (your log file writer) is slower than your producer (creating the pulse train) then the consumer will inevitably fall further and further behind, processing data elements that get older and older as time goes by. It is even possible to run out of memory as the queue has to maintain all teh old data that your consumer has yet to process.
12-01-2014 02:00 AM
12-01-2014 07:33 AM
@Reduts wrote:
Well, I would actually not like to que the data, I'd only want to get the last value the producer had recieved..
Then use a Notifier instead of a Queue.
12-01-2014 07:38 AM
@Reduts wrote:
Hey there Bob, good questions and thoughts.
I have several questions. First, you are writing True or False values to a file. I'm unclear how time is being represented here. Your example gave an X ms "True" state followed by an X ms "False" state, but then what? Are all transitions X ms apart? Are you setting the sampling/writing to X ms? Is X a "variable", determined by the first False-to-True-to-False pulse width?
All I need in the file is the sequence of 1's and 0's that the true/false values would represent. All signals are x ms long, but there can be more than one true/false value in a row, which is why I want to write at a specified interval. X in this case is a "constant", determined by how long I set each value to be sent.
It seems logical to me that time must be part of your problem, but it wasn't clearly defined. To see why, assume you simply write "True" whenever the signal goes from False to True, and False when it goes from True to False. Your output file will simply be "True, False, True, False, ...", and the only significant information in the file will be its length, or the number of transitions you measured (simpler just to write out that one number).
Agreed. But not what I'm after.
On the other hand, if this is supposed to be a regularly-timed sampling and writing exercise, how do you sent the sampling interval? It would seem that you might set it to be X, determined by the width of the first True pulse. Are all of the pulse widths exact multiples of X? If so, you'll end up with a sequence something like True, False, False, True, True, True, False (for pulses of width 1, 2, 3, unknown). On the other hand, if the first pulse is not the "unit" of pulse widths, then you may well miss a transition
Yes, all pulses are the same width/length, or atleast intented to be.
Another way to handle this is to sample faster than X, say, every millisecond. Now, by looking at the sequence of True and False, you can reconstruct (to the nearest millisecond) the pulse train, with each pulse's width explicitly determined by the number of True and False elements. The output format has the advantage of being "simple", but could be very large, particularly if the pulses are long -- a more compact format might be to write a cluster whose first element is the pulse value (True or False) and second the pulse width (an integer number of milliseconds).
This is more or less my idea, sample faster than x but only write "in the middle" of each pulse, as a way to eliminate small timing discrepancies. Thats why I wanted to delay the first write with x/2 ms and then write at every multiple of x. To write as a cluster with pulse width seems like a good idea, I could then use the pulse width to determine how many true/false values I've recieved. But sadly I don't know how. I'm quite new to both LabVIEW and programming in general.
For the final part, why does the writing of the output file need to be "timed" to the gathering of the data? Once the file is written, it is "static", and any timing information has to exist in the file contents (as suggested above). This being the case, you should consider a Producer/Consumer pattern of two independent loops, connected by a Queue. The Producer loop gets the Sample data at regular intervals (determined by your sampling hardware) and puts the data on a Queue. The Consumer loop, a While loop, dequeues the data (as fast as it arrives) and "does something" with it, in this case, writes it to the file. The virtue of this structure is that the two loops run independently at their own speed, and as long as the Consumer loop runs, "on average", faster than the Producer, it can "keep up" and not miss any of the incoming data points. Note that the Consumer doesn't need to always be faster than the Producer -- for example, sometimes writes require a little extra time because directories get shuffled, or something like that, but the Queue has some "elasticity" that can absorb extra elements.
The reason for having the write function work simultaneously is that I want to interpret the data in real time. Reading the file and outputing it's values. It might be an unneccesary step to write the values to a file, but is the only way I've concieved for making this possible.
This producer/consumer pattern seems interesting, but in my mind it should be the other way around from how you described it. The producer loop should output values at a faster rate than the consumer, which at a specified, slower interval should fetch this information and write it
The extra time writing takes has caused problems for me in the past, probably because I've only used pre-existing write VIs which contain alot of functionality I don't need in the same loop as the read function.
I hope this made any sense, thanks again for your time.
So I'm responding to the original problem that you posed, but it appears that there are other considerations that you haven't revealed. Your original post talked about streaming data to a file. When doing this, the most important thing, in my mind, is being certain that all of the data are being written, and no points are lost or written in the wrong order. The relative timing of the writes relative to the production of the data is immaterial, only that the data be accurately and completely captured. Hence the use of the Producer/Consumer pattern and the Queue, with the Consumer (the writer) running, on average, faster than the Producer, thereby guaranteeing that no data are lost (unless you run out of file space).
Now you say "interpret the data in real time", something quite different than streaming the data to a file.
You also keep saying that you are sampling every "X milliseconds" -- what order of magnitude is X? Single digits? Hundreds? Thousands? It would help to have a better idea of what rates you are actually considering.
Here's a current project of mine -- I'm sampling 7 analog channels and 9 digital channels at a rate of 1 KHz. I'm streaming this data (via TCP/IP) to another PC, where it is being written (Producer/Consumer) to disk. As the data passes through my Queue, I take every 50th point and display it (which means that the display updates at 20 Hz, and I basically see the last 30 seconds of data scroll past on my display). There are other things going on at the same time (such as stimulus generation), and occasionally the PC needs a few cycles to "get things going", which can cause the display to "stutter" (not necessarily update at 20 Hz) while the PC finishes its other tasks. But no points are lost as they are safe in the data queue, and all of the points show up on the display (most of the time it scrolls very smoothly). The point here is that I'm viewing the data "in Real Time", probably within 10-100 msec of when it is being collected (and 10-100 msec before it is being written to disk), and even if the time delays are somewhat unpredicatable, they are so short that it doesn't matter.
If your problem has tighter time requirements, it seems likely that your approach of writing to disk is simply wrong. Without a better understanding of what you are trying to accomplish, it is difficult to offer advice.
BS
12-01-2014 09:00 AM
So I'm responding to the original problem that you posed, but it appears that there are other considerations that you haven't revealed. Your original post talked about streaming data to a file. When doing this, the most important thing, in my mind, is being certain that all of the data are being written, and no points are lost or written in the wrong order. The relative timing of the writes relative to the production of the data is immaterial, only that the data be accurately and completely captured. Hence the use of the Producer/Consumer pattern and the Queue, with the Consumer (the writer) running, on average, faster than the Producer, thereby guaranteeing that no data are lost (unless you run out of file space).
Now you say "interpret the data in real time", something quite different than streaming the data to a file.
You also keep saying that you are sampling every "X milliseconds" -- what order of magnitude is X? Single digits? Hundreds? Thousands? It would help to have a better idea of what rates you are actually considering.
Here's a current project of mine -- I'm sampling 7 analog channels and 9 digital channels at a rate of 1 KHz. I'm streaming this data (via TCP/IP) to another PC, where it is being written (Producer/Consumer) to disk. As the data passes through my Queue, I take every 50th point and display it (which means that the display updates at 20 Hz, and I basically see the last 30 seconds of data scroll past on my display). There are other things going on at the same time (such as stimulus generation), and occasionally the PC needs a few cycles to "get things going", which can cause the display to "stutter" (not necessarily update at 20 Hz) while the PC finishes its other tasks. But no points are lost as they are safe in the data queue, and all of the points show up on the display (most of the time it scrolls very smoothly). The point here is that I'm viewing the data "in Real Time", probably within 10-100 msec of when it is being collected (and 10-100 msec before it is being written to disk), and even if the time delays are somewhat unpredicatable, they are so short that it doesn't matter.
If your problem has tighter time requirements, it seems likely that your approach of writing to disk is simply wrong. Without a better understanding of what you are trying to accomplish, it is difficult to offer advice.
BS
Yes, I agree that the problem description was incomplete. We are essentially sending the information one bit at a time with a laser turning on and off, reciving it via a photovoltaic cell connected to the USB-6003. A goal would be to have every signal "lenght" be 25 ms, and the sampling rate at a sufficient rate to avoid any small timing descripancies when performing each write, for example 1khz, and then write the first value at 12,5 ms and then every 25th ms.
Again Bob, thanks for trying to help.
crossrulz - I'll check out the notifier.
12-01-2014 02:01 PM
I still don't understand the timing issues. I get that the laser is on or off for approximately 25 msec (strictly alternating? sometimes on for two "quanta" and off for one?) and you want your data to "clock" at the same rate as the laser. But you describe writing to a file, so it should not matter when you sample as long as you are somewhere "in the middle".
Here's the scenario as it seems to me: You have a laser going on/off at approximately 40 Hz (25 msec), but you don't know precisely when. So you start sampling at, say, 1KHz, measure a few transitions, see its about 25 msec, and say "OK, when I see the next transition, I'll wait 12.5 msec to get myself "mid-pulse", then I'll sample at 40 Hz and start saving the data to disk".
Two problems. What if it is firing at 40.5 Hz? You'll "measure" 25 msec, and if you sample every 25 msec, you'll eventually get two transitions in a single sample window. You can "fix" this by always sampling "fast", finding the transition, and saving either the "new" point or the "old" point (this works only when the pulses are all more-or-less the same length, though if they were uneven and you saved the time interval, it would work for any pulse train).
But the bigger problem is so now you have this file you are writing to disk. At some point, you stop everything and look at the file (it's kind of hard to read the file while it's being written). At this point, the data on the file are "dissociated" from the flashing of the laser, so why does it matter whether the first "On" you record in the file happens when the Laser goes On, halfway through the On phase, or just before the Laser goes Off?
Granted, if you wanted to use the data in Real Time to "do something" (like activate a switch), the timing between the Laser's On/Off transitions and your action could be important (though why "midway" is unclear), but that was not part of the Problem Statement.
Anyway, good luck with this. I hope this exercise gives you a better understanding of LabVIEW, particularly how it manages Time.
Bob Schor