Real-time data Processing

[ap8888]

My sensor hardware outputs a data file with simple tabulated data at about 250Hz. I need this data to be read, mathematically manipulated (e.g. simple matrix multiplications to manipulate the read data), and then outputted.

(a) How would I go about doing this? Is the standard LV package sufficient? What tutorials should I look at?

(b) How quick would the 'read-manipulate-output' sequence be?

Thank you,
Alex

edit: To clarify, the original generated data file is constantly updated, and I need my analysis to be done (more or less) "in-sync" with this generation.


Message Edited by ap8888 on 06-02-2008 01:11 PM

[Cory_K]

Once you read in the data, put the entire sequence in a while loop.
Then if you put a wait (set to 250 Hz) it will read in data 4 times/sec.

Set up your matrix mathmatics, whatever you need for your operation, also inside the loop.

Lastly, make sure your output (ie 'write to ...' or display) is also in the loop.

The loop will ensure that data is read, manipulated, then outputted 4 times per second.

The sequence will run much faster than you have to worry about (ie within milliseconds)

- Cory

[ap8888]

Thank you for your help.

Does anyone else have further advice or other methods of fulfilling this  job? Also, what are the best ways of reading a changing data file? Is there a way to only read the last/latest line of a data file?


Message Edited by ap8888 on 06-02-2008 02:09 PM

[ap8888]

Would someone mind offering a little more advice?

I would really appreciate it.

Thank you.

[Norbert_B]

---

@ap8888 wrote:
My sensor hardware outputs a data file with simple tabulated data at about 250Hz. I need this data to be read, mathematically manipulated (e.g. simple matrix multiplications to manipulate the read data), and then outputted.

(a) How would I go about doing this? Is the standard LV package sufficient? What tutorials should I look at?

(b) How quick would the 'read-manipulate-output' sequence be?

Thank you,
Alex

edit: To clarify, the original generated data file is constantly updated, and I need my analysis to be done (more or less) "in-sync" with this generation.


Message Edited by ap8888 on 06-02-2008 01:11 PM

---

Alex,

there is no general answer to your questions. But i will give you some hints for further investigations:
a) How steady do the 250Hz to be acquired? Does it matter if you have delays of several ms up to seconds? If it matters, you definetly don't want to go with Windows or Linux. You have to switch to real time or FPGA.
b) Do you want to realize a control loop? If not, do not acquire single values! Acquire arrays of values and fetch them about once per second (which gives you plenty of time for your mathematics).
c) How long does your mathematics take? If each value is manipulated for a time >4ms, you will fail to keep your timing, regardless of which approach you choose to implement. If it is like that, you can only either improve the mathematics or you have to analyze the data offline (from file).
d) Have you ever implemented any continuous or finit data acquisition? If not (i asume so), please take a look into the example finder of LabVIEW for a fitting example. If you have DAQmx installed, there will be plenty of different examples available.....

hope this helps,
Norbert

[ap8888]

a) I'm not sure what you mean by "how steady". I haven't setup the hardware yet, so please bear in mind that I am merely planning for the future. However, I assume that the data is continously ("steadily") acquired at this rate. Delays of seconds are not acceptable, but for a beta version of the hardware implementation would be tolerated. Therefore, I am willing to sacrifice the loss of absolute "real-time" if I could at least show the software is acquiring and processing data. Besides, I do not think I'm ready to use a dedicated real-time machine.

b) What do you mean by "single values"? My hardware is connected via ethernet, and it automatically generated a data file on my PC. I believe this data file is formatted in a typical tabulated column format.


Edit: Also, for pedagogical reasons, how come there is "no general answer" to my question anyway?


Message Edited by ap8888 on 06-03-2008 09:29 AM

[ap8888]

Also, I've forgotten to mention that the vendor of the hardware also makes available a LabView executable to view the results coming from the hardware as well as save the results to a data file. So the setup is:

Hardware -- Ethernet --> PC with LabView executable viewing results -- output data file --> PC hard drive

Could I perhaps manipulate this LabView executable to not just view the results and output the data file, but instead perform my calculations and export THAT in the data file?




Message Edited by ap8888 on 06-03-2008 09:44 AM

[Norbert_B]

---

@ap8888 wrote:

Could I perhaps manipulate this LabView executable to not just view the results and output the data file, but instead perform my calculations and export THAT in the data file?

Message Edited by ap8888 on 06-03-2008 09:44 AM

---

Alex,

if you have access to the sources of the exe, sure, you could do that.
I take it that the mentioned "line of acquisition" is already "in place" so you have to add a case:

PC with LabVIEW executable viewing results -- output data file --> PC hard drive
                                                                          |
                                                                          -- manipulation --> Hardware Output

It is not easy to advise someone if we don't know, what hardware is involved. But as i read it, it's some kind of "Black Box" supplying the PC with data using an executable which already CAN log the data to file. Is that correct?
If yes: why not use the file to do the math?

Norbert

[ap8888]

It is an optical sensing device which outputs data in the format outlined above. And yes, the source code is supposedly available. So....

But I suppose you're right; I could just manipulate the source code of the provided LabView executable to perform these calculations.

Just to clarify, however, is the following also possible (this is actually what I meant to happen)?

PC with LabVIEW executable viewing results -- output data file --> PC hard drive
                                                                          |
                                                                          -- manipulation --> PC Hard drive output

----
I guess the question now is, how does perform continous calculations on a constantly-acquiring data sink? Would this just be a loop set to indefinitely read the acquired data (which is taken care of by the supplied executable, like we discussed), perform calculations, and then give the results? If so, what kind of loop would this be and how quickly should it cycle?


Message Edited by ap8888 on 06-03-2008 10:27 AM

[Ben]

Beat on your supllier for the LabVIEW source code. It will make your life a lot easier and improve your results.

Using the file as an intermediate will cause complications and eventually slow-down of performance. Some of the complicaitons include

1) Detecting when new data has been added to the file.

2) finding the "new part"

3) Potential file access conflicts

4) As teh file grows the processing required to read the file and grab the new stuff grow.

So beat on your supplier to provide the code.

Ben