Numbering data points

[DominicRoberton]

Hi All,

I am trying to get my head around numbers after discovering I have a number of issues with my data logging system which seem to be related to large numbers and after something which Altenbach mentioned to me in this thread.

Basically my two cRIO's seem to start playing up when assigning sample numbers to each sample.  After the 16,777,216th sample the cRIO no longer writes in consecutive numbers, it writes with a gap of 2, but then doesn't write anything so I get the following behaviour:

16,777,216

16,777,218

16,777,218

16,777,220

16,777,220

16,777,222

16,777,222

....

I am writing the numbers as singles, which is not ideal, however I am doing so so that I can feed the data numbers straight into the data point array.  I have attached the vi I use to do this.  You can see the data comes in from my FIFO as an array of singles, it gets reshaped into a n rows by 52 column array and then transposed so that I have 52 rows and n columns, it's at this point that I feed the sample numbers into row 0 of the transposed array, making it a 53 row array.

It doesn't seem to cause any problems with the actual data, but I use the sample numbers in a program to check for continuity (see the other thread for more info).

I am aware that if I change from a Single to another format I won't be able to feed it into the array in the same way that I do now.

Does anyone have any suggestions/ insightful comments which might solve this issue for me?

Cheers,

Dom

[CoastalMaineBird]

You're on the verge of figuring it out.

The definition of SGL includes a 23-bit mantissa in a 32-bit container.

Together with the implied MSB, that makes 24 bits.

2^24 = 16777216

When the value reaches or exceeds 2^24, you cannot hold the entire number precisely. You have 25 pounds of potatoes and a 24-pound sack. Something has to give.

What gives is the least-significant bit (that's why it's called least significant).

The SGL format is providing the best representation it can of your numbers.

SGL is not an appropriate container for a sample #, given that sample numbers are inherently integers.

Use a U32 for 4-billion capacity, or a U64 for even more.

Use an I32 or I64 for cases where "-1" could mean something like "no index specified", or "default".

The fact that you " won't be able to feed it into the array in the same way that I do now" is true, but you're using the wrong tool.

Use DBL to postpone the problem until your sample number exceeds 53 bits.

Use a cluster where you have a U32 for the sample number plus an array of SGL for the data.

Use an array of U32s, plus a 2-D array of SGL.

Why have an array of sample numbers at all?  If you're looking at a graph of data and need to know where this particular feature is, extract it from the array index.

If you're plotting data vs. sample number on an X-Y graph, you're wasting things anyway. Use a waveform graph.

[DominicRoberton]

Hi Steve,

Thanks for your help.

The reason I need a sample number is because my system has to be standalone for a minimum of 10hrs recording at 800 samples a second from two compct RIO devices, I won't go into more details, I have plenty of threads describing the system when trying to solve other problems, but that is 28,800,000 samples, and is likely to increase as the project matures.

I can't use a double as I have limited memory (around 15GB) and using singles I can get around 13hs of data, so going to a double would cause me a memory issue.  If I just use doubles for th esample number then it will coerced to a single anyway and thus bring back the same problem.  If I need to increase the sampling rate (which I probably will do) then I'llhave to change the representation again, but I'll probably then have to use a fixed point number that I define.

I could use a cluster, I'll look into it.

The reason why I want an array of Sample numbers is that I have had issues with the cRIO's dropping out, I need to know if this has happened or not and the easiest way to do this is to number each sample.

[CoastalMaineBird]

 I have plenty of threads describing the system when trying to solve other problems

I'm not going to chase your threads on the subject, if it's not here, I don't know about it.

If I just use doubles for the sample number then it will coerced to a single anyway and thus bring back the same problem. 

It's not clear why you would coerce it back to a single.

It's also not clear why the sample numbers help you detect dropouts.  Where do the sample numbers originate?

Can you do the detection in real time?  As data comes in, check the sample number or timestamp and declare this sample good (fresh) or bad (after a dropout). Record a boolean instead of a SGL.

Or perhaps record the missing sample numbers, rather than the present ones.  In an array of their own.

[DominicRoberton]

It's not clear why you would coerce it back to a single.

Will it not coerce it back to a single if all the other numbers in the array are singles?

The sample numbers are created at the point where the data points are retrieved from the FIFO.  So if the FIFO has 800 samples in it then a for loop runs 800 times incrementing by 1 as it goes.

I am fairly happy that the data collection is fine, but there have been instances in the past where the file save function has failed (although I am fairly happy that I have sorted it)  However, I would still like to ensure there is no discontinuity. 

I am not sure how I would record something that isn't there, if that makes sense.

Cheers for your help,

Dom

[CoastalMaineBird]

Will it not coerce it back to a single if all the other numbers in the array are singles?

Yes, but it's not clear why you would do that.

The fact is, a sample number is a different animal than a data measurement.  While you can do useful tricks sometimes by making it act the same, in your situation, you've discovered the limitations.

The sample numbers are created at the point where the data points are retrieved from the FIFO.  So if the FIFO has 800 samples in it then a for loop runs 800 times incrementing by 1 as it goes.

I don't understand how that helps you detect a dropout. 

I am not sure how I would record something that isn't there, if that makes sense.

You process the data as it goes IN TO the array.  Whatever checking you're doing on the sample number, do it as the data GOES IN, but record the results only, not the raw sample numbers.

Check for sequential-ness and record a BOOLEAN false if OK, or true if skipped.  A boolean is less memory than a number.

Or check for sequential-ness and record the first point AFTER the skip, into a separate array.  If you didn't skip any, this array contains only a solitary I32 = 0.  If you skipped samples 20-29, your array contains a 0 and a 30.

[DominicRoberton]

I don't understand how that helps you detect a dropout.

Ok, I clearly haven't explained myself very well.

I have two cRIO's connected via ethernet, one of these is also connected to a NAS Drive with 16GB of memmory.  I have three points where there is likely to be data drop out, I am assuming that the transfer from the FPGA to the RT is ok on both cRIO's.

cRIO A collects all of it's data in second chunks (52 x 800 Array of singles + 1 x 800 Array of singles which are the data point numbers) and then transfers all of it's collected data to cRIO B via the ethernet using network streams once a second.  This is the first point at which data could be lost.

cRIO B collects all of it's data in second chunks (52 x 800 Array of singles  + 1 x 800 Array of singles which are the data point numbers) then saves both sets of data into 30 second chunks (2 files containing 30 clusters of 53 x 800 arrays of singles).  This is the second point where data has been lost in the past.

cRIO B then transfers via FTP all the files located on it's hard drive to the NAS Drive every 60 seconds.  There have also been issues with the files being corrupt when they get to the NAS drive, but I think I have solved this now.

I hope this explains why I need to check the data continuity post data collection rather than doing it at the point of collection.

However your idea of using boolean indicators has given me another idea.

I will report back later.

Thanks again for your help.

Dom

[DominicRoberton]

Hello,

I don't know whether you are interested or not but I thought I would follow up with my solution.

In the end I simply added an extra row to the collected data into which I put either a 0 or a 1, I do a comparison whilst the number is an I32, whilst the number is below 2^24 the extra row contains a 0 and the sample counter counts up from 0.  When the number = 2^24, a 1 is put in the extra row and the sample number is reset to 0 and the sample counter continues.

When I interogate the file whilst checking for continuity I simply detect when the values in the first row change from 0 to 1 then add 2^24 to all the remaining samples.

Not particularly clever but quite simple and doesn't massively increase the size of the data file.

Dom