circular buffer

[labVIEWPMU]

Hi all,
I am trying to implement a circular buffer in my labVIEW interface for a phasor measurement unit. The circular buffer must take in 144 bit data (hexadecimal) at 10 ksps, and it must be able to store 10 seconds worth of data (6 seconds pre fault event, 4 seconds post fault event). This buffer must be able to write the data to an excel file if a fault occurs. If anyone has any experience with circular buffers in labVIEW, any assistance would be greatly appreciated. 
Thankyou. 

[GerdW]

Hi PMU,

---

@labVIEWPMU wrote:

I am trying to implement a circular buffer in my labVIEW interface for a phasor measurement unit.

The circular buffer must take in 144 bit data (hexadecimal) at 10 ksps, and it must be able to store 10 seconds worth of data (6 seconds pre fault event, 4 seconds post fault event).

---

• What exactly is your "LabVIEW interface"? (Do you know interfaces are closely related to OO programming in LabVIEW?)
• You read "144 bit data (hexadecimal)": what do you mean by this? Do you receive just 18 bytes?
• You read "at 10ksps": do you receive 10k sample items per second? What is a "sample" in your case? One (float?) value or your 18 byte message?
• You want to store 10s of data: this will result in a buffer size of 10s*10kSps= 100kS: you could predefine an array of this size…

---

@labVIEWPMU wrote:

This buffer must be able to write the data to an excel file if a fault occurs.

---

This is a completely different task - and should be separated from your "circular buffer" (IMHO)…

Why not write a different function that requests data from the buffer routine and saves this data into a file?

(I recommend to start with plain text files (aka CSV) instead of "real Excel" files!)

What have you tried and where are you stuck?

[Kevin_Price]

From the limited description, this sounds like a great use case for a fixed-size queue that you feed with the Lossy Enqueue function.  Such usage naturally creates a circular buffer, albeit one with limitations on data access.  The good news is that it sounds like those limitations will be acceptable for your usage.

Set the size to 10 sec worth when you create it, always use Lossy Enqueue to write to it.  This will naturally make a sliding window of the most recent 10 sec of data.  4 seconds after a fault occurs, extract all the queue data and write to file.  I'd extract data using Get Queue Status with 'return elements'=True.  This keeps the queue full in case another fault occurs right away.

-Kevin P

[RTSLVU]

---

@labVIEWPMU wrote:

Hi all,
I am trying to implement a circular buffer in my labVIEW interface for a phasor measurement unit. The circular buffer must take in 144 bit data (hexadecimal) at 10 ksps, and it must be able to store 10 seconds worth of data (6 seconds pre fault event, 4 seconds post fault event). This buffer must be able to write the data to an excel file if a fault occurs. If anyone has any experience with circular buffers in labVIEW, any assistance would be greatly appreciated. 
Thankyou. 

---

I have implemented pretty much the same thing in my qualification platform. In general there are two ways to do this Queues or Channel Wires, I chose Channel Wires and a Channeled Message Handler program architecture (CMH)  

I will try to explain the two loops that handle this Safe Operating Limit (SOL) and Log/Display loop.

The SOL loop is constantly taking measurements of all input and output conditions. Compares them to the safe operating limits, and places every measurement into a Channel Wire

The Log/Display reads each measurement from the channel wire, displays, places if into an array, and places it into a "lossy" Channel Wire that loops back into the Log/Display loop. I calculated the size of this Channel Wire (buffer) using the sample rate of the power analyzer taking measurements to hold enough readings for about 10 seconds prior to the SOL trip. Channel Wires and Queues are FIFO so it always holds about the last 10 seconds of measurements.

When the Log Interval timer expires the Min Control loop tells the Lod/Display loop to write the most current measurement to disk

If certain conditions (Input Current in most cases) is higher that the limit for longer than a certain duration (to prevent inrush tripping SOL) then the SOL loop send an Error using a custom error code to the Main Control Loop.  The main Control loop then tells the Log/Display loop to dump the buffer to the end of the data file, shuts down all input power to the UUT, and halts the test.

[UliB]

---

@Kevin_Price wrote:

From the limited description, this sounds like a great use case for a fixed-size queue that you feed with the Lossy Enqueue function.  Such usage naturally creates a circular buffer, albeit one with limitations on data access.  The good news is that it sounds like those limitations will be acceptable for your usage.

-Kevin P

---

What "limitations on data access" do you refer to in comparison to other implementations of a circular buffer?

[mcduff]

I have used Lossy Queues, they work well for most situations. They can be problematic if you need data access that is not FIFO ordered.

On VIPM there is https://www.vipm.io/package/jdp_science_malleable_buffer/?utm_source=vipm_desktop

which may work for your application.

[LVNinja]

I used a circular buffer before, it's basicaly a while loop with a non innitialized shift register. There is an input for the iteration of the while loop that will be calling this as a subvi. on the first iteration, the buffer is innitialized with the first set of data. From then on, every new data is added to the buffer and as new data is coming , the older data is discarted. So when the fault occour, you will have all the 10 seconds of data in memory to be saved.

[altenbach]

It does not need to be "circular" (verbatim) and you can use a fixed array where you simply keep track of the insert point and replace the oldest value for a fully in-place operation where only one element gets "touched".

When a fault occurs, you can push the current data (data array & insert point) to another processing loop that can reassemble the consecutive timeline [before, at, after]  from the array data and insert point for further processing/logging or similar.

As has ben mentioned, a significant amount of your post is ambiguous and we can typically offer much more targeted advice seeing some simplified sample code that simulates the process.

[Kevin_Price]

---

@UliB wrote:

---

@Kevin_Price wrote:

From the limited description, this sounds like a great use case for a fixed-size queue that you feed with the Lossy Enqueue function.  Such usage naturally creates a circular buffer, albeit one with limitations on data access...

---

What "limitations on data access" do you refer to in comparison to other implementations of a circular buffer?

---

Much like mcduff already responded, a lossy queue doesn't really support random access to data subsets.  You either retrieve data 1 element at a time in FIFO order or you retrieve the entire data set as a whole.

By contrast, a DAQmx continuous acquisition task buffer is an example of a circular buffer that provides some degree of random access through a combination of the "Offset" & "RelativeTo" properties, along with a specified "# samples to read".  Such access can be very handy for certain usages.

-Kevin P

[Kevin_Price]

A small "sea story" about circular buffers from somewhere around Y2K.  I was working in LabVIEW RT, there was no "lossy enqueue", and regular queues weren't all that efficient yet.  One of the cardinal RT rules was "avoid memory allocation during real-time execution."  So the use of "Build Array" was a big No-No.

    I had an app that wanted a circular buffer.  I would be writing 1 sample/channel at a time, and might often need to retrieve the entire X second buffer in order from oldest to newest sample.  [I don't recall why any longer, but there was *some* reason that whatever I was doing with the samples didn't yield to an Action Engine approach.]

The normal way to create such an ordered output array is to take a subset from the end, a subset from the beginning, and use "Build Array" to merge them into one output array.  But I was looking for a way to *avoid* Build Array.  What to do?

    What I came up with was to size my array for double the size I actually needed.  If I needed N samples from the recent past, I sized my array for 2N.  Then whenever I wrote a sample to the buffer, I wrote once at index i and again at index i+N.  I was making a back-to-back duplicate of my array.  The (assumed) payoff was that at retrieval time, I already had contiguous data in order no matter where my write pointer was pointing.  I could just do a subset operation and already have contiguous data without using Build Array and risking memory allocation.

At the time I thought it was a clever solution.  Looking back now I'm far less certain it made any appreciable difference.  I didn't have in-depth insight about RT, the LV compiler, memory reuse rules, etc.  Just some of the rules of thumb.   And nowadays, I'm pretty sure it'd be an inferior approach.  Just tossing it out there as a reminder that unconventional answers can sometimes be suitable for specific needs.

-Kevin P