Huge data storage question

[concorpoly]

Hi,  I am wonder how can I store huge data, say, 524,288 samples per second, one sample use 16 bit.

Now I store the data in datalog file, but file size becomes over 110M bytes in one minute, but I need one day or several days data to store.

I have no idea how to handle that, need help.

[JÞB]

If you REALLY need to save that much data (Often you only think you do as yourself what will we use the data for?)

A RAID Array can provide a effective solution here is an intro KB Article

[Ben]

---

@concorpoly wrote:

Hi,  I am wonder how can I store huge data, say, 524,288 samples per second, one sample use 16 bit.

Now I store the data in datalog file, but file size becomes over 110M bytes in one minute, but I need one day or several days data to store.

I have no idea how to handle that, need help.

---

I also question having to save all of that data (what will you use to review the data?).

For these cutting edge "how do I do THAT" challenges I often rach for "Divide and conquer" appraoch. First start with a high-end DB server that can serve a DB that large and use that to store the data. If the network can not keep up concider SCRAMnet (shared memory) to do the transfer to the server.

Do you really need all of that data?

YOu will be filling a dirve in a few days.

Ben

[concorpoly]

It's requirement of customer, in fact, I don't think they really need it. will discuss it with them.

[johnsold]

Sometimes doing the math and showing how many hard drives they need to buy each month is an effective way to get someone thinking more practically.

Another approach is to save summaries of data indefinitely and save all of the data for a short period of time in case it needs to be reviewed.  Start with three hard drives (X, Y, Z). One (Z) saves summaries.  If the summaries are compact enough this might be good for a year or more. On the other two drives save a few days worth of data. Start with X. When is it about 80% full switch to Y. During the 2-4 days it takes to fill Y, the customer must review the summaries and, if necessary, the raw data on X.  If something there must be preserved, they can copy it to DVD.  When Y reaches 80%, switch back to X overwriting the old files and start the review of the data on Y.

Lynn