Stream 2 x 120 fps cameras to disk ... HDD can't keep up?

[josborne]

So I have two Basler firewire cameras.  Each camera has its own firewire-B card.  And I I wrote some LabVIEW software that can grab frames at 120 fps per camera. 

But I am unable to stream to disk quickly enough to keep up with the frame grab.  Even if I use a producer-consumer architecture, my buffers end up overflowing.  Because I am logging from two cameras simultaneously, I am essentially streaming 240 fps to disk, and each image is color and 640x480.

Question:  any reccomendations on how to get around this?  I am curious if a new and fast PC with a super-fast HDD will solve the problem.  Or maybe even a SSD drive?

[Norbert_B]

It all depends on the data rate you have to stream. Different storages have different performance. Benchmarks are the key to get a feeling if a storage type (HDD, SSD, RAID) is sufficient to handle the given data rate. 

Let's make a small calculation on your figures:

640 * 480 (Resolution) * 2 (32 bit color depth) * 240 (fps) = 147456000 B/s. This is round about 147 MB/s. Pure data without any additional information.

So you can see that even SSD will most probable be not sufficient (read something about 70-80MB/s for SSD writing), so you will have to use RAID.

hope this helps,

Norbert

EDIT: The transferrates for SSD is very different. Maybe, this chart helps a bit, even though it does not state MB/s....

[EvenDeejay]

My SSD drive at home got 415 Read MB/sec and 260 MB/sec write. Buy two of these and put them in raid0 and you got the double. Or you can buy a Revo Drive X2. Believe me, speed is not a problem for SSD!

[BruceAmmons]

The problem is definitely the stream to disk rate, as others have said.  Two things are slowing you down - shear volume (about 75 MB/s for grayscale) and opening and closing 240 files every second.

One way around the second issue is to write everything to a single (or two) files, such as AVIs.  The only solution for the first issue is a very fast hard drive or several operating together to give you the high transfer rate.  I suspect several drives in RAID configuration would do the job.

Bruce

[EvenDeejay]

If you want speed...  1 Gigabyte Read/write!! Of course it costs alot, but it's the fastest thing you can get.

As budget disk I would have set 3 of these in raid0. 

[josborne]

Wow, thanks for all the tips everyone!  I really appreciate it.  Kudos will be distributed!

As Bruce mentioned, I was actually planning on writing all the images to two separate AVIs, so that will save on the open/close file operations.

To be honest, budget is not my biggest concern.  My goal is to make it work ... and to make it work reliably.  I also want to reduce the complexity (eg I'd rather buy a more expensive SSD drive than to purchase multiple drives and put them in RAID0).  So I'll probably go for a PCIe SSD drive like the models that were suggested.  But since my two PCIe slots are already occupied by the firewire frame grabbers, I will also have to purchase a new PC with three PCIe slots.  This leads to my next question:

I won't somehow overload my PCIe bus if connect two framegrabbers and an SSD drive to the bus?

Thanks!

[josborne]

Hmmm.  Interesting.  I just did a HDD test using Dacris Benchmarks.  It said my HDD can only do 31.4 MB/s which is a far cry from what I will require.

It is a 7200 rmp SCSI drive.

[dan_u]

---

@Norbert B wrote:

Let's make a small calculation on your figures:

640 * 480 (Resolution) * 2 (32 bit color depth) * 240 (fps) = 147456000 B/s. This is round about 147 MB/s. Pure data without any additional information.

---

Multiply that by 2 (32 bit color depth = 4 bytes per pixel) --> you need a HDD speed of ~300 MB/s, considering nothing else needs access to this HDD at the same time.

Have a look at the following links for NI products covering this requirement (PXI):

http://zone.ni.com/wv/app/doc/p/id/wv-466 

http://sine.ni.com/nips/cds/view/p/lang/en/nid/203949 

[Joseph_Loo]

Am I missing something. I come up with 589 MB/sec assuming (4 bytes per pixel, using only 8 bits per color)?

[dan_u]

---

@Joseph Loo wrote:

Am I missing something. I come up with 589 MB/sec assuming (4 bytes per pixel, using only 8 bits per color)?

---

So what would be your calculation?

640 * 480 (Resolution) * 4 (32 bit color depth) * 240 (fps) ~ 295 MB/s