Indeed we write the file to a RAID system. Is there anyway I could verify the transfer speed is near to 750MB/s, i.e. from camera to frame grabber. I am transiting my code to 64bit LabVIEW and this going to take some time as there are some dll being used in the 32bit program. To avoid confusion, I pre-allocated 100*2MB of buffer, which then being filled up in one loop; camera will stop once the buffer is filled up, the whole buffer is being written to the RAID, this is in contrast to real-time acquisition and writing at 250MB/s which might be perceived in this thread. My key concern is why is the time of writing the 200MB equivalent of data could varies so much (tends to become 30-40% slower as the program run longer)? Is it reasonable to expect it to be relatively constant? Re: Saving large amount of txt image file - getting slower as the program runs longer

[sp@davis]

Indeed we write the file to a RAID system.  Is there anyway I could verify the transfer speed is near to 750MB/s, i.e. from camera to frame grabber.  I am transiting my code to 64bit LabVIEW and this going to take some time as there are some dll being used in the 32bit program. To avoid confusion, I pre-allocated 100*2MB of buffer, which then being filled up in one loop; camera will stop once the buffer is filled up, the whole buffer is being written to the RAID, this is in contrast to real-time acquisition and writing at 250MB/s which might be perceived in this thread.  My key concern is why is the time of writing the 200MB equivalent of data could varies so much (tends to become 30-40% slower as the program run longer)? Is it reasonable to expect it to be relatively constant?

---

Re: Saving large amount of txt image file - getting slower as the program runs longer

‎04-01-2014 03:08 PM

If you consider the absolute ideal transfer speed of 6.0Gb/s is only 750MB/s. And that's ideal, IE, all one large file, not MFT writing, no mechanical movement.

The Average Sustained write speed is on the order of 50-75MB/s. This considers smaller files, mechanical movement, MFT writing, etc.

This all depends on your hard drive. You can get better or worse performance from this, but this is likely an average.

There's no way any current top of the line hard drive system can sustain 500MB/s without complete RAID.

Though it sounds like this is offloaded outside your datacaputre, it'll still take many seconds to write that data external to your collection.

As for your code, if you're jumping by 2GB over the course of running your code, then you are not preallocating all the images into ram. You are probably running into a significant performance reduction reallocating the structure or copying it. I'd recommend running DETT to see where your memory is being allowcated.

Also, as already pointed out, a 32bit LabVIEW is not capable of handling more then 4 GB of RAM. You're running at about 100% single CPU (assuming 8 logical cores), so there's definitately code performance issues to look into.

---

[sp@davis]

If you consider the absolute ideal transfer speed of 6.0Gb/s is only 750MB/s. And that's ideal, IE, all one large file, not MFT writing, no mechanical movement.

The Average Sustained write speed is on the order of 50-75MB/s. This considers smaller files, mechanical movement, MFT writing, etc.

This all depends on your hard drive. You can get better or worse performance from this, but this is likely an average.

There's no way any current top of the line hard drive system can sustain 500MB/s without complete RAID.

Though it sounds like this is offloaded outside your datacaputre, it'll still take many seconds to write that data external to your collection.

As for your code, if you're jumping by 2GB over the course of running your code, then you are not preallocating all the images into ram. You are probably running into a significant performance reduction reallocating the structure or copying it. I'd recommend running DETT to see where your memory is being allowcated.

Also, as already pointed out, a 32bit LabVIEW is not capable of handling more then 4 GB of RAM. You're running at about 100% single CPU (assuming 8 logical cores), so there's definitately code performance issues to look into.

---