04-14-2020 09:20 AM
HI-
I am trying to amass large amounts of data (13K bytes at 8KHz==104 MBytes) over a period of up to 20 seconds. This data is coming from a TCP/IP connected hardware device and I use their provided DLL to grab the data. Question is this: what is the fastest method to store it/write to file? Should i put it all in RAM first (memcpy) and then dump to file later, or should I go straight to file with an fprintf? I have used both and wonder if there is a difference based on bus speed or ram speed.
Is there a faster way to do this straight i/o scenario? I have pci-3 bus and solid state hard drives. I also have 256G RAM on my machine. I am under the impression that the ssd speed and the large amount of RAM are what determines how fast i can stream to file/memory. Am I wrong and should i really be concentrating on multiprocessors or multi-threading? I am not plotting anything during data acquisition - only displaying the count on the UIR so not much going on there.
I appreciate your help on this. I usually find what i am looking for by searching the forums so I don't usually post but please know I appreciate and use your advice ALL THE TIME!
Blue
04-14-2020 10:18 AM
First of all, I would not call "large amounts of data" a 104 Mbyte data block on a 256 GB RAM machine. 🙂
A packet will use 0.039 % of machine RAM.
Even imagining a constant flow of those "packets", I think it's better to buffer data in RAM, and then saving it to disk, with a proprietary strategy of buffering, instead to rely on the built-in C streams buffering.
A buffering strategy under your control has the purpose to defeat any delay that could be caused by event extraneous to your program, and out of your control.
For strategy I mean something from a simple single buffer, double buffer, or - if needed - a queue.
An advice I could give is to use a thread isolated from GUI's thread to perform receiving / buffering /saving task, and to use a deferred call to GUI thread to update any information on UIR, to completely decouple the two threads.
04-22-2020 01:46 PM
Thank you, Carlo. I will try your suggestions and post back.