The first binary file write operation for a new file takes progressively longer.

[I.Morgan]

I have an application in which I am acquiring analog data from multiple PXI-6031E DAQ boards and then writing that data to FireWire hard disks over an extended time period (14 days).  I am using a PXI-8145RT controller, a PXI-8252 FireWire interface board and compatible FireWire hard drive enclosures.  When I start acquiring data to an empty hard disk, creating files on the fly as well as the actual file I/O operations are both very quick.  As the number of files on the hard drive increases, it begins to take considerably longer to complete the first write to a new binary file.  After the first write, subsequent writes of the same data size to that same file are very fast.  It is only the first write operation to a new file that takes progressively longer.  To clarify, it currently takes 1 to 2 milliseconds to complete the first binary write of a new file when the hard drive is almost empty.  After writing 32, 150 MByte files, the first binary write to file 33 takes about 5 seconds!  This behavior is repeatable and continues to get worse as the number of files increases.  I am using the FAT32 file system, required for the Real-Time controller, and 80GB laptop hard drives.   The system works flawlessly until asked to create a new file and write the first set of binary data to that file.  I am forced to buffer lots of data from the DAQ boards while the system hangs at this point.  The requirements for this data acquisition system do not allow for a single data file so I can not simply write to one large file.   Any help or suggestions as to why I am seeing this behavior would be greatly appreciated.

[Kristi_P]

You might try periodically creating new directories for the files.  For example, after writing the first 20 files in one directory, create a new directory and begin creating files there.  Does that improve your performance?

Regards,

Kristi H

National Instruments

[I.Morgan]

Thank you for your response Kristi,

I have already tried something similar to what you are suggesting.  I created multiple directories on my hard drive before starting the data acquisition program.  At about file 20 I had the program switch directories for writing the new files.  This unfortunately did not improve the performance.  There is still an increasing delay for the start of each new file.

I have not however actually tried creating new directories on the fly as the program is running.  I will definitely give this a try just to be sure.

My thinking was in-line with yours in that perhaps too many files in one directory could cause a strange behavior, even though this starts at around file 30.

My PXI based data acquistition system has multiple FireWire hard drives connected for increased data storage.  Interestingly enough, I am not seeing this delay behavior with each new file on the secong hard drive connected to the system.  This second drive is daisy chained onto the first drive on port A of my PXI-8252 FireWire card.  I would have included this information in my last post but I only just recently configured the system with a full set of hard disks. 

I do not know why the second drive would work flawlessy when the first does not.  They are identical drives and I have already written over 100 files to this second drive without a hitch.  I will continue testing to see if this behavior appears on any of the other 4 hard drives that are connected; however, this test will take a few days.  Therfore, I am putting together a second identical system to retest writing to the first hard drive while changing directories on the fly to see if that helps. 

Thanks again for your help,

I.Morgan

[JohnSki]

Hello,

I was wondering if you ever found the problem with the ever-increasing file write times?  I am experiencing the same thing with my system and really need to find a solution.

Thanks, John Niezgoski

[I.Morgan]

Hi John,

Sorry I have not replied sooner.  You may have already found your own solution to this, but I would like to share with you my findings.  I have found no solution that completely eliminates this problem, but I have found a configuration that seems to work the best.  If you are developing a system that uses external firewire hard drives as I am, the firewire hard drive enclosure seems to make a big difference in system performance.   You can find a list of approved firewire chipsets that these enclosures use to help find a suitable enclosure through the NI knowledgebase.  I have personal experience with the Prolific chipsets and my Real-Time PXI system.  The Prolific chipset performed terribly.  I recently purchased about 20 CoolGear brand SC-S12Cl 2.5" Firewire hard disk drive enclosures.  These enclosures use the INIC1430L chipset and perform very well.  I have successfully daisychained these enclosures with LabVIEW RT.  Since installing these enclosures, the increasing delay during the first write to a new file did improve.  To deal with the remaining delay, I installed additional RAM in my Real-Time controller and simply increased the size of the DAQmx Input buffer during the data acquisition configuration.  That way, the buffer is large enough to handle any delays in writing to disk.  

The biggest performance increase for my application was definitely the firewire enclosure.  I wish I had a good explanation for why the delay exists and gets worse with each file, but the best I have been able to do is minimize this delay and buffer data to accomodate it.

Please feel free to reply if you have any other questions.

Thanks,

I Morgan

[JohnSki]

Thank you very much for the response.  Actually, we just did find the reason for the delay.  Our application stores a significant amount of data during a test, and then writes a moves a couple of small files at the end of each test.  We discovered this problem when the PLC we are communicating with reported timeout errors at the end of each test.  It turned out that our application was taking longer and longer to write these small files at the end of each test, and eventually it took too long and the PLC timed out.  Our analysis showed that that a simple 181byte file was taking only 30ms to write initially, and over time it took as long as 12 seconds to write the same file!!!  It turns out that there are two issues with RT and specifically the FAT32 formatting that the internal HD has to use.  1)  long file names - it seems that anything over 8 char really degrades the performance of writing/moving files on a FAT32 hard drive. 2) most importantly - the reason we were experiencing such long file write times was due to the number of files in the directory to which we were trying to write.  When no files existed in a directory, then 30ms is all the time it took to write the file.  When there were 1000 files in the destination directory (common for our app - 1 test every min), it took as long as 12 seconds.  NI recommended creating a new destination directory every 6 hours or so during a test day, and this seems to have solved the problem, or at least bypassed it.  Good luck, and thanks again for the response.  We are not using firewire drives, although we may in the near future.

[Daniel_Chile]

I am experiencing the same problem. Our program periodically monitors data and eventually save it for post-processing. While it's searching for suitable data, it creates one file for every channel (32 in total) and starts streaming data to these files. If it finds data is not suitable, it deletes the files and creates new ones.

On our lab, we tested the program on windows and then on RT and we did not find any problems.

Unfortunately when it was time to install the PXI on field (an electromechanic shovel on a copper mine) and test it, we've come to find that saving was taking to long and the program screwed up. Specifically when creating files (I.E. "New File" function). It could take 5 or more seconds to create a single file.

As you can see, field startup failed and we will have to modify our programs to workaround this problem and return next week to try again, with the additional time and cost involved. Not to talk about the bad image we are giving to our costumer.

I really like labview, but I am particularly upset beacuse of this problem. LV RT is supposed to run as if it was LV win32, with the obvious and expected differences, but a developer can not expect things like this to happen. I remember a few months ago I had another problem: on RT Time/Date function gives a wrong value as your program runs, when using timed loops. Can you expect something like that when evaluating your development platform? Fortunately, we found the problem before giving the system to our costumer and there was a relatively easy workaround. Unfortunately, now we had to hit the wall to find the problem.

On this particular problem I also found that it gets worse when there are more files on the directory. Create a new dir every N hours? I really think that's not a solution. I would not expect this answer from NI.

I would really appreciate someone from NI to give us a technical explanation about why this problem happens and not just "trial and error" "solutions".

By the way, we are using a PXI RT controller with the solid-state drive option.

Thank you.

Daniel R.

Message Edited by Daniel_Chile on 06-29-2006 03:05 PM