Read from text file takes very long after the first time

[Florian.Ludwig]

Thanks for your answers. I'm Sorry I couldn't answer - I was out of office for a couple of days (rather unexpectedly).

@Yamaeda: That would indeed be easier. I'm doing it like this because the file might be very large and I don't want to have it in memory all at once.

@Camerond: What Altenbach said. I was just sticking to the save habit. Plus I wasn't sure if the read vi would behave the way I wanted without it (i.e. read subsequent parts of the file with each loop). Turns out that works without the shift register too.

@Altenbach: To measure reading speed I just threw a "timer (ms)" vi with a feedback node and subtraction into the loop. I thought taking the loop time would in this case be equivalent to the read time (indexing out of a small array of 100 or less lines should be fast).

@crossrulz: I'm afraid that might be the root cause. I don't see it at first glance in my main vi - but today I will finally be able to dig down into it.

I'll report back if I figure it out (or need more help )

Regards

Florian

[Norbert_B]

Florian,

regarding your benchmark: I hope you initialized the benchmark like this:

Norbert

[Florian.Ludwig]

Just so you see everything, the complete sub vi including the (reworked) benchmark:

The sub vi does some other stuff (proprietary file format requires this - yay! ) which should be of no concern for the benchmark.

"mean" should reflect the read time pretty accurately. It is about 0,1 ms when the vi is quick (running for the first time or alone) and 5 ms when it is slow. I don't know why I posted durations ten times longer previously, sorry. But the increase by a factor of 50 in execution time of the read remains.

"mean 2" should reflect everything else going on in the loop. It is not measurable like this - it is always 0. I don't think the problem lies there.

The file I benchmarked is 7.3 MB and consists of 120007 lines of text. "y" (the number of lines read per iteration) is 8 (eight).

I'll keep on digging into the main vi.

There was a file reference kept open there. Closing it before execution of the sub vi makes no difference however.

Regards

Florian

[Yamaeda]

Could it be that it gets progressively slow the larger the file is? That it needs to look into the file to position reader? 7Mb isn't much, i stand by my previous tip of reading all at once. 🙂

/Y

[Florian.Ludwig]

7 MB isn't much - but that is only an example file. Files may be hundreds of MB - more than I'm willing to dump into RAM.

My aim is to create a reduced sample rate preview of a file that would be too big to load.

My problem is that the operation is slower by a factor of 50 on the same file when calling the sub vi for the second time (even on the small 7MB file).

Otherwise your tip is great - reading the 7 MB file all at once works flawlessly every time . This kind of makes think that the problem shouldn't be with the main vi performing disk operations. Why would it obstruct reading 8 lines (15k times) and not obstruct reading the whole file?

I'd be willing to implement the preview in another way as long as it is easy on memory and reasonably fast. Nothing comes to mind right now though.

Regards

Florian

[Yamaeda]

Does it make any difference if you set the file open to Read instead of Read/Write? Is it all loop times that slow down, or just a lot of extra file access time? Can it be it's waiting for the system to recognize the previous write is complete?

Yes, with several hundred MB i understand your concern. You can ofc read in chunks of e.g. 8k lines. 🙂

/Y

[Florian.Ludwig]

Funny thing is when I set it to read only it seems like the sub vi is always slow when called from the main vi (sub vi alone remains fast). I'm not sure if I'm testing this correctly though since there is no save way to trigger a fast execution when running the main vi so it might be bad luck. But it definitely doesn't improve anything.

All iterations are slower. Most take 5ms some few take a little longer (up to 16ms).

I was thinking about the chunks too - but increasing chunk size increases read time proportionally. No gain there.

Regards

Florian

[Yamaeda]

The Read-option performance sounds strange to me, if anything Read-only should be faster ...

Reading in chunks is more effective, since you'll read a full disk block anyway (usually 2kb in NTFS and up to 32kb in FAT32).

/Y

[camerond]

Might you attach your Main.vi (not a picture) to a message (taking out any proprietary bits), or did I miss it? It might help.

Cameron

[Florian.Ludwig · Accepted solution]

So...

I'm done disassembling my main vi, and I've found my mistake in the process. (Why do the bugs in my code always have to be my fault? )

In another sub vi I was reading the first line of the file to check whether it is a valid file for that operation. I FAILED to close the reference to file in that sub vi. Beginners mistake really.

Well at least I've found it now.

Still don't know why this had no impact when loading the file whole though...

Can anyone shed light on that one? There's something to be learned there.

Thanks for all the suggestions

Florian