LabVIEW VI is hanging while reading ~1GB csv file

[c_rituc]

Hi daveTW

Thanks for your reply, 

Sure, this sounds like good approach . i have one doubt , in this way we need to read file again and again every time search string changes ; will that cause any issue . i will give this a try and see if it helps in memory efficiency

Thank you 

[c_rituc]

Hi BertMcMahan , 

i will check with team if i can share the part of code , will share if allowed to do so .

the core vi is more or less to the vi in below mentioned link ( i tried the same approach ) , the only difference data format is string not numeric and after parsing the data from csv file we are storing it in variant. Parallel loop feature is not used since we are using shift register to store data in variant between different iteration

https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000015CQ0SAM&l=en-SG

when we tried to use the code from above VI , read is fine ; However it is during writing / reading from Variant ; program is hanging . 

i will try all the different ways suggested by all of you in different posts and will get back to you . 

[drjdpowell]

---

@c_rituc wrote:

Hi GerdW , 

Thanks for your reply .

For the 500 MB file columns used are : 34 , and rows : 1048576 .

The size if the file varies and thus the content too .

we are reading whole file as need to search few strings in first column and  and then take data corresponding to cell. The operation is done in sequence ; every time we come to this step , different string is being searched . The file is read only first time ; after that we are storing data in varaint in FGV and using that , However even the first time read is hitting the memory.

---

This sounds like a textbook application for SQLite.  And a simple one.  

[daveTW]

As a first approach you can start with the attached project. It's not confidential  😉