Big string operation

[alex.]

To import a big array into database, it need to convert the array into a long string and using the string generate a SQL statement.

But the process of generator long string take very long time as the attached vi.

It seems mostly the time is spend on 'concentrate the string' in each loop. May be because the system is keep assign new size memory to the string.

One way to solve it come into mind is define a long fixed size string first and then use string subset replacement everytime. But there seems not funtion of define a fixed size string in LV.

Any suggestion, thanks. 

[GerdW]

Hi Alex,

no, there is no fixed size string in LV.

The loop runs fast (ok, that's relative ) in your example, only the "flatten to string" needs some more time (for the first run). On my computer your VI only takes less than 2s. So you have to define this "take very long time" and what timings you need...

[Yamaeda]

Array to spreadsheet string is ALOT faster, though the result isn't identical.

/Y

[smercurio_fc]

---

alex. wrote:

One way to solve it come into mind is define a long fixed size string first and then use string subset replacement everytime. But there seems not funtion of define a fixed size string in LV.

---

Actually, there is, sort of. In fact, you sometimes need to do so when calling DLLs that expect a pre-allocated string. You use the Initialize Array to create an array of U8 values and then use the Byte Array to String function.

As for your code: Can you provide an explanation of the conversion you're doing? It seems a bit ... convoluted.

[alex.]

Sorry, I have commented off the most time cost part in the first attachment. I have attached again with the original copy.

The array may be even bigger.

I also tried spreadsheet string ever, but it may take around 1min when array is bigger.

I also tried the way smercurio_fc suggested, but when the array size goes up to 800000, the definition also takes around 1min. Any suggestion, thanks.

[MoReese]

I'm just curious as to why such a long string.  Perhaps there is a more efficient way to do this?

[altenbach]

First things you need to initialize the shift register, else the output string grows with every run of the VI. The lenght is 15MB after the first run and 30MB after second run, and so on. (and that's for the 5000x500 input)

Is there some online documentation for the conversion you are trying to do? Seems convoluted.

[alex.]

I have a tdms file around 10M, it need to import the data into center sql database.

[altenbach]

---

@alex. wrote:

I have a tdms file around 10M, it need to import the data into center sql database.

---

10M of what? Bytes? DBL values?

Your 5000x500 DBL sample array is already ~20MB (5000x500x8bytes) and the output is 6bytes/value for a total ot 15MB as I already said.

You mentioned problems going to 80000. Is this still a 2D array? Is 80000 the product of the dimensions or just one of them. Please be very specific with units and describe the problem fully. It is very difficult to understand single-sentence soundbites posts out of context.

[Yamaeda]

---

@alex. wrote:

To import a big array into database, it need to convert the array into a long string and using the string generate a SQL statement.

But the process of generator long string take very long time as the attached vi.

It seems mostly the time is spend on 'concentrate the string' in each loop. May be because the system is keep assign new size memory to the string.

One way to solve it come into mind is define a long fixed size string first and then use string subset replacement everytime. But there seems not funtion of define a fixed size string in LV.

Any suggestion, thanks. 

---

Btw, you cant import it directly like that, a SQL command is limited to 8000 characters (if memory serves).

If you need to import it you'll probably want to make 1 insert per tuple/row and loop that.

/Y