Generating 0 and 1 sequences - Performance Issue

[RavensFan · Accepted solution]

Thanks for adding more detail.  I still don't know how a string of 1's and 0's correlates to a "peak detection" tool.  Or why you would even need to use 2^30 of them all at once.

Do you really need that many to be used all at once?  Why not generate each sequence as you require it?  Or if they are to be strung together is some way, generate than on the fly?  Now you are saying that you don't want to use all of the sequences, but perhaps only ones that meet a certain condition such as a minimum number of 1's.  Generation as you need to use them could determine if the sequence you just generated doesn't meet the rules and it moves on to the next number for generation and tests it against the rules.

Anything you do where you generate the sequence as you need it would still be far quicker than trying to generate them all at once and saving them or loading them to a text file.  Even if you just create an array of U32 or U64 integers that meet your required rules ahead of time, then convert them to a string of 0's and 1's as needed would save a whole lot of memory space compared to a (38+4)*2^38 +8 byte long array of 2^38 string elements.  (Actually I don't know if you can even create an array that as more than 2^32 elements.

[altenbach]

---

Ravens Fan wrote:

Actually I don't know if you can even create an array that as more than 2^32 elements.

---

Since array indices are I32, the max is 2^31.

[RavensFan]

---

altenbach wrote:

---

Ravens Fan wrote:

Actually I don't know if you can even create an array that as more than 2^32 elements.

---

Since array indices are I32, the max is 2^31.

---

Yep, you're right.  I was just think 32 bit which is a long ways off from an array with 2^38 elements.  But what's a factor of 2 among friends?

I'm hoping that the original poster will soon realize what he is asking for is impossible and starts rethinking the way he is doing things.  Just yesterday, the requirement was only for a 23 or 24 "bit" sequence.  Today, it has jumped up to N=38!

[altenbach]

---

Ravens Fan wrote:

I'm hoping that the original poster will soon realize what he is asking for is impossible and starts rethinking the way he is doing things. 

---

Yes, if it does not fit into memory, processing the same number of cases (peak analysis, etc.) will take forever!!!

With the same reasoning, nobody has written a chess program that generates all possible positions (~10^43), does a retrograde analysis, and plays perfectly. The game tree complexity of Chess is 10^120.

For a comparison, the number of atoms in the known universe is only about ~10^80 😄

[dan07]

Hello all

After all the advices that you gave me, I decided to don't create the array with all the sequences. I put the code that I want to run inside a FOR LOOP and each iteration feeds the code with a different sequence and it's not saved to an array. It is working perfectly!

Thanks for the help.

Dan07