Modifying a large string

[mperlick777]

I have a For Loop that reads and modifies a 40 meg file. The file is a log of GPS and CAN messages. The Labview App will modify the data of one certain message type. This all works but it takes 4 and a half minutes to complete. Is there a faster way to do this other than creating a DLL? Is it due to Labview creating a new string every time the string's modified with the Replace Substring Function?

Thank You

[RavensFan]

Are  you trying to read in the 40MB file all in one shot?  Do you need to do that?

Have you tried benchmarking your code to see what is taking the longest?

Your Read Distance VI seems to take the string and work on subsets of it.  Are those sequence structures necessary?  It doesn't look like it.  You have a lot of manipulation going on there, but some of it seems unnecessary.  Like the U8 bullets aren't necessary if you define the output coming from hexadecimal string to number to be a U8.  But then  you do a lot of merging to create I32's.  I bet you can do all of that conversion in shot.

Likewise with the Insert New Distance.  I don't know what is going on there, but it seems far more complicated than it should be.

If you can post a small sample file with some typical data, and a good description of what you are trying to do with it, I'm sure it can be done much more easily.

[Charles_CLA]

Could you post a subset of the file and describe what you are trying to do? It is a bit difficult to improve the efficiency of your code without knowing the goals.

[crossrulz]

From what I can tell, you don't need any of your sequence structures.  Dataflow will handle the order for you.

That conversion from a hex string to number with U8 bullets and join numbers seems convoluted.  With a small sample of your file and an explanation of the format, I bet we can simplify that quite a bit.

I'm also not seeing the point of using a FOR loop.  Just use a WHILE loop and stop when you can't find your distance.

[mperlick777]

I've attached a small portion of my log. Below is one CAN message and an explanation of what I do with it.

J-18FEC100,08,EC,62,C2,09,FF,FF,FF,FF,

In my log I am searching for this type of CAN message by doing a search of FEC1. I then read the first four bytes of the CAN message which would be EC 62 C2 09. This value, 0x9C262EC or 163734252, is a unit of distance. This is done in Read Distance in Log Subvi.vi. I then perform addition and subtraction to this value. I reassemble this message and replace the old value with this new value in the log.This is done in the Insert New Distance Subvi.vi.

I can replace the For Loop with a While Loop. I know there's other recommendations and I will try those as well. My other option is to create a DLL with Visual Studio. I've done that with a similair Labview App and it takes less than a second to modify the GPS portion of the Log. However, I prefer to use Labview as it's easier to debug.

[RavensFan]

I would read the file 1 line at a time.  (You can right click on the Read from Text File function and choose Read Lines).

Do that in the loop.  Search for the FEC1 string.  If it doesn't exist, rewrite the line to the new file.  If it does exist, then determine what the new characters would be in the string and write the new string out.

If you  take the string and put it through Spreadsheet String to Array with comma as a delimiter, wire a string constant to the datatype, and you'll have an array of elements between the comments.  If you set your format code to %x and the datatype to U8, it will convert it to U8 elements automatically (although your initial string element that is a code will be forced to 0).

[crossrulz]

Ah, I see what you are trying to do now.  You are taking the read distance, recomputing the distance, and replacing the same data in the file.

One of your biggest issues is that you are forking your wires a lot.  And that is a potential data copy.  And if you are copying large data, that takes a long time.  Instead, you should be trying to do as much in series as possible.  I changed your subVIs to use Scan From String and Format Into String instead of trying to increment value to read and convert to number multiple times.  This also allows you to build up the string to replace and do a single replacement.

It might be worth trying Raven's suggestion of reading a single line at a time.  You might also want to look into the Producer/Consumer architecture to put the writing to the new file in parallel with your processing of the file.

[mperlick777]

Thanks RavesFan and others for your help. I'll implement these suggestions and give it a try. I'm quite sure this should speed things up.