Looping through .hex file takes too long

[ApeKind]

Hi guys,

Developing an application that currently reads in a .hex file, and then sorts it from a excessively long string into a 2D Array line by line.

The following code is used to read the .hex file

void ReadHexFile(char *name)
{
	FILE *file;
	char *buffer;
	unsigned long fileLen;

	//Open file
	file = fopen("Project.hex", "rt");
	if (!file)
	{
		fprintf(stderr, "Unable to open file %s", name);
		return;
	}
	
	//Get file length
	fseek(file, 0, SEEK_END);
	fileLen=ftell(file);
	fseek(file, 0, SEEK_SET);

	//Allocate memory
	buffer=(char *)malloc(fileLen+1);
	if (!buffer)
	{
		fprintf(stderr, "Memory error!");
		fclose(file);
		return;
	}

	//Read file contents into buffer
	fread(buffer, fileLen, 1, file);
	fclose(file);

	//Sort out buffer into a 'structure' ready to send down the CAN
	sortBuff(buffer);

	free(buffer);
} 

Next is where im having some problems, the sortBuff() function that sorts through the buffer array takes a good 15min or so to complete it. The .hex file has 32,964 characters which the function then splits into a number of lines for further processing.

void sortBuff(char * input)
{
	char HexArrSort[760][45] = {0};
	int	line = -1, hexChar = 0, i = 0, size;
	char eof = 0;
	
	
	while(!eof)
	{  
	
		if( input[i] == ':') 
		{
			++line;
			hexChar = 0;
		}
		else if(input[i] == '\n')
		{
			input[i] = '\0';
		}
		
		HexArrSort[line][hexChar++] = input[i];
		
		++i;  
		if((HexArrSort[line][7] == '0') && (HexArrSort[line][8] == '1'))
		{
			HexArrSort[line][9] = 'F';
			HexArrSort[line][10] = 'F';
			eof = 1;
		}
		
	}
}

Any idea on how I can optimize the function/s to perform this task of reading then sorting a lot quicker?

Thanks in advance

[RobertoBozzolo]

Hi ApeKind,

it seems to me that this thread should give you a lot of ideas on reading and parsing a file: take particulare attention to the code posted by nickb in his last posts.

[nickb]

Hmmm... that's curious.  15 minutes seems like a *very* long time for your code to be running.  I copied your function exactly and ran it against a .HEX file of 30k bytes, and I was able to execute your function 1000 times in 275ms:

I'm really not sure why you'd be seeing such long execution times.  My only guess is in your call to fopen.  You pass "rt" as the mode, but this is not a valid mode for fopen.  You want to pass "rb" - this way you can be sure that the value you get back from ftell is representative of the number of bytes in the file.  If I were you, I'd validate that you are reading what you expect from the file.

I took a quick crack at using strtok to split up the hex file, and found that it was a little less than twice as fast as the code you posted - you could also give something like the following a try:

void SortBuff2 (char *input)
{
  char hexArrSort[760][45] = {0};
  int i = 0;
  char *tok;

  for (tok = strtok (input, "\n"); tok; tok = strtok (NULL, "\n")) {
    memcpy (hexArrSort[i], tok, strlen (tok) + 1);
    i++;
  }
  assert (!strcmp (hexArrSort[--i], ":00000001FF"));
}

I was honestly surprised to see my code execute more quickly than yours.  I think the reason for this is my use of the CVI release compiler.  The strtok and strlen functions I used were compiled with optimizations, whereas the CVI compiler is not an optimizing compiler.  This means that the strtok method may be faster if you are using the CVI release compiler, but if you're able to use a 3rd party optimizing compiler for your release compiler, you should see better performance with the code you wrote.  I tried both the clang 1.0 optimizing compiler and the MSVC 10.0 optimizing compiler, and in both cases your code ran about 25% faster than mine.

NickB

National Instruments

[ApeKind]

Ahhhh Newby mistake, it was taking so long as I was running it in Debug and not as an executable (still strange why it takes so long).

Thanks for all advice tho 

nickb - will be changing my function to your suggestion as it seems to flow a bit better. 

Thanks