Rif.: .csv data to array FileToArray, decimal comma

[razzzz]

Hello,

I need to read a .csv file and writ the values in an array.

The csv is generated externally and I have no options to change it's generation format.

The problem is: the values are double and the decimal numbers are seperated by a comma instead of a dot. The seperation of each individual data is already in columns if i open it in excel, and seperated by semicolons if i open txt-editor.

FileToArray interprets comma as a seperator. Is there a way to change it, so that it is not interpreted at all but the semicolon. Is there a way so that cvi changes the comma to a dot for decimal-seperation of the double-type values? Is there another option to read the csv in an array?

Thanks in advance.

[RobertoBozzolo]

I'm not aware of a way to tailor FileToArray behaviour. If you don't want to programmatically process the file scanning numbers out you could think to preprocess the file and substitute the decimal symbol before passing it to the command.

The following code fragment permits you to process the file in a single pass and save it in-place:

#include <formatio.h>
#include "toolbox.h"
ssize_t	size;
int		fH;
char *buffer = NULL;

FileExists ("myFile.csv", &size);
buffer = malloc (size);
fH = OpenFile ("myFile.csv", VAL_READ_WRITE, VAL_OPEN_AS_IS, VAL_BINARY);
ReadFile (fH, buffer, size);
SetFilePtr (fH, 0, 0);
DelocalizeNumberString (buffer);
WriteFile (fH, buffer, size);
CloseFile (fH);
free (buffer);

DelocalizeNumberString is part of the Programmer's Toolbox that ships with CVI.

[razzzz]

I was already searching for something like read the csv as string and try to replace the comma with dots. Didnt find this intern function, though. SO thanks alot. I think this will be the right way.

Yet I can't get it to work:

"FATAL RUN-TIME ERROR:   "functions.c", line 73, col 27, thread id 0x000027A4:   Null pointer argument to library function." (error occurs in ReadFile())

I am pretty confident to have implemented the way you did (SEE EDIT):

int FileHandle;
ssize_t allocation_size;
char *FileBuffer = NULL;
	
FileExists (pathname, &allocation_size);
	FileHandle=OpenFile(pathname,VAL_READ_WRITE,VAL_OPEN_AS_IS,VAL_BINARY);
ReadFile (FileHandle, FileBuffer, allocation_size);
SetFilePtr (FileHandle, 0, 0);
DelocalizeNumberString (FileBuffer);
WriteFile (FileHandle, FileBuffer, allocation_size);
CloseFile (FileHandle);
free (FileBuffer);

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

EDIT: sorry i did not see the allocation with malloc, implemented it, but get another error, but have to look into it first myself

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Also one further question: why is it VAL_BINARY and not VAL_ASCII?

[razzzz]

The new error code appears in function DelocalizeNumberString and is:

FATAL RUN-TIME ERROR:   "functions.c", line 76, col 5, thread id 0x00002350:   The program has caused a 'General Protection' fault at 0x687BC4E8.

I can't figure out what is causing this. I remember to have this error a few days before within another project and somehow resolved it, yet didn't note the solution and can't help me to remember. Does it have something to do with the libraries? The toolbox.fp is added already.

[RobertoBozzolo]

This error normally is not related to a missing library, which is usually detected at compile time.

If you have loaded the toolbox as an instrument you could try running it at source code level and see where the error arises: execute Instrument >> Edit function, select the toolbox in Edit Instrument window and press Attach and Edit Source button. After this, run the code and see if and where it stops on error within the toolbox function.

Is the system using a multibyte OS localization? This should be handled in the function but it may be worth knowing.

EDIT What happens if you use a fixed-sized string instead of dynamically allocating it?

[razzzz]

Dear RobertoBozzolo,

thanks for your effort. A fixed size does not help, the error is the same but the "at" adress changes to  0x0040242E if that may be of interest to someone.

The first suggestion of you, I did not understand at all. The toolbox is added as library. I tried to add it as instrument, but could not, as it is already a library-entry. I also cant access the edit. Do you mean by excecute, to press the green arrow (debug project)? Then i can not acess the upper menu-bar, unless i click once on the red stop sign (break excecution).

What is mjultibyte OS localization? I am using windows 7, 64bit , german localization but with multiple language sets available, if that is what you mean.

It is sad that the function where the error occurs does not open up with the highlighted line which causes the error 😞

EDIT: i also tried with a smaller csv, but same error

I cant add toolbox.fp als instrument, even when i cut it from the library list and restart cvi it says it is already included as library. But i also think it is not the toolbox.

[RobertoBozzolo]

Since DelocalizeNumberString pertains to the Programmer's Toolbox, it is very likely that the error arises in that library. Now, to have it added as an instrument you may need to unload some other instruments that rely on that library (e.g. the inifile instrument, if you are using it). After unloading all libraries, you may add them into the instrument menu. After doing that, on the libraries distributed with the source code (like the toolbox) you can attach the source so that you can process the code step by step.

Now, I understand that this procedure may be boring... you could try copying function code from the toolbox and create a new function into your sources with a different name and execute that one, just to check what is happening.

Multibyte strings are those for languages like chinese or japanese or so, where each character is actually stored into two bytes. As you will see, DelocalizeNumberString  function makes use of multibyte macros to advance on the string, so I thought that could be a point of interest. German locale is not multibyte so you should not be affected by this.

May it be that the csv file includes some strange (i.e. not text) character?

[RobertoBozzolo · Accepted solution]

Another hint that comes to my mind: a GPF may be related to the function reading/writing  to a non-legitimate area. Since the function expects a string, it must be properly terminated with a null byte. Try allocating a size+1 area and set all to 0 before reading the file and calling DelocalizeNumberString. 

[razzzz]

Dear RobertoBozzolo,

your hints were gold. But first of all, sorry for checking your suggestions just now, we had a long weekend / holiday.

With your first instruction regarding the "library as instrument" (which was an excellent advice for the future as well, thank you very much) I was able to track down the generated GPF. Within the function delocalizenumberstring the error occured:

FATAL RUN-TIME ERROR:   "toolbox.c", line 6105, col 16, thread id 0x00002154:   Dereference of out-of-bounds pointer: 1 bytes (1 elements) past end of array.

After inserting your second hint:

int i=0;
int FileHandle;
ssize_t allocation_size;
char *FileBuffer = NULL;
	
	FileExists (pathname, &allocation_size);
	FileBuffer = malloc (allocation_size+1);
	for (i=0;i<allocation_size+1;i++)
		 FileBuffer[i]=0;
	FileHandle=OpenFile(pathname,VAL_READ_WRITE,VAL_OPEN_AS_IS,VAL_BINARY);
	ReadFile (FileHandle, FileBuffer, allocation_size);
	SetFilePtr (FileHandle, 0, 0);
	DelocalizeNumberString (FileBuffer);
	WriteFile (FileHandle, FileBuffer, allocation_size);
	CloseFile (FileHandle);
	free (FileBuffer);

It did work finally and your thoughts on the error reason seem to be correct.

Feel hugged and know that I appreciate your help and effort very much.

[RobertoBozzolo]

I'm happy you solved your problem.

Just as a suggestion, you could replace the for loop zero-ing the memory with a single

memset (FileBuffer, 0, allocation_size + 1);