LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Read a .CSV file generated by LabView into Python

Hi,

 

I'm trying to help with a problem my daughter has...

 

She has a CSV file generated by LabVIEW from a myRIO oscilloscope and wants to read it into a python script to plot it using matplotlib. The file can be read when you open it in Excel but in python the read command fro mthe CSV module returns an error:

 

rawdata=open('Ch0_007.csv',"r").read()
File "C:\Users\Username\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 142: character maps to <undefined>

 

I've tried changing the encoding to UTF-8 and also to detect the encoding with Chardet but I get the same errors.

 

Can anyone point to how I can read this CSV using python, please?

 

Thanks in advance

 

0 Kudos
Message 1 of 16
(5,333 Views)

Why do you ask about the python on the NI forum and why you ask how to read the file, but do not show the file?

0 Kudos
Message 2 of 16
(5,309 Views)

Well it is not really about python - more the format of the csv file coming out of LabVIEW and the myRIO device (Not an oscilloscope apparently - it measures electromagnetic forces of protons spinning in a water sample).

 

I don't know anything about LabVIEW or myRIO. I googled on the two and got buckets of NI links so I made the assumption that NI is the owner of LabVIEW. Not the case? Any ideas who is?

 

On the file per se - I didn't attach it because it is massive. My apologies. Here is an excerpt which I have edited in excel and as far as I can tell has the same format.

0 Kudos
Message 3 of 16
(5,300 Views)

LabVIEW yes, but csv - not (see https://en.wikipedia.org/wiki/Comma-separated_values).
CSV is just text file, it's not an excell-file (open it with notepad++ or any othet text editor).
Error text: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 142
So, you need only 150-200 first bytes for analyse the error
Also you can zip this file, because text files is good for archiving.
But, you've written "Here is an excerpt which I have edited in excel and as far as I can tell has the same format", but I don't see the file

0 Kudos
Message 4 of 16
(5,296 Views)

Apologies for the missing file - I'm having trouble up loading stuff (Site keeps saying the upload is failing. Seems like at least partially it is.)


If LabVIEW produces .csv files, I'm not clear why a question on the encoding of .csv  is not legitimate especially given that the protocol is not fully defined on exactly the point I was asking (encoding)


Thanks for your thoughts, however I guess we are both wasting our time...

0 Kudos
Message 5 of 16
(5,282 Views)

Hi John,

 

If LabVIEW produces .csv files,

It's not "LabVIEW", which is producing those CSV files. It's a software made using LabVIEW - and it's up to the programmer to create those CSV files. Did oyu ask the creator of those tools made using LabVIEW?

(You also don't talk about "Visual C" when you want to discuss DOCX file format, don't you?)

 

I'm not clear why a question on the encoding of .csv is not legitimate especially given that the protocol is not fully defined on exactly the point I was asking (encoding)

It's a legitimate question, but your failed in uploading (a part of) your CSV file! By default there should be a comma as separator (CSV = comma seperated values), but for me Excel uses semicolon as separator char: so please show an example of your files!

Use a (better) text editor like Notepad++, cut the file down to the first 1 or 2kB of content, save the file. Then either change the suffix to "TXT" or (better) ZIP the file before attaching!

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
0 Kudos
Message 6 of 16
(5,280 Views)

@MyBrotherJohn wrote:

If LabVIEW produces .csv files, I'm not clear why a question on the encoding of .csv  is not legitimate especially given that the protocol is not fully defined on exactly the point I was asking (encoding)

 


LabVIEW does not produce csv files, but has a tool to generically write arrays into formatted files, given a format, a column delimiter (tab by default, but could be anything such as comma or semicolon. Care must be takes to not create conflict with the decimal delimiter). The row delimiter is newline and there are special codes if the array has more than 2 dimensions.

 

Any file create in LabVIEW can be given the .csv extension, but that still does not make it a csv file. Even windows does not enforce that a csv file must contain data compatible with the format. It just uses the extension to suggest suitable applications to open it and it is up to the creator if it will work.

 

You still have not attached your csv file, else we could tell you exactly what the problem is. Most likely all you need is to adapt the python code accordingly.

 

Do you have the LabVIEW code that writes the file? Maybe you can attach that VI too.

 

0 Kudos
Message 7 of 16
(5,274 Views)

As others have mentioned, a CSV file is a text file with a "column-separator character" (traditionally a comma, as CSV means Comma-separated values, but the default column-separator if LabVIEW is writing the file using Write Delimited Spreadsheet is a tab, 0x09) between strings (typically representing numeric values) and with rows delimited by the system's "End-of-Line" character(s), typically 0x0d, 0x0a (or <CR><LF>).

 

Try opening your .CSV file in Notepad.  If it doesn't open, or doesn't look like a lot of lines of (probably) numbers separated by commas or <tab>s, then it isn't in .csv form.  Otherwise, you should be able to save, say, 3 lines of this file (trimming it to 5-10 columns if necessary) and attach that for us to see.  A pure text file such as your "reduced CSV" should be no problem to attach, and should only be a few hundred bytes.

 

Bob Schor

0 Kudos
Message 8 of 16
(5,271 Views)

Assuming that your CSV file was created with the Write Delimited Spreadsheet function that comes with LabVIEW there is really nothing special about the file created. And yes this is a big assumption although not entirely unfounded, but there is nothing in Windows that enforces a specific file ending for a specific format. For all it's worth it could be simply written with the Write Binary File function and the file given a CSV file ending just for the lolz of pestering the next user (or not understanding the difference between these LabVIEW functions).

 

So assuming the file was written with the Write Delimited Spreadsheet function it will then be in ASCII (not UTF-8 or any other Unicode format) although treating an 7-bit ASCII file as UTF-8 should not be a problem (and I'm not really sure how the Write Delimited Spreadsheet function could generate any 8-bit ASCII characters). And since the cRIO is a Unix/Linux platform the line ending will be a single linefeed character, not a carriage return/linefeed combo as is common on Windows.

 

Then there remain two more possible culprits, the field seperation character can be defined as a parameter to the Write Delimited Spreadsheet format function, so did your daughter really use a comma there? The default for the Write Delimited Spreadsheet is actually a tab. Last but not least you have the potential problem of decimal character difference. While the US uses a decimal point as decimal indicator, some countries in Europe use a decimal comma, which can of course wreck havoc when using a comma separated spreadsheet format. For that reason I usually prefer to use tab separated fields in Spreadsheet files. What decimal character the Write Spreadsheet File function uses is by default defined by the current locale of the system. On the myRIO that is the Unix locale which most likely is set to US/en but on your Windows system that is defined by the settings in your Control panel, and the numeric format can be changed seperately too.

Rolf Kalbermatter  My Blog
DEMO, Electronic and Mechanical Support department, room 36.LB00.390
0 Kudos
Message 9 of 16
(5,245 Views)

Thanks for all the replies!

 

To [try to] answer everyone - I hadn't appreciated that LabVIEW was a tool to create code which then output files. My ignorance. (I've never seen LabVIEW). I think this code must have been written more than 3 years ago in academia from an email thread I have going with my daughter. I will ask my daughter (Student) if she can get hold of it but I suspect the chances are slim.

 

I understand the point about the relevance of this question in this forum, now - so I doubly appreciate your contributions.

 

I opened the .csv file in Notepad and it is not a comma (or separator) separated values as a number of you divined might be a problem. Neither do I think it is an xml file (I have in the back of my mind the xml file have to start with a '<' or something similar). Either way it opens in excel both on a mac and windows and doesn't open in a browser if I change the extension to .xml and double click it. (No complaints tho')

 

I've attached an excerpt (I hope. Third time lucky!). I created it by opening it in excel and deleting all the rows except the first 20 and then saving it. Visually in notepad the original and the excerpt look the same.

0 Kudos
Message 10 of 16
(5,230 Views)