topic Re: How to determine string is ASCII or Unicode? in LabVIEW

How to determine string is ASCII or Unicode?

harry0725 — Sun, 22 Jan 2017 04:59:54 GMT

As title, I'm trying to detect input string is ASCII or Unicode.

I cannot figure it out. Pls kindly help if anyone.

Thanks!

Re: How to determine string is ASCII or Unicode?

paul_cardinale — Sun, 22 Jan 2017 05:44:08 GMT

For plain Unicode vs ASCII, you can't really tell for certain (although you may be able to guess based on content).

However if it's UTF-16 vs ASCII, the first two bytes will tell you: For UTF-16, the first two bytes are a byte-order-marker (either FF FE or FE FF).

Re: How to determine string is ASCII or Unicode?

harry0725 — Sun, 22 Jan 2017 07:28:55 GMT

I'm not familiar with this, could you explain how to read first 2 bytes is FF FE or FE FF in LabVIEW?

I tried to use "Read from Binary", but always get 0 0 as U8 type.

Re: How to determine string is ASCII or Unicode?

Bob_Schor — Sun, 22 Jan 2017 15:24:58 GMT

Why don't you attach both a sample data file and your code, not a picture, but the actual .VI file you are trying to use? We'd be able to make suggestions that may work for you in your context, and also help you with specific misconceptions or mistakes you've made with your code.

Bob Schor

Re: How to determine string is ASCII or Unicode?

rolfk — Mon, 23 Jan 2017 00:32:02 GMT

As already mentioned Unicode is a coding scheme that does not only contain a single possibility. There are at least 3 major variants called UTF-8, UTF-16 and UTF-32 with the number behind the UTF indicating the number of bits a single code point does occupy. Except for UTF-32 a codepoint is not the same as a character.

UTF-8 text that only contains ASCII-7 bit characters is indistinguishable from an ASCII text file unless it contains an optional (and not recommended) BOM header (0xEF,0xBB,0xBF), although that name is really misleading since UTF-8 is a single byte coding scheme, so there is no byte order to indicate.

UTF-16 can represent most characters of all the living languages around the world in a single code point but the higher characters (usually representing special characters or from artificial languages like Klingon) are represented with multiple codepoints.

UTF-32 can currently represent all defined characters in a single codepoint of 32 bit but the Unicode standard allows for future extensions to include new characters.

To make things more interesting the two multibyte encodings UTF-16 and UTF-32 can be in big endian or little endian byte order and that is where the BOM header as the first character point (mostly used in UTF-16 encoding) comes in handy.

Most Unix systems nowadays use UTF-8 as default text encoding scheme while Windows settled on UTF-16 little endian encoding both internal to the application and when exporting and importing text data. But on Unix, Unicode aware applications use normally internally the wide char representation which on Unix maps to 32 bit Unicode encoding. But when exporting it to and importing it from streams like files, network connections, etc it is usually converted to UTF-8.

Re: How to determine string is ASCII or Unicode?

harry0725 — Wed, 25 Jan 2017 14:43:34 GMT

Thanks for above reply.

Let me explain my question clearly.

I used Unicode support in my LabVIEW development as following metioned:

http://forums.ni.com/t5/Reference-Design-Portal/LabVIEW-Unicode-Programming-Tools/ta-p/3493021

In my application, I hope to programmatically determine the decode type of input String, which could be ASCII or Unicode.

Following is an simple example for attached VI:

String "ASCII" is original LabVIEW text, and "Unicode" is Unicode string produced by method "Force Unicode Text" mentioned in above link.

I'm not sure if this is possible or not, but I cannot figure out a nice solution.

Pls kindly let me know hot to achieve this.

Thanks!

Re: How to determine string is ASCII or Unicode?

paul_cardinale — Wed, 25 Jan 2017 18:21:34 GMT

Make sure that whatever supplies the string uses UTF-16 LE when it's Unicode, then just check the first two bytes of the string.

Re: How to determine string is ASCII or Unicode?

harry0725 — Tue, 31 Jan 2017 15:25:51 GMT

The input string is not supplied from Unicode file.

It's just fixed constant string in LabVIEW.

Most of time, there won't be first two bytes for Unicode string.

Re: How to determine string is ASCII or Unicode?

billko — Tue, 31 Jan 2017 16:07:03 GMT

If the first two bytes are not present, then it isn't "truly" Unicode. It's just an array of bytes waiting to be interpreted somehow. I think you will have to interpret the bytes depending on where they came from.

Re: How to determine string is ASCII or Unicode?

paul_cardinale — Tue, 31 Jan 2017 16:15:33 GMT

In that case, there is no sure-fire way to tell the difference. An ASCII string might contain any sequence of bytes, and a Unicode string also might contain any sequence of bytes, If you display a string as ASCII, then as Unicode, you may find that one is readable (in some language) and the other looks like garbage; thus you could infer the coding. However doing that programmatically is probably intractable. Your best chance (and it's not a good one) is to assume that any Unicode string will contain some (maybe many) characters in the codepoint range 1-255. If there are nulls in the even bytes, it's probably UTF-16 LE. If there are nulls in the odd bytes, it's probably UTF-16 BE. If there are no nulls, it's probably ASCII.

Re: How to determine string is ASCII or Unicode?

paul_cardinale — Tue, 31 Jan 2017 18:01:29 GMT

Not really. Unicode just defines the codepoints. The presence of a Byte Order Marker (BOM), signifies a Unicode Transformation Format (UTF). UTFs are methods of implementing Unicode encoding (i.e. they are wrappers for Unicode, not part of Unicode).

Re: How to determine string is ASCII or Unicode?

harry0725 — Thu, 02 Feb 2017 01:42:33 GMT

I agree with Bill said, there are just array of bytes as Unicode in LabVIEW.

That's why I cannot determine first two bytes to recognize it.

As Paul said, UTF-16 LE is used to present Unicode as even bytes are nulls.

Basically, I can check even bytes and string length to find it's Unicode or not.

However, the biggest problem is that my application including multi-language exchange will perform in English, Traditional Chinese, even Japanese.

In pure English, even bytes method may be working.

But, in Chinese or Japanese, every byte has value for Unicode strings.

Each Chinese character will include two bytes.

Since now, I got one additional problem for how to determine that Unicode string is English or Chinese.

If there's way to define, I can use eve bytes method to figure out English parts, and maybe use first two bytes method for Chinese.

Re: How to determine string is ASCII or Unicode?

billko — Thu, 02 Feb 2017 02:20:38 GMT

This whole thing sounds a bit backwards to me. I think the language issue should be settled at install time. You shouldn't be trying to determine it on the fly.

Re: How to determine string is ASCII or Unicode?

harry0725 — Thu, 02 Feb 2017 09:21:21 GMT

Maybe I make some confuse.

I'm writing a project which will include multi-language function for user to switch.

And I use Unicode support in my LabVIEW development environment.

So I cannot decide which language user is going to use.

In other words, my program should be working for each kind of language.

Of course I can make some additional terminal for my Unicode VIs to define data type and language format.

But it will take lots of unnecessary case structures, make coding seems like hard code.

I prefer not to use this kind of solution.

That's why I'm trying to determine Unicode automatically.

Re: How to determine string is ASCII or Unicode?

billko — Thu, 02 Feb 2017 11:56:05 GMT

You could make identically formatted and named Unicode ini files with keys matching the names of controls you need to change. The values will be Unicode representations of what you want to have there. The ini files will be in different folders, one for each language. Have the user select a language, go to that folder and pull the contents of the ini file into the captions, text or whatever for each control you need to modify. Tedious.

Re: How to determine string is ASCII or Unicode?

rolfk — Thu, 02 Feb 2017 13:08:26 GMT

So you have a user selection for a language and based on that select some language file to read in the strings and apply to the controls. Why would you then need to determine the type of string that is applied? I'm still not really understanding the problem here. The LabVIEW user interface will either be using the (unsupported) Unicode setting or MBCS, but never both. If you need to define multiple languages, and have determined that you can live with the many restrictions of Unicode support in LabVIEW when using the unsupported ini key, make all the necessary controls Unicode and be done with it.

Since you know the language you want to apply, sort the strings accordingly, if it comes from language files do as bill has suggested by putting them in different files (or as I have done in the past to different columns in a tab seperated file and load them accordingly. Have these files correctly encoded, matching the controls encoding. Each file (or column) then defines a default encoding to use, where you can decide if you need to convert the string from the file into the Unicode UTF16LE that the LabVIEW controls need. Et voila!

There simply is no reliable way to detect if a random byte stream is an ASCII string, a Unicode string of any specific type or just some garbage. Sure you can try to parse the byte stream with the various unicode encodings and if you end up with an invalid codepoint, assume that it can't be that encoding, but that is not only unreliable, but also extremely performance intense. So don't do that!!!

You know the language setting your application is in currently. Use that to determine if you need to do special handling rather than trying to guess from a random byte stream what it might me.

Re: How to determine string is ASCII or Unicode?

J-M — Thu, 02 Feb 2017 15:07:48 GMT

Did you take a look to this post:

http://forums.ni.com/t5/LabVIEW/Multilanguage-user-interface/m-p/2744846/highlight/true#M810815

Re: How to determine string is ASCII or Unicode?

harry0725 — Thu, 02 Feb 2017 15:42:02 GMT

To rolfk:

Yes, you are totally correct about what I'm trying to do for my project. Which has multi-language selection for user to switch. I'll try to use your suggestion and see if any solution. Thanks for kind help.

To J-M:

I never read this thread before. Thought it didn't diretly solve my problem, it inspire me how to improve my Unicode system translation. Thanks for that! Sorry about that I can only set one best solution.