I know there’s various standards of unicode, but I don’t know what the difference is between them. Does anyone actually do know?
As far as I’m aware, there’s UTF-8, UTF-16, UTF-24, UCS-2, and UCS-4. Did I miss any? What’s the difference between all these?
The number is the minimum number of bits used.
utf8 can be 8, 16, oŕ 32 bits
utf16 can be 16 or 32 bits
utf8 (or part of it) is backward compatable with ascii.
Thats all I know
Doing a web search show that there is no official UTF-24.
Perhaps that should have been utf32
Okay, maybe it was just a tpyo in the previous post. Buy yes there is a UTF-32.
The number of hex digits required to represent a character in UTF-8 depends on the number of bytes used:
| Byte Count | Hex Digits Required | Example Characters |
|---|---|---|
| 1 byte | 2 hex digits | A, z, 5, ! |
| 2 bytes | 4 hex digits | é, ñ, ü |
| 3 bytes | 6 hex digits | 中, 日, 한 |
| 4 bytes | 8 hex digits |  ,  ,  |
(from google AI)
I dont understand how UTF8 can use a variable number of bytes per character. ?
In a byte stream, how does it know where the character boundaries are?
To be honest, I don’t understand UTF at all, so the following is what I read from asking Copilot about it.
"Why UTF exists at all
Because Unicode has 1.1 million possible characters, and ASCII only had 128.
UTF is the bridge between:
This is weird.
I Unicode just a list of characters … without any computer byte codings?
I take it this is about non-graphic characters … ie fonts are not involved
How are non-graphic characters physically represented ? What are the limits on non-graphic devices? I know an ascii terminal can only do ascii chadacters. What can character mode on a graphics terminal do?
UCS-2 is a fixed width character encoding . … 2 bytes per character.
I can see how fixed width works.
I cant see how something like UTF-8 with variable width character encodings could work. … one would never know what size the next character in a stream of bytes was going to be… it must put in more bytes to tell it how long each upcoming character is.
Well Neville, this is completely out of my ballpark. I worked with IBM mainframes and they used EBCDIC and hexadecimal. I don’t mind using an AI and this is a bit interesting because it is all new to me. So here the info I learn from Copilot.
It seems the first part of the first byte tells the length of the variable length character.
Examples: 0xxx xxxx (only a one bye char.) 110x xxxx (2 bytes in length)
1110 xxxx (3 bytes) and 1111 xxxx (4 bytes)
So asking for an example, Copilot gave me é (e acute). Confusing, but in UTF the code is an E9 (U+00E9) and would be C3 A9 - 1100 0011 1010 1001 with the 11 in the 1st byte telling the length and the first 10 in the second byte saying the second part. The actual code is 1110 1001 (E9).
That is Unicode, I think, not UTF
It seems that if writing in C you should specify the Unicode \U000000E9 because then it is independent of what your computer is set at.
So e acute is a 3 byte utf code , plus one byte at the front for its length. … but 1100 means 2 bytes … I dont get it?
… or is those 4 bytes C3A9 without the length byte? .. so it would actually be 5 bytes with the length added?
Yes I encountered EBCDIC on IBM computers, and BCD before it.
Control Data computers worked with
ASCII.
The explanation I got was;
“U+00E9 is the character’s number in Unicode. UTF‑8 is the byte encoding of that number.”
So yes, E9 (U+00E9 is the code and the C3 A9 is the way it is encoded with UTF-8. The example of C3 A9 is only 2 bytes. The code plus the E9 (one byte) looks very strange to me also. As I understand it, the C3 being one byte looks like 1100 0011. The 11 tells it is going to be 2 bytes, but E9 only takes up one byte.
Look at the second byte 1010 1001. Again the 1st two bits, 10, says this is a continue from the first byte. What is left from the second byte after the 10 is removed 10 1001. But E9 takes up one byte, so to complete the byte the last 2 bits from the 1st byte (11) is added to the front. 1110 1001 (E9). 11 form the 1st byte and 10 1001 from the second byte.
That’s my understanding from using 2 bytes, I assume a 3 or 4 byte encoding works the same way.
I feel a headache coming on.
Will try again.
An easier way to see it.
C3 A9 - 1100 0011 1010 1001
-----------ccxx–xx-----cc----------- The cc is the code.
-----------------------11----10 1001 = 1110 1001 = E9
I had to use ‘-’ to make it line up.
I get it thanks