Difference between revisions of "UTF8 strings and characters"

From Free Pascal wiki
Jump to navigationJump to search
Line 39: Line 39:
  
 
Due to the ambiguity of Unicode in general (regardless of encoding), Pos() (just like any compare) might show unexpected behavior, when e.g. one of the string contains decomposed characters, while the other uses the direct codes for the same letter. This is not automatically handled by the RTL.
 
Due to the ambiguity of Unicode in general (regardless of encoding), Pos() (just like any compare) might show unexpected behavior, when e.g. one of the string contains decomposed characters, while the other uses the direct codes for the same letter. This is not automatically handled by the RTL.
 +
 +
==== Search and copy ====
 +
 +
Another example of how Pos() and Copy() work with UTF-8. This function has no code to deal with UTF-8 encoding, yet it works with any valid UTF-8 text always.
 +
 +
<syntaxhighlight>
 +
function SplitInHalf(Txt, Separator: string; out Half1, Half2: string): Boolean;
 +
var
 +
  i: Integer;
 +
begin
 +
  i := Pos(Separator, Txt);
 +
  Result := i > 0;
 +
  if Result then
 +
  begin
 +
    Half1 := Copy(Txt, 1, i-1);
 +
    Half2 := Copy(Txt, i+Length(Separator), Length(Txt));
 +
  end;
 +
end;
 +
</syntaxhighlight>
  
 
====Accessing UTF8 characters====
 
====Accessing UTF8 characters====

Revision as of 01:50, 6 February 2015

Please note that simply iterating over chars as if the string was an array does not work in Unicode. This is not something specific to UTF-8: one simply cannot suppose that a character will have a fixed size in Unicode. If you want to iterate over the characters of an UTF-8 string, there are basically two ways:

  • iterate over the bytes - useful for searching a substring or when looking only at the ASCII characters of the UTF8 string. For example when parsing XML files.
  • iterate over the characters - useful for graphical components like synedit. For example when you want to know the third printed character on the screen.

The beauty of UTF-8

Bytes starting with '0' (0xxxxxxx) are reserved for ASCII-compatible single byte characters. With multi-byte characters the number of 1’s in the leading byte determines the number of bytes the character occupies. Like this :

  • 1 byte : 0xxxxxxx
  • 2 bytes : 110xxxxx 10xxxxxx
  • 3 bytes : 1110xxxx 110xxxxx 10xxxxxx
  • 4 bytes : 11110xxx 1110xxxx 110xxxxx 10xxxxxx

The design of UTF-8 has some benefits over other encodings :

  • It is backwards compatible with ASCII and produces compact data for western languages. ASCII is also used in markup language tags and other metadata which gives UTF-8 an advantage with any language.
  • The integrity of multi-byte data can be verified from the number of '1'-bits at the beginning of each byte.
  • You can always find the start of a multi-byte character even if you jumped to a random byte position.
  • A byte at a certain position in a multi-byte sequence can never be confused with the other bytes. This allows using the old fast string functions like Pos() and Copy() in many situations where UTF-16 would need more complex and slower code. See examples below.
  • Robust code. Code that deals with codepoints must always be done right with UTF-8 because multi-byte characters are common. For UTF-16 there is plenty of sloppy code which assumes characters to be fixed width.
  • The most widely used operating systems, including Android, now use UTF-8 natively. It makes sense to use it in applications, too. Windows used to be a dominant platform but it is no more.

Searching a substring

Due to the special nature of UTF8 you can simply use the normal string functions for searching a sub-string. Searching for a valid UTF-8 string with Pos will always return a valid UTF-8 position:

uses lazutf8;
...
procedure Where(SearchFor, aText: string);
var
  BytePos: LongInt;
  CharacterPos: LongInt;
begin
  BytePos:=Pos(SearchFor,aText);
  CharacterPos:=UTF8Length(PChar(aText),BytePos-1);
  writeln('The substring "',SearchFor,'" is in the text "',aText,'"',
    ' at byte position ',BytePos,' and at character position ',CharacterPos);
end;

Due to the ambiguity of Unicode in general (regardless of encoding), Pos() (just like any compare) might show unexpected behavior, when e.g. one of the string contains decomposed characters, while the other uses the direct codes for the same letter. This is not automatically handled by the RTL.

Search and copy

Another example of how Pos() and Copy() work with UTF-8. This function has no code to deal with UTF-8 encoding, yet it works with any valid UTF-8 text always.

function SplitInHalf(Txt, Separator: string; out Half1, Half2: string): Boolean;
var
  i: Integer;
begin
  i := Pos(Separator, Txt);
  Result := i > 0;
  if Result then
  begin
    Half1 := Copy(Txt, 1, i-1);
    Half2 := Copy(Txt, i+Length(Separator), Length(Txt));
  end;
end;

Accessing UTF8 characters

Unicode characters can vary in length, so the best solution for accessing them is to use an iteration when one intends to access the characters in the sequence in which they are. For iterating through the characters use this code:

uses lazutf8;
...
procedure DoSomethingWithString(AnUTF8String: string);
var
  p: PChar;
  CharLen: integer;
  FirstByte, SecondByte, ThirdByte: Char;
begin
  p:=PChar(AnUTF8String);
  repeat
    CharLen := UTF8CharacterLength(p);

    // Here you have a pointer to the char and its length
    // You can access the bytes of the UTF-8 Char like this:
    if CharLen >= 1 then FirstByte := P[0];
    if CharLen >= 2 then SecondByte := P[1];
    if CharLen >= 3 then ThirdByte := P[2];

    inc(p,CharLen);
  until (CharLen=0) or (p^ = #0);
end;

Accessing the Nth UTF8 character

Besides iterating one might also want to have random access to UTF-8 Characters.

uses lazutf8;
...
var
  AnUTF8String, NthChar: string;
begin
  NthChar := UTF8Copy(AnUTF8String, N, 1);

Showing character codepoints with UTF8CharacterToUnicode

The following demonstrates how to show the 32bit code point value of each character in an UTF8 string:

uses lazutf8;
...
procedure IterateUTF8Characters(const AnUTF8String: string);
var
  p: PChar;
  unicode: Cardinal;
  CharLen: integer;
begin
  p:=PChar(AnUTF8String);
  repeat
    unicode:=UTF8CharacterToUnicode(p,CharLen);
    writeln('Unicode=',unicode);
    inc(p,CharLen);
  until (CharLen=0) or (unicode=0);
end;

Mac OS X

The file functions of the FileUtil unit also take care of Mac OS X specific behaviour: OS X normalizes filenames. For example the filename 'ä.txt' can be encoded in Unicode with two different sequences (#$C3#$A4 and 'a'#$CC#$88). Under Linux and BSD you can create a filename with both encodings. OS X automatically converts the a umlaut to the three byte sequence. This means:

if Filename1 = Filename2 then ... // is not sufficient under OS X
if AnsiCompareFileName(Filename1, Filename2) = 0 then ... // not sufficient under fpc 2.2.2, not even with cwstring
if CompareFilenames(Filename1, Filename2) = 0 then ... // this always works (unit FileUtil or FileProcs

See also