FPC Unicode support

Introduction

Free Pascal compiler and RTL/FCL should natively support Unicode. Upcoming Delphi release codenamed Tiburon will natively support Unicode as well. FPC must be compatible with Delphi in Unicode support.

Even if Tiburon is not released yet we can start to do background work for Unicode support in FPC.

Tiburon Unicode support

Currently we have some information about Tiburon's Unicode support implementation.

http://blogs.codegear.com/abauer/2008/01/09/38845

http://blogs.codegear.com/abauer/2008/07/16/38864

FPC Unicode support

FPC must have the following string types with transparent conversion between them (like current AnsiString <-> WideString conversion):

• ansistring
• widestring
• utf8string
• utf16string
• utf32string
• ucs2string (?)
• ucs4string (?)

Development and further maintenance of these string types must be as simple as possible.

Compiler uses generic structure and helper routines to handle all refcounted string types.

String header:

type
  TRefStringRec = packed record
    Encoding: word;    // encoding of string
    ElementSize: byte; // size in bytes of string's element (1-4)
    Ref: SizeInt;      // number of references
    Len: SizeInt;      // number of elements is string 
  end;

Helper routines will know how to handle string from its header.

Extra parameter with string type information is passed to some routines (like fpc_RefString_SetLength) to allow properly initialize new strings.

widestring type on Windows targets remains non-refcounted and OLE compatible. Minimal number of helper routines is used for it. On non-Windows targets widestring is alias to utf16string.

The compiler uses helpers for string type conversions like this:

 procedure fpc_ansistring_to_utf16string(out dst: utf16string; const src: ansistring);
 procedure fpc_utf32string_to_utf16string(out dst: utf16string; const src: utf32string);

The compiler generates helper procedure name from type names. The compiler does not perform any string conversion handling by itself.