Difference between revisions of "shared library"

From Free Pascal wiki
Jump to navigationJump to search
Line 78: Line 78:
  
 
== Shared Exe Memory manager ==
 
== Shared Exe Memory manager ==
Lars says: regarding a single memory manager for BPL style packages: what about using the executable and exporting its memory manager? I have had problems trying to use CMEM but have successfully shared the same memory manager using SetMemoryManager/GetMemoryManager tricks. See http://www.freepascal.org/contrib/delete.php3?ID=543 for an example of sharing memory between exe and dll without using CMEM. Powtils CGI library also uses this trick for dynpwu.pas so that ansistrings can be used without any sharemem or cmem unit.
+
Lars says: regarding a single memory manager for BPL style packages: what about using the executable and exporting its memory manager? I have had problems trying to use CMEM but have successfully shared the same memory manager using SetMemoryManager/GetMemoryManager tricks.  
 +
 
 +
See http://www.freepascal.org/contrib/delete.php3?ID=543 for an example of sharing memory between exe and dll without using CMEM. Powtils CGI library also uses this trick for dynpwu.pas so that ansistrings can be used without any sharemem or cmem unit. The memory manager is exported from the executable or a single library and shared with all the other DLL/EXE's that are connecting to the single module.
  
 
== References ==
 
== References ==

Revision as of 06:21, 17 September 2007

(here if I say shared library I mean both .so as .dll, unless I really say "unix shared library" or "dll")

Currently there are a lot of Shared libraries bug reports. However, to my knowledge there is no real documentation about how shared libraries work in combination with FPC. That is both how shared libraries work now, how they should work, and how Delphi treats them.

Let's start with a simple sketch of what forms Delphi supports, because we will of course try to be as compatible as multiplatformwise reasonable. Note that this is all from memory and webpages. If you can detail it more, or specify more exact borderconditions, by all means, do).

I'm also totally unaware of Kylix details, so if you know how Kylix implements a certain feature by all means add it :-)

Delphi

Delphi to my knowledge knows three (or four) ways of dynamic linking.

  1. create a standalone shared library. (compiling a library unit without "runtime packages" selected).
    • This means the RTL will be linked into the shared library
    • This also means the memory manager will be its own island. Using automated types in functions that communicate with it is not possible.
    • Classes use is not possible. (both program and shared library have own copies of the VMT, which breaks e.g. IS and AS operators)
  2. create a standalone shared library. (compiling a library unit without "runtime packages" selected), but while using unit sharemem
    • This means the RTL will be linked into the shared library
    • But the memory manager is switched to COM compatible. This means other components/programs that also switch their memory manager to sharemem can call functions that use automated types. (because it doesn't matter who returns the block to the COM memmanagement system).
    • Note that afaik the COM memory manager is quite slow. This road is probably not desired unless you really want to mess with COM, or your componentization is more important than speed.
    • Classes use is not possible. (both program and DLL have own copies of the VMT)
  3. Library packages These are shared libraries for which all dependancies on pascal level are known (which units they contain and depend on), and can be treated as parts of the main program that reside in a DLL.
    • The RTL is in a separate package(DLL), and both the mainprogram and package use it. Therefore there is only one copy of any unit including system.
    • This also means there is only one memory manager (at least for the mainprogram and the packages it uses. IOW shared libs that are not a package can still use an own RTL and memory manager)
    • a package can only depend on units in its own package or in other packages.
    • because no units are duplicated, there is only one copy of each VMT, making classes use transparent.
    • Probably packages can also switch to sharemem, making it compatible with other sharemem using systems. Some way must be found to initialise sharemem as early as possible though (does Delphi do this? Possible test for this is to pass an ansistring created in a init section of a unit in a package to a different sharemem using shared lib that is not a package) T.B.D.

Besides these, Delphi can also generate dlls that are ActiveX components. T.B.D.

Sharemem implementation details

As said, sharemem switches the memory manager to a global one. Under Windows this is afaik the COM manager. On *nix a similar memmanager doesn't always exist ( Gnome and KDE component architectures might have something similar), but it is not guaranteed there.

One could probably simply have all programs use cmem, which could be a "level 0" implementation for everything that runs in this process.

I explictely name this because, it might be necessary to impose an own initialisation order (independant of OS shared lib initialisation) to allow the main program to initialise units right after the RTL (system unit) initializations, but before other libs. e.g. a different memory manager.

VMT duplication

I've heard various opinions about if VMT duplication is a real problem when passing classes between DLL and mainprogram (for the various scenario's). I don't exactly know when it is a problem or not. (if you do, feel free to explain it here). Therefore I just list for all possible scenario's if VMTs are duplicated or not.

In earlier discussions about packages, there was some confusion about this topic. People assumed that packages would somehow tap into the RTL VMT's in the main binary. I'm however pretty sure this is not the case, at least not in Delphi, since when using packages, the RTL always is a package too. And every dependancy of an unit in a package must be in the same package or in a package it has a dependancy on. This means an unit only exists once in the greater program (main program + its packages)

Initialization and finalization sections and RTL

In all cases of shared libraries, the problem of how to initialize and finalize. The ground rule is of course that a unit, and all the units it depends on, must be initialized before it is used. In the case of packages this can lead to a bit more rigid order of initialization, since packages must be inited as a whole (?). Also the (possible?) ability in the mainprogram to insert units before the system unit is used is a problem.

For stand-alone libraries with internal copies of RTL to function, the RTL needs to be initialized, and the initialization section of the library needs to be called as well.

Roughly there are two options here:

  • Use (e.g. on ELF platforms) the ABI .init and .fini sections or similar constructs in other binary formats.
  • Leave the above initializers mostly empty sections, and enforce an own order using a set of own initializers and finalizers. The ELF init and fini sections merely register the real initializers.

Mixed forms are also possible, e.g. initialize standalone shared libraries via the initializer sections, but with packages use an own definition.

E.g. for packages there are two (possible) special requirements that influence the ini/finalize behaviour:

  • A dynamically (loadlibrary/dlload) package must be checked for requirements. If it is auto-initialized, it might initialize before these requirements are checked. This goes in general for version checks too.
  • A mainprogram may want to run units as early as possible (e.g. right after System unit initialization), before the other statically linked packages are initialized (exception hooking, memory managers, threads and widestring plugins etc)

For the latter, assume the mainprogram code looks like this:

uses 
     Drivers, cwstring,cmem;
 begin
 end.

Nearly all of the 4 units (include unit system) allocate and modify hooks globally in their startup. Their initialisation order is important.

However Drivers is part of FV, cwstring and System of the RTL, cmem is in the packages. So at least three packages are involved.

It is logical to leave cmem (and similar units) out of a package, and cwstring can't be in the RTL package anyway because it links to libc. (avoiding which is the prime reason for its existence)


Shared Exe Memory manager

Lars says: regarding a single memory manager for BPL style packages: what about using the executable and exporting its memory manager? I have had problems trying to use CMEM but have successfully shared the same memory manager using SetMemoryManager/GetMemoryManager tricks.

See http://www.freepascal.org/contrib/delete.php3?ID=543 for an example of sharing memory between exe and dll without using CMEM. Powtils CGI library also uses this trick for dynpwu.pas so that ansistrings can be used without any sharemem or cmem unit. The memory manager is exported from the executable or a single library and shared with all the other DLL/EXE's that are connecting to the single module.

References