Packages

This is an old revision of Packages, edited by L505 at . The current revision is 38366.

versioning discussion

See also Shared Libraries

The FPC/Lazarus project unfortunately is a bit ambiguous with respect to the term packages. In general it is a collection of related units, but the precise way these units are related varies. The most common definitions:

  1. A set of units in FPC that are treated together for installation purposes. Both in the fpcmake as the newer fppkg packagemanager contexts.
  2. A set of units in Lazarus (with designtime parts that require registration into the IDE). See Lazarus Packages
  3. A set of units compiled into a DLL with some extra features that make the package a logical part of the main program, instead of an "external" part. Also called Delphi package. I call this Library package from now on.

This article is meant as a brainstorm session, and requirements analysis relating to the latter definition, one of the missing pieces of the Delphi compability. Most created from browsing an hour on the web, looking at pages about fixes and modding the Delphi packages system, and examining the linker backend of FPC.

Note that in time Lazarus packages might be implemented as Library (Delphi) packages.

Delphi packages

Since Delphi packages is a ambiguous name (it refers both to the IDE angle and the principle), I think Library package would be a better designation. I'll at least use it for this article till something better comes up.

A Library-package can be seen from an implementation point, then it is essentially a DLL with some extra's, or from a language point, then it is a library that consists out of a couple of units, and the library itself (the package) is also an own entity with respect to dependancies. (have a dependancy on the library, then you have a dependancy on all units in it).

What do we need them for?

A library package allows to transparently split a program in multiple binaries (exe + dll/so) . The dynamical variant also allows for plugin systems. The transparant issue is the key that sets library packages apart from normal libraries. Packages allow splitting the program up by grouping units into librarypackages without much additional effort. The transparancy is not 100%, unit dependancies may shift, and not any division of a codebase into packages might be possible due to unit initialization order requirements. However these cases will be rare.

A side Windows specific use might be implementing COM components that fully integrate with the mainprogram. This because a COM component is usually done by registering a DLL (not a .EXE). A package is also a DLL, but they are still part of the main program. I mention this because the Delphi OO bridge package (Open Office connectivity pack by Clootie) might use this.

Implementation

Some implementation points:

References

What needs to be done

This roughly splits into three or four parts: (note that afaik Delphi supports all this)

  1. Be able to create and use packages on all platforms. (statically)
    1. RTL into a DLL
    2. mainprogram uses RTL, and can access all symbols it could without. (properties, vars, typed constants, functions, RTTI)
    3. Add another package to the mix. (and multiple in general)
  2. Be able to use package dynamically
    1. Make sure the RTL checks all required packages for a dynloaded package are loaded.
  3. Implement the language/semantic parts.
    1. language and compiler support for package dependancies (requires etc)
    2. Some way to handle versioning (PPL versioning?)
    3. Delphi's "build with packages" and the package list to link is signaled from outside the source. IOW cmdline parameters or a small file with data in the FPC case.

The language support requires a new concept above mere ppu level, that of collection of ppus, that is crosslinked to ppu level. This because on package level, the dependancies are administrated on package level, and in the source on unit level. (the keywords of a "library" unit explicit hint on this "requires" is for package level dependancies, and the unit list is the unit list the package supports). However instead of "requires" this can also be done using cmdline parameters (toggling "build with packages" and specifying the package list in the Delphi IDE)

The system unit init problem (plugin units like memmanager)

In normal FPC programs, the system unit is inited first, and then the main program uses clauses is walked. This allows the user to specify units to initialize before any other unit. If the system unit is in the RTL package, the RTL package will init as a whole. If it is not in a package, it makes plugging units into the basic RTL difficult. So probably the RTL package is an exception in the rule that packages init as a whole. Of course the RTL package can't be loadlibraried, since it is always a requirement for program startup.

So most likely it works as follows:

  1. the system unit inits first, and must be (kept) hardened so that plugin units (like other memmanagers) can plug in. (it already is)
  2. Then units are inited in normal order as would it be a whole program, but a package initializes fully when the first unit would initialize.

Lazarus and Library packages

Even though Delphi uses this system for its packages, it doesn't mean Lazarus should also. Borland was mostly fixed on one platform (this is pre-Kylix) when they designed this.

Since we need packages regardless of what Lazarus does, Lazarus can evaluate at a later time what they use or do. They might e.g. decide against using packages out of versioning reasons (they want to release more frequent than Delphi) or because it can't be implemented in a reasonable way on one of the core platforms Lazarus must support.

Versioning

A somewhat harder nut to crack is library versioning. Borland only released a dozen delphi's over more than a decade (updates are apparantly kept compatible), which keeps the versioning problem somewhat managable for them.

We all know the notorious cygwin, but keep in mind that cygwin also only releases an incompatible DLL every six months, less in more recent times. Granted, currently we don't make a semiannual schedule, but we are also not THAT far from a comparable frequency, and thus similar versionitis.

I don't have any ready made proposals here. It will probably need evaluation after the initial implementation period, how easy it is to maintain compatible versions. However this has the potential to become a stifling compability hold on the minor release schedule.

During a quick brainstorm at a FPC meeting possible problems are:

Temporary Versioning Solution

Lars says: a temporary solution and compromise is to forget about putting all the DSO/DLL/SO files into common system folder. Instead, ship the DLL's with the application in the same directory as the executable. I realize this defeats the purpose of libraries somewhat, but not entirely (and we need a working compromise in the meantime). For example, this allows a plugin system to work since the plugins are stored in the same directory as the program executable. So all plugins for lazarus would be stored in the same directory as the lazarus.exe.

Yes this means multiple copies of the RTL.dll or RTL.so on the hard drive. But most people will not be using packages - only special larger applications like lazarus will be using the RTL.dll. So having multiple copies of RTL.dll is no big deal as a temporary compromise.

As a side note, what we do with Powtils/PSP is embed the DSO/DLL version number into the DLL, and any executable that tries to load the wrong DLL/DSO is simply quits with an error. The library version in Powtils/PSP is stored in an INI file and the user can specify which version to use. I realize FPC has a more complex versioning solution to solve than PSP/Powtils has but thought I would offer some ideas and brainstorming.

Compiling a package

That's easy a library unit is compiled, and "build with packages" is on, a library package will be created. If a unit required during compilation is in a package, that package will become a dependancy.

Compiling with packages

This is a bit harder. The problem is that in the source we find only unit names, and we must somehow translate this to packages. This means that a table must be constructed with unit to package mappings, and package dependancies.

Such a table can be constructed in two ways:

  1. Dynamically -> Scanning for library package header files (*.ppl) and examining them on startup if true.
  2. Statically -> packages must be registered.

I suspect delphi does this dynamically, since runtime packages don't need to be registered. This means that a scan for .ppls (*.dcp in Delphi) and some structures must be added, and the unit search mechanism must be changed to search in these structures first only IF "BUILD WITH PACKAGES" IS ON.

Extensions

Delphi FPC Description
*.dcp *.ppl (ppumove) Combined .ppu files for this library.
*.bpl, .so ? , .so The linkable library file.

Note that combining .ppu's into a ppl means that there won't be duplicate .ppu's (one for static, one for packages), so no need to complicate the dir structure.

Debugging

Our achilles' heel. Can't say much about this. Probably similar to dll/.so debugging.

PPUMove

ppumove is a standalone binary in the FPC distribution that creates a shared libs from already compiled units. As far as I can see, this is already the beginning of a package system, except that it is manual (create packages manually yourself). This support and the makefile targets mostly dates back to the late 0.99.x/1.0.x series. (already before fpcmake)

The RTL (on linux only?) makefile has a "shared" target, and Florian says it should be possible to create a shared lib of the rtl by running make clean all CREATESHARED=1 on Linux. Non x86_32 Linux needs the units to be compiled with PIC though, but the makefile seems to already add -Cg this for non x86_32.

See also