Difference between revisions of "Jenkins"

From Free Pascal wiki
(idea about building fpc without fpc.cfg)
(started db test section)
Line 10: Line 10:
 
Whichever way you install Jenkins, make sure the SVN plugin is enabled
 
Whichever way you install Jenkins, make sure the SVN plugin is enabled
  
 +
==== Find out which user Jenkins runs under ====
 +
If you have problems with scripts/shell commands in jobs not having permissions, it helps if you know which user Jenkins runs under.
 +
Try this command (or of course the documentation):
 +
<syntaxhighlighting lang="bash">
 +
ps aux |grep jenkins | grep -v grep | cut -d" " -f1 | uniq #perhaps only works if jenkins is running a job
 +
</syntaxhighlighting>
 +
On Debian, it gives the tomcat6 user.
 +
 +
==== Stable FPC setup ====
 
Install binutils+stable FPC:
 
Install binutils+stable FPC:
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
Line 118: Line 127:
 
You could archive the test suite results using a Jenkins post-build step.
 
You could archive the test suite results using a Jenkins post-build step.
  
=== FPCUnit/DBTestframework tests ===
+
=== FPCUnit/DBTestframework tests with database output ===
 +
If you want to run the dbtestframework tests and output the results to a separated database server, you can use the testdbwriter programs.
 +
 
 +
Though this section contains a fair amount of setup, fortunately it is once only; each job will reuse the existing configuration.
 +
 
 +
You could of course also run the normal dbtestframework code with output to plain text or XML.
 +
 
 +
Check if mercurial/hg is installed:
 +
<syntaxhighlight lang="bash">
 +
hg --help
 +
</syntaxhighlight>
 +
if not install it (e.g. <syntaxhighlight lang="bash">apt-get install mercurial</syntaxhighlight>).
 +
 
 +
Get the repository version into a fixed directory:
 +
<syntaxhighlight lang="bash">
 +
mkdir -p /opt/testdbwriter
 +
cd /opt
 +
hg clone https://bitbucket.org/reiniero/testdbwriter #will create /opt/testdbwriter
 +
cd /opt/testdbwriter
 +
hg pull #get newest changes from remote repo
 +
hg up #adapt local version to newest changes we just downloaded
 +
</syntaxhighlight>
 +
 
 +
==== Database setup ====
 +
Now the files in this directory can be used to compile the test program with the FPC in each build. We will also need to setup our database connections.
 +
 
 +
===== Database driver setup =====
 +
Make sure the relevant operating system database drivers/client connection libraries are installed, e.g.:
 +
<syntaxhighlight lang="bash">
 +
apt-get install firebird2.5-super postgresql-client freetds-bin freetds-dev mysql-client #adjust to taste/distribution
 +
# todo: add odbc
 +
</syntaxhighlight>
 +
 
 +
===== Tested databases =====
 +
First the databases where the tests will run on. Please create an empty database (or at least one that the test user can thrash) on each db server that you want to test, and have a username and password ready.
 +
 
 +
Then edit database.ini, used by the db test framework tests (see [[Databases#Running_FPC_database_tests]]
 +
<syntaxhighlight lang="bash">
 +
cp database.ini.txt database.ini
 +
nano database.ini # or another editor.
 +
# now set up your credentials in each section
 +
</syntaxhighlight>
 +
 
 +
 
 +
===== Test results database =====
 +
Now the database where the results should go to. Set up a database with the instructions pointed to in the readme.txt file (e.g. use the ''testdbwriter.sql'' for Firebird/Interbase databases and ''testdbwriter_postgresql.sql'' for PostgreSQL databases).
 +
 
 +
Now edit the ini file:
 +
<syntaxhighlight lang="bash">
 +
cp testdbwriter.ini.txt testdbwriter.ini
 +
nano testdbwriter.ini #or another editor
 +
# now set up your credentials in the right section, and choose your profile/database in the [Database] section
 +
</syntaxhighlight>
 +
 
 
* To do: download https://bitbucket.org/reiniero/testdbwriter code and compile/run it in the job.
 
* To do: download https://bitbucket.org/reiniero/testdbwriter code and compile/run it in the job.
  

Revision as of 14:07, 19 October 2012

Overview

Jenkins is a continuous integration server that can be used to automatically build and test projects based on commits to your source control system.

Use with FPC/Lazarus

This section describes how to set up Jenkins to automatically build FPC when a commit happens in SVN.

Jenkins set up

A possible version to use is Turnkey Linux Jenkins on Debian 6.0.5 (Squeeze)

Whichever way you install Jenkins, make sure the SVN plugin is enabled

Find out which user Jenkins runs under

If you have problems with scripts/shell commands in jobs not having permissions, it helps if you know which user Jenkins runs under. Try this command (or of course the documentation): <syntaxhighlighting lang="bash"> ps aux |grep jenkins | grep -v grep | cut -d" " -f1 | uniq #perhaps only works if jenkins is running a job </syntaxhighlighting> On Debian, it gives the tomcat6 user.

Stable FPC setup

Install binutils+stable FPC:

apt-get instal build-essential
#fpc would get old FPC 2.4 on this Debian Squeeze=> symptom in Jenkins build log: 
#generic.inc(2505,13) Fatal: Internal error 8
#Fatal: Compilation aborted
#make[7]: *** [system.ppu] Error 1

#get stable compiler (2.6 at time of writing); we're on a 32 bit system:
cd /root
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-compiler-2.6.0_2.6.0-0_i386.deb/download
mv download fp-compiler-2.6.0_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-compiler_2.6.0-0_i386.deb/download
mv download fp-compiler_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-units-rtl_2.6.0-0_i386.deb/download
mv download fp-units-rtl_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-units-rtl-2.6.0_2.6.0-0_i386.deb/download
mv download fp-units-rtl-2.6.0_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-units-i386-2.6.0_2.6.0-0_i386.deb/download
mv download fp-units-i386-2.6.0_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-units-i386_2.6.0-0_i386.deb/download
mv download fp-units-i386_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-utils-2.6.0_2.6.0-0_i386.deb/download
mv download fp-utils-2.6.0_2.6.0-0_i386.deb
wget http://sourceforge.net/projects/freepascal/files/Linux/2.6.0/deb/fp-utils_2.6.0-0_i386.deb/download
mv download fp-utils_2.6.0-0_i386.deb

dpkg -i fp-units-rtl-2.6.0_2.6.0-0_i386.deb
dpkg -i fp-units-rtl_2.6.0-0_i386.deb
dpkg -i fp-compiler-2.6.0_2.6.0-0_i386.deb
dpkg -i fp-utils-2.6.0_2.6.0-0_i386.deb
#no idea what the other .debs are for that we downloaded
#todo: this could be trimmed down a bit, but this seems to work

# Get resource compiler, not provided by fpc 2.6 packages:
apt-get install mingw32-binutils
# symlink:
ln -s /usr/bin/i586-mingw32msvc-windres /usr/bin/windres

Setting up jobs

Use the job screen to set the svn source to the FPC trunk or branch you want to use (e.g. http://svn.freepascal.org/svn/fpc/trunk)

Set 'use svn update as much as possible'

Polling every 5 minutes could be a good setup. Alternatively, you can set up your own SVN server and write a commit hook script for that. This would result in this entry:

# every 5 minutes
*/5 * * * *

As build commands, the shell commnds

make all

and, to install inside the "workspace" or top of the build directory environment:

make install PREFIX=$WORKSPACE

Note: installing somewhere else may be a good idea, please refer to the Build FAQ.

Make install does not generate an fpc.cfg for you; you have to do that yourself:

$WORKSPACE/bin/fpcmkcfg -o $WORKSPACE/bin/fpc.cfg

A problem: /etc/fpc.cfg

The stable we compiled using packages works, but puts an fpc.cfg into the /etc directory. This fpc.cfg is picked up by default and there is no way to disable that unless you e.g. call a script fpc.sh as your compiler instead of the fpc executable. This script passes options to the real compiler so it ignores /etc/fpc.cfg

This will give problems with running e.g. the compiler test suite.

  • to do: investigate use of PPC_CONFIG_PATH environment variable or fpc.cfg in home dir, which may work.
  • to do: perhaps a simpler solution is just not to install FPC/FPC sources, but only download a bootstrap compiler. Therefore no fpc.cfg, and no problem. Bonus is that we build it the "official way".

To work around is, you can use fpcup; see the next section.


Solution: getting FPC using fpcup

The fpcup tool has done this (and more), so a solution could be:

  • get fpcup into your workspace
  • change build steps to call fpcup with --only=fpc --fpcrevision=$SVN_REVISION

We could have run fpcup so it downloads the latest SVN version, but then we lose the integration with subversion

Download fpcup (bitness should match your Jenkins machine). Put e.g. fpcup_linux_x86 in your /usr/local/bin directory, then:

chmod ugo+rx /usr/local/bin/fcpup_linux_x86

And set your build steps like this:

rm --force $WORKSPACE/fpcup.log #remove any existing logs; use force to not generate error message

Adjust repository (trunk, fixes etc) to taste - and to match the repository you selected in Jenkins - below:

/usr/local/bin/fpcup_linux_x86 --fpcbootstrapdir=$WORKSPACE/fpcbootstrap --fpcdir=$WORKSPACE --fpcuplinkname= --fpcURL=http://svn.freepascal.org/svn/fpc/trunk --fpcrevision=$SVN_REVISION --keeplocalchanges --lazdir=$WORKSPACE/lazarus --lazlinkname= --logfilename=$WORKSPACE/fpcup.log --noconfirm --primary-config-path=$WORKSPACE/lazarusconfig --only=fpc
$WORKSPACE/bin/i386-linux/fpc.sh -iD; $WORKSPACE/bin/i386-linux/fpc.sh -iW #get FPC version to if it was built correctly

The last build step runs the "compiler test suite".

cd $WORKSPACE/tests; make all TEST_FPC=$WORKSPACE/bin/i386-linux/fpc.sh #also make full doesn't work, can't delete file

You could archive the test suite results using a Jenkins post-build step.

FPCUnit/DBTestframework tests with database output

If you want to run the dbtestframework tests and output the results to a separated database server, you can use the testdbwriter programs.

Though this section contains a fair amount of setup, fortunately it is once only; each job will reuse the existing configuration.

You could of course also run the normal dbtestframework code with output to plain text or XML.

Check if mercurial/hg is installed:

hg --help

if not install it (e.g.

apt-get install mercurial

).

Get the repository version into a fixed directory:

mkdir -p /opt/testdbwriter
cd /opt
hg clone https://bitbucket.org/reiniero/testdbwriter #will create /opt/testdbwriter
cd /opt/testdbwriter
hg pull #get newest changes from remote repo
hg up #adapt local version to newest changes we just downloaded

Database setup

Now the files in this directory can be used to compile the test program with the FPC in each build. We will also need to setup our database connections.

Database driver setup

Make sure the relevant operating system database drivers/client connection libraries are installed, e.g.:

apt-get install firebird2.5-super postgresql-client freetds-bin freetds-dev mysql-client #adjust to taste/distribution
# todo: add odbc
Tested databases

First the databases where the tests will run on. Please create an empty database (or at least one that the test user can thrash) on each db server that you want to test, and have a username and password ready.

Then edit database.ini, used by the db test framework tests (see Databases#Running_FPC_database_tests

cp database.ini.txt database.ini
nano database.ini # or another editor.
# now set up your credentials in each section


Test results database

Now the database where the results should go to. Set up a database with the instructions pointed to in the readme.txt file (e.g. use the testdbwriter.sql for Firebird/Interbase databases and testdbwriter_postgresql.sql for PostgreSQL databases).

Now edit the ini file:

cp testdbwriter.ini.txt testdbwriter.ini
nano testdbwriter.ini #or another editor
# now set up your credentials in the right section, and choose your profile/database in the [Database] section

Enhancements

Possible enhancements:

  • run make clean before building!?!
  • run make install into a different prefix
  • run Lazarus compile
  • run dbtestframework test suite, save results (in database or via XML if Jenkins supports that)
  • add Windows (etc) build slaves
  • add installer builders
  • different repositories


Windows/Linux... build slave

You can set up other machines apart from the Jenkins server to act as "slaves": they are controlled from Jenkins and used to run jobs.

This can be useful when running a Windows build/test run from a Linux server.

See [1] for details on how to set up Windows build slaves via DCOM (note: there are other ways to set up (Windows) build slaves; please refer to the Jenkins documentation)