The Horde release train

The Horde project did push out 906 releases in the last 8 months since the initial Horde 4 release. PEAR packages, better release management tools and continuous integration seem to pay off. The stream of bug fixes and improvements available to the users has switched to high speed.

At the same the time the use of PEAR packages and automatic DB migrations has lowered the effort of updates on the administrator side to an absolute minimum.

pear upgrade -c horde

And few clicks later you are up-to-date.

The code quality of the basic PEAR tools is an entirely different matter... BUT it is just soo damn cool anyway ;)

Horde4 debian packages - Second round

Nearly half a year passed since I started my first attempt at Debian packages for Horde4. Back then it was just about snapshots which I generated via a continuous integration setup. But Horde4 has been released now, there are packages and it is time for the real thing.

I did get a first set of packages installable today but this is just the initial draft. The main point about it is that the majority of it is automated.

If you want more details and the installation steps then I would suggest to follow my discussion with Mathieu Parent on pkg-php-pear@lists.alioth.debian.org. These were the mails exchanged so far (with the last two detailing my current status):

1. On PEAR packaging (25.5., Mathieu)
2. On PEAR packaging (7.6., Gunnar)
3. On PEAR packaging (7.7., Mathieu)
4. On PEAR packaging (17.7., Gunnar)
5. On PEAR packaging (20.7., Gunnar)

There is not much more to say right now. Any testing and feedback is welcome and there is obviously still a lot of work ahead until this pops up in your default package channels. But it is at least on the horizon and the variant that I ship on files.pardus.de should become useable very soon.

Mixing stable Horde components with snapshots

One thing I completely love about Horde 4 being a component framework is the ability to have stable installations into which I can inject experimental snapshots of packages further developed, patched or hacked in some other way. Just an overview of one such installation:

INSTALLED PACKAGES, CHANNEL PEAR.HORDE.ORG:
===========================================
PACKAGE                   VERSION              STATE
Horde_ActiveSync          1.0.0                stable
Horde_Alarm               1.0.1                stable
Horde_Argv                1.0.1                stable
Horde_Auth                1.0.3                stable
Horde_Autoloader          1.0.0                stable
Horde_Browser             1.0.0                stable
Horde_Cache               1.0.3                stable
Horde_Cli                 1.0.0                stable
Horde_Compress            1.0.1                stable
Horde_Constraint          1.0.0                stable
Horde_Controller          1.0.0                stable
Horde_Core                1.1.2dev201105300702 stable
Horde_Crypt               1.0.2                stable
Horde_Data                1.0.0                stable
Horde_DataTree            1.0.0                stable
Horde_Date                1.0.1                stable
Horde_Date_Parser         1.0.0                stable
Horde_Db                  1.0.1                stable
Horde_Editor              1.0.1                stable
Horde_Exception           1.0.1                stable
Horde_Feed                1.0.1dev201106011224 stable
Horde_Form                1.0.1                stable
Horde_Group               1.0.0                stable
...

How do you get these snapshots? Ensure you have the horde-component helper set up. Then enter the package you wish to snapshot (e.g. Horde_Core) and run:

horde-components snapshot

The snapshot package will be assembled in this directory, can be uploaded to your webserver and installed using:

pear upgrade --offline --force Horde_Core-1.1.2dev201105300702.tgz

What happens if you patched and deployed your package, sent the patch upstream, it gets accepted and a new package released?

pear upgrade horde/Horde_Core

Magic!

Generating a PEAR test environment for releasing PEAR packages

As a developer you will often have a high number of different libraries installed. This will usually not match the users situation that will be nearer to the minimal number of required libraries. This may turn out to be a problem when releasing packages as missing dependencies can be easily overlooked.

For releasing PHP PEAR packages it makes sense to get a separate PEAR environment for testing if the package to be release has a correct set of dependencies. If PEAR is already installed on the system, the steps for that are straight forward.

Setup the new repository

Create the test environment with

# mkdir ~/pear-test

A separate PEAR configuration will be needed there:

# pear config-create ~/pear-test ~/pear-test/.pearrc

CONFIGURATION (CHANNEL PEAR.PHP.NET):
=====================================
...

All that is required to complete the test environment is the installation of PEAR itself:

# pear -c ~/pear-test/.pearrc install -o PEAR

WARNING: channel "pear.php.net" has updated its protocols, use "channel-update pear.php.net" to update
Did not download optional dependencies: pear/XML_RPC, use --alldeps to download automatically
pear/PEAR can optionally use package "pear/XML_RPC" (version >= 1.4.0)
downloading PEAR-1.7.2.tgz ...
Starting to download PEAR-1.7.2.tgz (302,744 bytes)
..........................done: 302,744 bytes
downloading Archive_Tar-1.3.2.tgz ...
Starting to download Archive_Tar-1.3.2.tgz (17,150 bytes)
...done: 17,150 bytes
downloading Structures_Graph-1.0.2.tgz ...
Starting to download Structures_Graph-1.0.2.tgz (30,947 bytes)
...done: 30,947 bytes
downloading Console_Getopt-1.2.3.tgz ...
Starting to download Console_Getopt-1.2.3.tgz (4,011 bytes)
...done: 4,011 bytes
install ok: channel://pear.php.net/Archive_Tar-1.3.2
install ok: channel://pear.php.net/Structures_Graph-1.0.2
install ok: channel://pear.php.net/Console_Getopt-1.2.3
install ok: channel://pear.php.net/PEAR-1.7.2
PEAR: Optional feature webinstaller available (PEAR's web-based installer)
PEAR: Optional feature gtkinstaller available (PEAR's PHP-GTK-based installer)
PEAR: Optional feature gtk2installer available (PEAR's PHP-GTK2-based installer)
PEAR: To install optional features use "pear install pear/PEAR#featurename"

A real example

As I am currently working on releasing the Kolab modules in Horde as PEAR packages I'll provide this process as an example.

In order to wrap a PEAR package the appropriate channel needs to be known for pear. For horde packages this is pear.horde.org. This channel needs to be discovered first:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc channel-discover pear.horde.org
Adding Channel "pear.horde.org" succeeded
Discovery of channel "pear.horde.org" succeeded

The initial package to be released will be Kolab_Format. PEAR packaging happens with

# cd Kolab_Format
# ~/pear-test/pear/pear -c ~/pear-test/.pearrc package package.xml

Analyzing lib/Horde/Kolab/Format/XML/contact.php
Analyzing lib/Horde/Kolab/Format/XML/distributionlist.php
Analyzing lib/Horde/Kolab/Format/XML/event.php
Analyzing lib/Horde/Kolab/Format/XML/hprefs.php
Analyzing lib/Horde/Kolab/Format/XML/note.php
Analyzing lib/Horde/Kolab/Format/XML/task.php
Analyzing lib/Horde/Kolab/Format/Date.php
Analyzing lib/Horde/Kolab/Format/XML.php
Analyzing lib/Horde/Kolab/Format.php
Package Horde_Kolab_Format-0.9.0.tgz done
Tag the released code with `pear cvstag package.xml'
(or set the CVS tag RELEASE_0_9_0 by hand)

As most Horde PEAR packages have not yet been marked stable PEAR will still refuse to install the new package:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install Horde_Kolab_Format-0.9.0.tgz 

Failed to download horde/Horde_DOM within preferred state "stable", latest release is version 0.1.0, stability "alpha", use "channel://pear.horde.org/Horde_DOM-0.1.0" to install
Failed to download horde/Horde_NLS within preferred state "stable", latest release is version 0.0.2, stability "alpha", use "channel://pear.horde.org/Horde_NLS-0.0.2" to install
Failed to download horde/Horde_Util within preferred state "stable", latest release is version 0.0.2, stability "alpha", use "channel://pear.horde.org/Horde_Util-0.0.2" to install
Did not download optional dependencies: horde/Horde_Prefs, use --alldeps to download automatically
horde/Horde_Kolab_Format requires package "horde/Horde_DOM" (version >= 0.1.0)
horde/Horde_Kolab_Format requires package "horde/Horde_NLS"
horde/Horde_Kolab_Format requires package "horde/Horde_Util"
horde/Horde_Kolab_Format can optionally use package "horde/Horde_Prefs"
No valid packages found
install failed

The required packages must be installed manually first:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install channel://pear.horde.org/Horde_DOM-0.1.0

downloading Horde_DOM-0.1.0.tgz ...
Starting to download Horde_DOM-0.1.0.tgz (4,256 bytes)
.....done: 4,256 bytes
install ok: channel://pear.horde.org/Horde_DOM-0.1.0

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install channel://pear.horde.org/Horde_Util-0.0.2

Did not download optional dependencies: horde/Horde_Browser, use --alldeps to download automatically
horde/Horde_Util can optionally use package "horde/Horde_Browser"
downloading Horde_Util-0.0.2.tgz ...
Starting to download Horde_Util-0.0.2.tgz (16,603 bytes)
......done: 16,603 bytes
install ok: channel://pear.horde.org/Horde_Util-0.0.2

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install channel://pear.horde.org/Horde_NLS-0.0.2

downloading Horde_NLS-0.0.2.tgz ...
Starting to download Horde_NLS-0.0.2.tgz (75,779 bytes)
.................done: 75,779 bytes
install ok: channel://pear.horde.org/Horde_NLS-0.0.2

This time installation should suceed:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install Horde_Kolab_Format-0.9.0.tgz 

Did not download optional dependencies: horde/Horde_Prefs, use --alldeps to download automatically
horde/Horde_Kolab_Format can optionally use package "horde/Horde_Prefs"
install ok: channel://pear.horde.org/Horde_Kolab_Format-0.9.0

All that is missing now is to tell PEAR that it should provide PHP only with one include directory: the one we just setup. Ensure that you replace USER with the name of the current user.

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc config-set php_bin "`~/pear-test/pear/pear -c ~/pear-test/.pearrc config-get php_bin` -d include_path=/home/USER/pear-test/pear/php"

After installation succeeded the unit tests should be run in order to validate the dependencies. If the tests are based on PHPUnit then this tool will have to be installed first:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc channel-discover pear.phpunit.de

Adding Channel "pear.phpunit.de" succeeded
Discovery of channel "pear.phpunit.de" succeeded

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc install phpunit/PHPUnit

And now we can finally run the last check before releasing the PEAR package:

# cd ~/pear-test/pear/tests/Horde_Kolab_Format/Horde/Kolab/Format/
# ~/pear-test/pear/pear -c ~/pear-test/.pearrc run-tests -u 

Python egg fun

Now that I'm back working on some of my Python packages it was time to look at Python package management again. So far I just used the embedded distutils package but apparently "setuptools" is the thing that starts to be used widespread. So I looked at this in more detail and will summarize some notes here.

"setuptools" does actually not deliver too much fancy new functionality. The main benefit lies in the area of dependencies and the creation of distributable packages.

I basically used something like this as setup.py when I only used distutils:

from distutils.core import setup

PACAKGES = ['libpardus',        
            'libpardus.configs',
            'libpardus.utils',
            'libpardus.web']

import sys
sys.path.insert(0, './')             
from libpardus.version import VERSION

setup(name          = 'libpardus',
      version       = VERSION,                
      description   = 'p@rdus Python Library',
      author        = 'Gunnar Wrobel',
      author_email  = 'p@rdus.de',                       
      url           = 'http://libpardus.sourceforge.net',
      packages      = PACKAGES,
      license       = 'GPL',
      **extra
      )

Now, with setuptools things do not become much more complicated:

try:                                           
    from setuptools import setup, find_packages
except ImportError, e:              
    from distutils.core import setup
    extra = {}              
    PACAKGES = ['libpardus',        
                'libpardus.configs',
                'libpardus.utils',
                'libpardus.web']
else:            
    extra = dict(           
        install_requires = [
            'setuptools',
            'web.py',        
            'zope.interface',
            ],            
        extras_require = {
            'WEB': ['web.py'],
            }
        )                     
    PACKAGES = find_packages()

import sys
sys.path.insert(0, './')
from libpardus.version import VERSION
                                  
setup(name          = 'libpardus',
      version       = VERSION,                
      description   = 'p@rdus Python Library',
      author        = 'Gunnar Wrobel',
      author_email  = 'p@rdus.de',                       
      url           = 'http://libpardus.sourceforge.net',
      packages      = PACKAGES,
      license       = 'GPL',
      **extra
      )

I'm mainly using find_packages() to generate the package list and add some packages as basic requirements. In order to make this also work on "non-setuptools" system the whole is embedded in a try: ... except: ... statement.

The whole thing is not yet complete since I don't know every detail of "setuptools" yet, but the main point here is that things are not too different if you decide to upgrade from "distutils".

I also created a setup.cfg-file to make the creation of snapshot and release packages easier:

[egg_info]
tag_build = .dev
tag_svn_revision = 1                           

[aliases]
release  = egg_info -RDb ''
relpatch = egg_info -db ''  

Running python setup.py bdist_egg will now create an egg that I can use for testing purposes. It gets the current subversion revision attached to the file name (e.g. libpardus-0.9.2.dev_r38-py2.4.egg. Because of the alias definition

release  = egg_info -RDb ''

the command python setup.py release sdist will create a release source package (e.g. libpardus-0.9.2.tar.gz). The relpatch command is for releasing a patch level package after the main release.

But during development it is really nice to produce these testing eggs without the need to build/install the tools every time.

"setuptools" also makes it easy to handle foreign eggs. This way you can build whole test applications without modifying your site-wide python library. This can be done using the easy_install tool:

easy_install -zmaxd lib/ web.py

This way you would place web.py as a packaged egg within the lib directory.

Within a python library you can the either directly include the full egg filename in the sys.patch:

sys.path.insert(0, 'lib/web.py-0.23-py2.4.egg')

Using setuptools you can also make this:

from pkg_resources import require
require("web.py")

The details can be found on the setuptools homepage.

I'll probably continue this once I learn how to upload python packages to PyPI, the python package index.