A sneak peek of the new Horde 5 user interface

To get an idea on how Horde 5 will look like: click the link or the image of this post.

Why does Horde 5 get a face lift? Simply because the current UI was mentioned often enough as an issue by many Horde users. And since the Horde 4 release had a very technical focus the switch from Horde 3 to Horde 4 last year did not help - it even degraded consistency between the applications. At the same time the competition does not sleep and there are more and more large installations that offer their user base two different webmails - one of them being Horde for the power users that feel they need a lot of features but that care less about the UI. Time to get our act together.

So what is the primary target of the redesign? First and foremost we want to unify the main user interfaces. At the moment we have the static application views, the dynamic webmailer, and the dynamic calender as the core parts. All looked somewhat different. These are the elements that we wish to give a consistent look. The special views such as the minimal webmailer or the smartphone UI will remain untouched.

We also hope the new design looks somewhat fresher than what we had before but please keep in mind that we are oriented towards people that use the interface for their daily work. We do not aim for a UI that looks like the last hype. It should be functional instead.

The Horde LLC has been the driving factor behind the redesign. At least financially. A subset of the Horde core developers started the LLC a while back as a contact point for people that want to pay for Horde support or feature development. A part of the money that such contracts pay goes to the developers dealing with the particular customer request. But another part of the money remains within the LLC. The idea is to use the latter to drive features that we consider to be important for Horde and its community. The redesign is the first project that has been financed this way. The Horde team tried finding designers interested in contributing to an Open Source project several times before. This was unsuccessful however and paying a designer for the work remained the only reasonable alternative.

We contracted No agency for the design. After several rounds of communication between them and all Horde developers we managed to end up with the draft displayed above. This has been converted to HTML and CSS this week and will be hammered into code during the next week by Jan Schneider. We do hope to present you with an alpha of Horde 5 - including the draft of the new design - on the 22nd May of 2012.

Feedback and comments - as usual - are welcome!

The library section of the Horde website supports component documentation now.

Our new libraries section has been started a while ago to further push the PHP components we offer. Now this section supports publishing the component documentation as well. You can take a look at the documentation of the Cli_Modular package for example. In fact: it is pretty much the only package that fully uses the new system already. So there is still work ahead.

If you use any of the components: We are grateful if you start writing a bit of documenation or describe some examples on how to successfully use the package. It will help us and others tremendously. Thanks!

Let me try to explain how the system is currently intended to work - feedback and critical comments of course welcome:

1. When creating new documentation for one of the Horde components you start a new section on the developer documentation of our wiki. See the Horde_Cli_Modular link below Library components for example. Please note: It is not mandatory to write the documentation in the wiki. You can have static documentation files within a component as well. This is just meant as an option for those situations where it makes sense to work on documentation files together with people that do not have direct git commit access. Hopefully there will be many component consumers interested in updating and fixing the documentation.
2. How you structure that new section is up to you. For a simple package you might wish to keep all documentation in a single file but for the more complex one it might make sense to have several files. In that case the new section page should probably just be a link list to the various wiki pages that document the component. In case of Horde_Cli_Modular I used a single page.
3. Once the documentation has been written in the wiki format it is time to download it into the doc directory of the component. The horde-components helper is what you should use for that operation. The tool expects to find a DOCS_ORIGIN file within the doc or docs (for the applications) directory of the component. This file must conform to the reStructuredText format and map remote URLs to local file paths relative to the component root. The DOCS_ORIGIN from Cli_Modular links the wiki page exported as reStructuredText to the path doc/Horde/Cli/Modular/README. The links can of course point anywhere so you are in principle free to use any remote source. But you should ensure the page provides readable reStructuredText.
4. Now you can run horde-components fetchdocs and it should fetch the URLs into the respective local files. You can then run horde-components update to refresh the file list in the package.xml file if the documentation files were not included before.
5. Once a component was released we can now run something like this: horde-components Horde_Cli_Modular webdocs -D ~/git/horde-web/ --html-generator=git/horde-support/maintainer-tools/docutils/html.py --allow-remote in order to update the documentation on the website. Right now this does not happen automatically during a release but I plan to add this soon.

There are of course still a few problems with this approach:

• The reStructuredText exporter of our wiki engine "Wicked" is not yet complete. You may experience problems with the export if you use constructs it does not know yet. Please ping me if you experience problems with the export.
• The wiki syntax and the syntax required by reStructuredText is not 100% aligned. You can write a wiki page that looks just fine but leads to errors or formatting problems in the reStructuredText export. After creating / updating a new wiki documentation page it make sense to run the horde-components fetchdocs/horde-components webdocs sequence to check for problems.
• We still have some components with documentation files not in reStructuredText format. These will result in non-existing links on our website for now but I plan to fix the few faulty packages soon.

Horde_Push now supports sending to Blogger.com

Horde_Push supports sending to blogger.com now.

The Horde_Push library is a tool to send content elements to various external services. Such as Twitter, Facebook, Blogger and others. Right now it only all
ows sending Tweets, publishing entries on blogger.com and sending e-mails.

The idea is to allow publishing content you curate on a Horde installation to the social networks out there. At the moment the code is not yet there as the
integration into the base horde package is still lacking. Right now there is only a small command line helper that I use for testing.

Mixing stable Horde components with snapshots

One thing I completely love about Horde 4 being a component framework is the ability to have stable installations into which I can inject experimental snapshots of packages further developed, patched or hacked in some other way. Just an overview of one such installation:

INSTALLED PACKAGES, CHANNEL PEAR.HORDE.ORG:
===========================================
PACKAGE                   VERSION              STATE
Horde_ActiveSync          1.0.0                stable
Horde_Alarm               1.0.1                stable
Horde_Argv                1.0.1                stable
Horde_Auth                1.0.3                stable
Horde_Autoloader          1.0.0                stable
Horde_Browser             1.0.0                stable
Horde_Cache               1.0.3                stable
Horde_Cli                 1.0.0                stable
Horde_Compress            1.0.1                stable
Horde_Constraint          1.0.0                stable
Horde_Controller          1.0.0                stable
Horde_Core                1.1.2dev201105300702 stable
Horde_Crypt               1.0.2                stable
Horde_Data                1.0.0                stable
Horde_DataTree            1.0.0                stable
Horde_Date                1.0.1                stable
Horde_Date_Parser         1.0.0                stable
Horde_Db                  1.0.1                stable
Horde_Editor              1.0.1                stable
Horde_Exception           1.0.1                stable
Horde_Feed                1.0.1dev201106011224 stable
Horde_Form                1.0.1                stable
Horde_Group               1.0.0                stable
...

How do you get these snapshots? Ensure you have the horde-component helper set up. Then enter the package you wish to snapshot (e.g. Horde_Core) and run:

horde-components snapshot

The snapshot package will be assembled in this directory, can be uploaded to your webserver and installed using:

pear upgrade --offline --force Horde_Core-1.1.2dev201105300702.tgz

What happens if you patched and deployed your package, sent the patch upstream, it gets accepted and a new package released?

pear upgrade horde/Horde_Core

Magic!

Horde release cycle

• Horde 1.0.0: 1999
• Horde 2.0: 2002
• Horde 3.0: 2004

... and finally Horde4 in 2011.

The gap between 3.0 and 4.0 has been too large. The Horde team tries to keep backward compatibility within a major version. Probably nobody tried running a recent Kronolith from 2010 on Horde 3.0 from 2004. There'd probably be some issues and limitations but they should be minor.

While this kind of long term support may be useful for some edge cases it can also impede development progress. A good example are the Kolab drivers in the currently released version: Kolab support within Horde started to improve in 2006. At that time Horde3 was already the active branch and there was some very basic Kolab functionality in there. So the new code had to keep those interfaces stable. Which basically meant twisting and bending it into code that would do just that but otherwise be really problematic. That code is still a reality in the current stable release from Horde3.

This and similar problems did not pass unnoticed though and changes to the release cycle have been discussed internally for a while already. With the release of Horde4 approaching it now makes sense to discuss such changes with the Horde community. The envisioned target is a time based release cycle.

The discussion started today on the Horde development mailing list. Feel free to listen in or to add your own comments if you want to influence the direction of our future release mode.

A new old PHP framework

With Horde4 the old "framework" block of code contained in the base Horde application will be split into its single component parts. This should help to underline the modular framework aspect of Horde itself.

The Kolab Server is one of the few systems that tried to use part of the Horde software stack even with Horde3. Both the free/busy system and the Resource management do not require more than a few packages from the Horde core framework.

With Horde3 this approach has always been somewhat awkward. The Horde release process made no use of the fact that the Horde framework was split into modules. With the Kolab Server we had to work around this limitation which resulted in the split packaging layout which will be available with Kolab-Server-2.3. But this is hand made on the Kolab side.

For Horde 4 p@rdus is pushing the use of PEAR with a specialized "component helper". The idea is to facilitate the handling of many small Horde components - each a PEAR package.

The tool currently allows automatic updates to Horde component manifests. It packages development snapshots. Builds continuous integration configuration for a component. It creates packaging specs for a distribution - and lists package dependencies.

Last but not least: It install components. And it does that primarily based on the code repository.

This is an essential feature for the continuous integration setup as all testing of the newest code will happen based on installed components. As a result the packaging will automatically be part of the quality control. Both the developers as well as the packagers will benefit from that.

Once the complete components setup is in place it will be very easy to derive a set of packages from it. Whether these packages are targeted for OpenPKG (Kolab Server 2.2.4 or 2.3) or some later native packaging will not really matter. p@rdus is of course actively working on such packaging and you can watch for commits on the horde4 branch of the Kolab server mercurial repository.

Python egg fun

Now that I'm back working on some of my Python packages it was time to look at Python package management again. So far I just used the embedded distutils package but apparently "setuptools" is the thing that starts to be used widespread. So I looked at this in more detail and will summarize some notes here.

"setuptools" does actually not deliver too much fancy new functionality. The main benefit lies in the area of dependencies and the creation of distributable packages.

I basically used something like this as setup.py when I only used distutils:

from distutils.core import setup

PACAKGES = ['libpardus',        
            'libpardus.configs',
            'libpardus.utils',
            'libpardus.web']

import sys
sys.path.insert(0, './')             
from libpardus.version import VERSION

setup(name          = 'libpardus',
      version       = VERSION,                
      description   = 'p@rdus Python Library',
      author        = 'Gunnar Wrobel',
      author_email  = 'p@rdus.de',                       
      url           = 'http://libpardus.sourceforge.net',
      packages      = PACKAGES,
      license       = 'GPL',
      **extra
      )

Now, with setuptools things do not become much more complicated:

try:                                           
    from setuptools import setup, find_packages
except ImportError, e:              
    from distutils.core import setup
    extra = {}              
    PACAKGES = ['libpardus',        
                'libpardus.configs',
                'libpardus.utils',
                'libpardus.web']
else:            
    extra = dict(           
        install_requires = [
            'setuptools',
            'web.py',        
            'zope.interface',
            ],            
        extras_require = {
            'WEB': ['web.py'],
            }
        )                     
    PACKAGES = find_packages()

import sys
sys.path.insert(0, './')
from libpardus.version import VERSION
                                  
setup(name          = 'libpardus',
      version       = VERSION,                
      description   = 'p@rdus Python Library',
      author        = 'Gunnar Wrobel',
      author_email  = 'p@rdus.de',                       
      url           = 'http://libpardus.sourceforge.net',
      packages      = PACKAGES,
      license       = 'GPL',
      **extra
      )

I'm mainly using find_packages() to generate the package list and add some packages as basic requirements. In order to make this also work on "non-setuptools" system the whole is embedded in a try: ... except: ... statement.

The whole thing is not yet complete since I don't know every detail of "setuptools" yet, but the main point here is that things are not too different if you decide to upgrade from "distutils".

I also created a setup.cfg-file to make the creation of snapshot and release packages easier:

[egg_info]
tag_build = .dev
tag_svn_revision = 1                           

[aliases]
release  = egg_info -RDb ''
relpatch = egg_info -db ''  

Running python setup.py bdist_egg will now create an egg that I can use for testing purposes. It gets the current subversion revision attached to the file name (e.g. libpardus-0.9.2.dev_r38-py2.4.egg. Because of the alias definition

release  = egg_info -RDb ''

the command python setup.py release sdist will create a release source package (e.g. libpardus-0.9.2.tar.gz). The relpatch command is for releasing a patch level package after the main release.

But during development it is really nice to produce these testing eggs without the need to build/install the tools every time.

"setuptools" also makes it easy to handle foreign eggs. This way you can build whole test applications without modifying your site-wide python library. This can be done using the easy_install tool:

easy_install -zmaxd lib/ web.py

This way you would place web.py as a packaged egg within the lib directory.

Within a python library you can the either directly include the full egg filename in the sys.patch:

sys.path.insert(0, 'lib/web.py-0.23-py2.4.egg')

Using setuptools you can also make this:

from pkg_resources import require
require("web.py")

The details can be found on the setuptools homepage.

I'll probably continue this once I learn how to upload python packages to PyPI, the python package index.

libpardus: Getting another python project back on track

libpardus is another one of my python projects that I hope to get revived soon. It provides a collection of different utilities that I require for the other python tools I wrote. Right now I'm just busy getting the project basics updated and this is a first blog post for the tool.

A mercurial patch update session

We discussed patch management recently on the Kolab development mailing list and today it was time to upgrade to php-5.2.5. I had to run a patch update cycle again. Some time ago Thomas recommended using mercurial for the patch management. And I must admit that this completely changed the way I deal with code patches and it made my life a lot easier. Let me describe how I currently deal with patches when upstream delivers a new version.

Setup

Some time ago I started the patch management on php by downloading the package ...

> wget ftp://php.net/distributions/php-5.2.4.tar.bz2

... and unpacking it:

> tar xfj php-5.2.4.tar.bz2

Now I turned this version into a mercurial repository and added all files to version control:

> cd php-5.2.4
> hg init
> hg commit --addremove -m "php-5.2.4"
> cd ..

Since I needed to patch php I derived (cloned) a second repository from the original one:

> hg clone php-5.2.4 php-PATCHED

In order to add patches to a repository you have to activate the queue extension within mercurial:

> cat ~/.hgrc
[ui]
username = Gunnar Wrobel
[extensions]
hgext.mq =

That allows to add patches on top of a version controlled repository. This had to be activated on the php-PATCHED repository:

> cd php-PATCHED
> hg qinit -c

The -c option makes the new patch directory that got created by qinit under php-PATCHES/.hg/patches a version controlled directory. This is not strictly necessary but I find it convenient to version control the patches, too. For the Kolab specific patching one patch had to be added to php-PATCHED:

> hg qnew KOLAB_Annotation.patch
> patch -p1 < ~/Kolab-php.patch
> hg qrefresh -m "Provides get/set ANNOTATIONS support to PHP. [Version: 5.2.4]"

qrefresh compiled the changes in php-PATCHED into the KOLAB_Annotation.patch within php-PATCHES/.hg/patches. Now I was able to grab the patch from that location and apply it to the modified packages on both the OpenPKG platform and Gentoo

Update

The whole setup would be useless if we wouldn't need to upgrade our patches from time to time. The following describes the cycle that I perform once upstream released a new version. Fetch the new version:

> wget ftp://php.net/distributions/php-5.2.5.tar.bz2

Now we clone the original version ...

> hg clone php-5.2.4 php-5.2.5

... and replace it with the new one:

> cd php-5.2.5
> hg locate -0 | xargs -0 rm
> cd ..
> tar xfj php-5.2.5.tar.bz2
> cd php-5.2.5
> hg commit --addremove -m "php-5.2.5"
> cd ..

Now the original repository holds the new version and we need to update our patched version to this. For this we first need to remove all applied patches since we don't know if they apply any longer:

> cd php-PATCHED
> hg qpop -a

qpop -a removes all currently applied patches. They are of course still present in our php-PATCHED/.hg/patches directory. But they are not applied. This allows to cleanly update our base php version now:

> hg pull ../php-5.2.5
> hg update

Pulling and updating brings the php-PATCHED directory to the php-5.2.5 version. Now we can push the stack of patches again and fix any that do not cleanly apply:

> hg qpush

In case our patch fails we need to modify the source until everything is working fine again. Often the changes will be minimal. After the updates the patch has to be refreshed:

> hg qrefresh -m "Provides get/set ANNOTATIONS support to PHP. [Version: 5.2.5]"
> hg qci -m "PHP patch for 5.2.5"

The final command runs hg commit within the php-PATCHED/.hg/patches directory. For a single patch the whole procedure might be a little bit too complicated. It is rather clean though. But the whole story gets really efficient once you have about twenty patches on an application. In that case the mercurial queue extensions comes in really handy.