Friday, February 29, 2008

I really don't want to force you...

Skin the onion

"... no pressure whatsoever ...". Isn't that how it always starts?

Initially I intended to head over to the Chemnitzer Linuxtage for a chat here and there and maybe a beer or two. Just for fun.

But when I realized that there are still some possibilities for giving a short overview on a Gentoo-specific topic, I told Tobias (dertobi123) that I could give a talk on the Kolab2/Gentoo project. That is definitely still within the fun area of things.

What I did not expect was that I would get called with the request to jump in for one of the speakers in on of the main tracks who got sick ... no pressure whatsoever ... sure. Today is going to be hectic...

Thursday, February 28, 2008

Ups... time for recovery

051211_test_d50_2.jpg

Hrm,... no clue how I managed to break my Gallery installation. I probably played around too much with gallery2flickr. Though maybe I killed it when bumping and testing the ebuild last time.

Anyhow it must have been a while ago so I actually had to ask dar to go into recovery mode. And since this happens very infrequently I always forgot how to select a sub-path of the whole archive dar creates.

Here is my reminder:

dar --crypto-block 20480 --key MYKEY -x 01-18-2008_0358 -g localhost/htdocs/gallery

Hello again, my dear Gallery... Backups are indeed a nice thing.

Wednesday, February 27, 2008

Multiply your knowledge

Gentoo Linux

I've waited since Friday and it finally arrived today. Another German Gentoo book is available now!

It concentrates on the experienced Linux user and tries to achieve two things:

  • Get you running on Gentoo if you never used it before
  • Highlight the central Gentoo tools and provide a reference for their usage

For new users it should be possible to grab a laptop, insert the DVD and run through large parts of the book without ever connecting to the net. Thus the first steps with Gentoo are hopefully pleasant.

Even the early chapters feature larger sections that provide in-depth knowledge for the more experienced Gentoo users. All tools, options, variables and concepts are referenced in the thirty index pages of the book. So it should be a good companion while working with Gentoo.

If you need more details you can check the table of contents or even read the chapter on writing ebuilds.

One of the later chapters - "Extending Gentoo" - can be considered the main origin of the whole story. It talks about overlays.gentoo.org and layman.

About two years ago I sat down on a weekend to code the basis for layman in order to make the use of overlays.gentoo.org as easy as possible. I certainly didn't expect this to have any significant effect. After all layman always was - and still is - a rather trivial script.

But since I liked the whole idea about overlays I decided to write a small article about this concept as well as how layman fits into that. This got published in the German Linux magazine and had one unexpected result: I got an email from OpenSourcePress asking whether I'd like to write a whole book about Gentoo. At that time the answer was "yes".

Would it be "no" today? I'm not certain. I have to say that I hated the three weeks of pure text editing in the final phase of the project. It reminded me far too much of my PhD thesis. Yes, I like writing stuff: emails, wiki pages, blog entries, source documentation, ... you name it. Small stuff. Epic texts turn out to be much harder.

What definitely made it bearable in this case were the systems provided by OpenSourcePress: They give you a subversion repository and the whole text is done in Latex. They also work with Emacs on their end which happens to be my favorite editor too.

And one thing about their subversion repository is really great: You commit crudely written techno-babble on your side and a few revisions later it comes back in well readable German. This is what I definitely liked most about the whole project: Getting rewritten to readable language. Big kudos to the team of editors which really does great job.

So would I do it again? Well, I don't have to decide on that anymore. The basis is there so all there will be are further revisions. And that will hopefully be easier than the first version.

But for now I'll keep the book closed and continue coding...

Horde: Synching with HEAD

Feels good to be finally back working on Horde. Last week saw the generation of Kolab patches for Horde-3.2-RC2 and this week I finally synchronized my work environment with Horde CVS. The first Kolab commit to Horde CVS went in yesterday. It felt like the last one was ages ago.

Anyhow there are more commits ahead. Now that most parts of Horde work with Kolab the time for the second round of coding is approaching fast: restructuring and optimization. There is still a lot that needs to be done and I'm desperately waiting for Horde 4 to finally restructure the whole Kolab module and get it on a hopefully sane path for the future.

Thursday, February 14, 2008

Python egg fun

Now that I'm back working on some of my Python packages it was time to look at Python package management again. So far I just used the embedded distutils package but apparently "setuptools" is the thing that starts to be used widespread. So I looked at this in more detail and will summarize some notes here.

"setuptools" does actually not deliver too much fancy new functionality. The main benefit lies in the area of dependencies and the creation of distributable packages.

I basically used something like this as setup.py when I only used distutils:

from distutils.core import setup

PACAKGES = ['libpardus',        
            'libpardus.configs',
            'libpardus.utils',
            'libpardus.web']

import sys
sys.path.insert(0, './')             
from libpardus.version import VERSION

setup(name          = 'libpardus',
      version       = VERSION,                
      description   = 'p@rdus Python Library',
      author        = 'Gunnar Wrobel',
      author_email  = 'p@rdus.de',                       
      url           = 'http://libpardus.sourceforge.net',
      packages      = PACKAGES,
      license       = 'GPL',
      **extra
      )

Now, with setuptools things do not become much more complicated:

try:                                           
    from setuptools import setup, find_packages
except ImportError, e:              
    from distutils.core import setup
    extra = {}              
    PACAKGES = ['libpardus',        
                'libpardus.configs',
                'libpardus.utils',
                'libpardus.web']
else:            
    extra = dict(           
        install_requires = [
            'setuptools',
            'web.py',        
            'zope.interface',
            ],            
        extras_require = {
            'WEB': ['web.py'],
            }
        )                     
    PACKAGES = find_packages()

import sys
sys.path.insert(0, './')
from libpardus.version import VERSION
                                  
setup(name          = 'libpardus',
      version       = VERSION,                
      description   = 'p@rdus Python Library',
      author        = 'Gunnar Wrobel',
      author_email  = 'p@rdus.de',                       
      url           = 'http://libpardus.sourceforge.net',
      packages      = PACKAGES,
      license       = 'GPL',
      **extra
      )

I'm mainly using find_packages() to generate the package list and add some packages as basic requirements. In order to make this also work on "non-setuptools" system the whole is embedded in a try: ... except: ... statement.

The whole thing is not yet complete since I don't know every detail of "setuptools" yet, but the main point here is that things are not too different if you decide to upgrade from "distutils".

I also created a setup.cfg-file to make the creation of snapshot and release packages easier:

[egg_info]
tag_build = .dev
tag_svn_revision = 1                           

[aliases]
release  = egg_info -RDb ''
relpatch = egg_info -db ''  

Running python setup.py bdist_egg will now create an egg that I can use for testing purposes. It gets the current subversion revision attached to the file name (e.g. libpardus-0.9.2.dev_r38-py2.4.egg. Because of the alias definition

release  = egg_info -RDb ''

the command python setup.py release sdist will create a release source package (e.g. libpardus-0.9.2.tar.gz). The relpatch command is for releasing a patch level package after the main release.

But during development it is really nice to produce these testing eggs without the need to build/install the tools every time.

"setuptools" also makes it easy to handle foreign eggs. This way you can build whole test applications without modifying your site-wide python library. This can be done using the easy_install tool:

easy_install -zmaxd lib/ web.py

This way you would place web.py as a packaged egg within the lib directory.

Within a python library you can the either directly include the full egg filename in the sys.patch:

sys.path.insert(0, 'lib/web.py-0.23-py2.4.egg')

Using setuptools you can also make this:

from pkg_resources import require
require("web.py")

The details can be found on the setuptools homepage.

I'll probably continue this once I learn how to upload python packages to PyPI, the python package index.

Friday, February 08, 2008

SVN backup using SVN::Mirror

I do have a number of subversion repositories that I like to backup once every hour. The tool I'm using for that is the perl package SVN::Mirror. I didn't document the usage of this tool for myself yet and I feel it is time to correct that.

The tool itself is available on CPAN and on Gentoo you can install it via Portage:

gentoo # emerge dev-perl/SVN-Mirror

This delivers the tool /usr/bin/svm that holds all the functionality to replicate Subversion repositories.

I always had strange locking issues with this perl script so I added a small section for the occasional unlocking:

--- /usr/bin/svm        2007-06-21 15:17:55.000000000 +0200
+++ /usr/bin/svm-expanded       2008-01-07 14:59:54.000000000 +0100
@@ -82,6 +82,17 @@
        m/connection timed out/;
 }
 
+sub unlock {
+    my $path = shift;
+    my $what = shift;
+    my $pool = SVN::Pool->new_default;
+    my $m = SVN::Mirror->new(target_path => $path, target => $repospath,
+                            pool => $pool, auth => $auth,
+                            get_source => 1);
+
+    $m->unlock($what);
+
+}
 
 sub sync {
     my $path = shift;

To backup a repository you will first have to initialize a new repository using svm:

gentoo # export SVMREPOS=/var/svn/backup/libpardus
gentoo # svm init / https://libpardus.svn.sourceforge.net/svnroot/libpardus

Updating the repository to the newest state is not much more complex:

gentoo # export SVMREPOS=/var/svn/backup/libpardus
gentoo # svm sync /

If you use the unlocking patch provided above this makes:

gentoo # export SVMREPOS=/var/svn/backup/libpardus
gentoo # svm unlock / force
gentoo # svm sync /

I package these few lines into a small shell script and add it to cron so I get a regular update on all the repositories. Just in case Sourceforge ever dies (which I hope it does not).

libpardus: Getting another python project back on track

libpardus is another one of my python projects that I hope to get revived soon. It provides a collection of different utilities that I require for the other python tools I wrote. Right now I'm just busy getting the project basics updated and this is a first blog post for the tool.