Tuesday, August 07, 2012

Modular git with "git subtree"

One thing I always disliked about the way we organized our Horde repository was the fact that we have all library modules and applications lumped together in a single git repository. Of course there are some good reasons for that type of monolithic repo. But for someone just interested in our (really powerful) IMAP library this is a drawback: The library is hidden somewhere between the other libraries and if you want to work on it you will nevertheless have to clone the whole repository. And there are other situations in which small, module specific repositories would make sense. So far I wasn't aware of a solution that would allow for a reasonable compromise.

Originally I only knew that git submodule would allow including additional git repositories into a master repository. This approach has some drawbacks though. We could construct the current horde repository out of a bunch of submodules. But the work flow within this master repository would be significantly more cumbersome as git submodule interferes with the default way of working with git.

git subtree to the rescue!

git subtree however seems to allow for the perfect solution: Separate subrepositories can co-exist with the monolithic master repository. And any commits to either of them can be exchanged between them. The stream of commits to the monolithic master can even be transmitted automatically to the splitted repositories. None of these steps seem to introduce any additional overhead to any of these repositories.

Installing "git subtree"

The subtree command has been added to the git-1.7.11 release. But as many distributions do not yet offer this variant you can install the tool in a more hackish way if desired:

cd /usr/lib/git-core/
wget https://raw.github.com/apenwarr/git-subtree/master/git-subtree.sh
mv git-subtree.sh git-subtree
chmod 755 git-subtree

Replacing a "submodule" with a "subtree"

A while ago I pulled the Jenkins installation procedures into our horde-support repository using git submodule. In order to give git subtree a first test run I replaced the Jenkins submodule by the subtree approach. The first step had to be the removal of the old submodule:

git rm .gitmodules ci/jenkins
git commit -m "Remove the jenkins installation procedures as a submodule. Prepares for replacement by 'git subtree'"

Now I imported the repository previously registered via git submodule using git subtree:

git subtree add --prefix=ci/jenkins --squash https://github.com/wrobel/jenkins-install.git master

This pulled the external repository into the current horde-support repository at prefix "ci/jenkins" and squashed all commits of the imported repository into a single commit. The imported code is now an equivalent citizen to the rest of the code in the repository - none of the standard git work flows are affected in any way.

Of course the interesting question is whether updates to this imported code can be merged back into the original repository. I commited a small change within the imported code:

git commit -m "Update to jenkins-1.475" ci/jenkins/jenkins.mk

This change can indeed now be splitted into the subtree again and exported to the original archive:

git subtree split --prefix=ci/jenkins --annotate="(horde-support) " d73edc4878c8.. --branch ci-jenkins

What happens here is that git subtree splits the path specified with the prefix option into a separate branch named ci-jenkins. It will prefix any commit transported into this branch with (horde-support) to indicate the origin of the commit. Usually the branch range given here (d73edc4878c8..) is unnecessary for the operation. But the code within ci/jenkins had been included as submodule before commit d73edc4878c8. This part of the history should not be imported into the splitted branch.

After the splitting operation created the new ci-jenkins branch in my repository it should be equivalent to our original, imported repository. Thus I was able to push back to it:

git push git@github.com:wrobel/jenkins-install.git ci-jenkins:master

Using "git subtree" for the horde repository

Can the subtree approach be used to have both a monolithic horde repository as well as the small modular repositories at the same time? This would be the best of both worlds: While we develop in the monolithic horde repositories we allow other developers to also watch and patch single modules. If the commits from the monolithic repo can be transferred to the modular repos on a regular basis while we can also import patches the other way around without blowing up any of the associated git repos: I'd be really happy.

I admit that I didn't test the subtree approach large scale yet - but everything I have tested so far indicates that the situation detailed above can indeed be achieved and automated.

In order to automate the splitting of the monolithic repository into different modules I would use an intermediate git repository that handles the splitting within a post-receive hook. Any pushing to a branch of this repository would then update the same branches in the various splitted repositories. The core of the splitting procedure in the post-receive hook I established looks like this so far:

git config --bool core.bare false
git checkout $short_refname
git reset --hard HEAD
if [ -z "`git branch | grep subtrees/$short_refname`" ]; then
    git branch subtrees/$short_refname
fi
git checkout subtrees/$short_refname
git merge $short_refname
git subtree split --prefix=$subtree --annotate="(horde) " --branch subtrees/$module/$short_refname --rejoin
git push git@github.com:horde/$module.git subtrees/$module/$short_refname:$short_refname
git config --bool core.bare true

The procedure runs in a loop that walks through the different modules of the horde repository. $module refers to the current module, $subtree to the path corresponding to this module, and $short_refname indicates the branch that was pushed to.

I'll walk you through the different steps...

The remote repository needs to be in a state where you can push updates to a branch to it: it needs to be "bare". Updates to the branch currently checked out in the remote repository would otherwise be impossible. git subtree however requires us to work on a real checkout. So before initiating the splitting process the repository is marked as non-"bare":

git config --bool core.bare false

And subsequently the branch that was pushed to is being checked out and resetted to HEAD - this is the basis for git subtree to work its magic.

git checkout $short_refname
git reset --hard HEAD

The splitting procedure benefits from using a separate branch that remembers the previous splitting operations using merge commits. It would also work without such a branch but the subtree operation would always have to walk through each and every commit of the repository again - a waste of time. Just in case this subtree specific branch does not exist it will be created with the next step.

if [ -z "`git branch | grep subtrees/$short_refname`" ]; then
    git branch subtrees/$short_refname
fi

All updates that were just pushed into the remote repository are now being merged into the branch specifically created for the subtree operation. Here the original line of development and the subtree marker merges (which do not affect the code itself) live together in one branch.

git checkout subtrees/$short_refname
git merge $short_refname

This prepared the stage for the splitting operation which can now analyze the incoming commits for any changes to the module currently handled.

git subtree split --prefix=$subtree --annotate="(horde) " --branch subtrees/$module/$short_refname --rejoin

If there were changes that affected the current module they will be pushed to the corresponding git repository on github using the following command:

git push git@github.com:horde/$module.git subtrees/$module/$short_refname:$short_refname

And finally the repository will be marked as bare again to prepare it for the next commit:

git config --bool core.bare true

Of course of all this is still somewhat untested. It still has to be shown to work large scale - with about one hundred different Horde modules at the same time. But at least it looks very promising.

Friday, May 25, 2012

Creating your own Horde module - Step 1

We always had a wiki page on how to create your own Horde module. The steps outlined there allow you to get started quickly.

There was no specific reason though why those steps could not be automated. Based on the core script presented on the aforementioned wiki page I created a complete script that allows you to initiate a new Horde module with a single command.

framework/devtools/horde-generate-module.php fancy "Gunnar Wrobel <wrobel@horde.org>"

The line above will create the stub for the new Horde application Fancy.

Once you created the new module this way you still need to register the new application in the Horde registry. This can be done within the directory config/registry.d. For details see the README file in that directory.

In addition you can set an icon that should represent your module. Refer to the instructions on the wiki page for that.

Horde at LinuxTag 2012


We have a great time here at LinuxTag. Somehow the KDE project managed to get us placed right beside them again. Apparently the constant stream of gummy bears we fed them during CeBIT has turned into a serious additiction. By now they kind of depend on us. And of course we can't let them off the hook now.

Beside that we had a good amount of chats with people interested in the Horde groupware. And we received a lot of positive feedback about the new design which is great. Looks like everyone is looking forward to it.

And since the mornings are usually somewhat quieter Jan even has some time to work on the redesign and commit code. Can't wait for Horde 5.0 anymore...

Wednesday, May 23, 2012

Generating default preferences for your Horde installation

While updating our demo server at demo.horde.org I also wanted to get some kind of automatic reset for the demo user preferences. The aim was to allow each user testing our demo system to play around with the system in every way desired - while at the same time resetting to the defaults once the next user logs in. This way the second user wouldn't get an unpleasant surprise and turn away from Horde if the first user selected our beloved "Barbie" theme. By the way: this one will be dropped with the redesign - I hope nobody flames our mailing lists and starts a revolution to get this pink atrocity back :)

In order to achieve that I needed to move from SQL based preferences to Session based preferences. Using the latter would purge any preference changes at the end of the session. In order to provide the users with some sane and useful defaults though I also Wanted to copy the old values from the SQL database into the prefs.local.php files. This way I can set the preferences so that users are immediately greeted with a portal screen that demonstrates the twitter and facebook integration.

Converting SQL based preferences to prefs.local.php default settings is something you might be interested in for your installation as well. It allows you to set appropriate defaults for one initial test user and convert those into site-wide defaults for your installation.

How to do that without much hassle? horde-prefs to the rescue!

horde-prefs is a small tool that allows printing, exporting, and importing of preference values stored in a backend.

In order to use the tool you need to define a configuration file for the specific backend you want to access. For the demo server this has been a SQL database:


   $conf['driver'] = 'Sql';
   $conf['params']['db'] = new Horde_Db_Adapter_Mysql(
     array(
      'persistent' => false,
      'username' => 'root',
      'password' => 'PASSWORD',
      'socket' => '/var/run/mysqld/mysqld.sock',
      'protocol' => 'unix',
      'database' => 'horde',
      'charset' => 'utf-8',
      'ssl' => true,
      'splitread' => false,
      'phptype' => 'mysql',
    )
  );

Now you can access the stored preference values by calling the horde-prefs tool. As horde-prefs is not stored under /usr/bin on the home server but uses a non-standard location I prefix the command with an explicit PHP include path:

php -d include_path="/var/www/pear/php" /var/www/pear/horde-prefs config.php guest print horde

The previous command prints all preferences values stored for the guest user for the application horde:

...
$_prefs['twitter']['value'] = "a:2:{s:3:"key";s:50:"183748047-vr6RLMOiYhfbfTH3qI8Lc8E32jF4UGGFbIxdkZyt";s:6:"secret";s:42:"i2DGXInJBW4kk3r2bvBdrxzUKMxL6AYS4u97WAJDyQ";}"
$_prefs['upgrade_tasks']['value'] = "a:7:{s:5:"horde";s:6:"4.0.13";s:9:"kronolith";s:5:"3.0.5";s:8:"imp_auth";s:5:"5.0.8";s:3:"imp";s:5:"5.0.8";s:5:"turba";s:5:"3.0.7";s:5:"whups";s:10:"2.0-ALPHA1";s:6:"gollem";s:10:"2.0-ALPHA2";}"
...

To convert this into a format suitable for the prefs.local.php files I simply used sed.

php -d include_path="/var/www/pear/php" /var/www/pear/horde-prefs config.php guest print horde | sed -e "s/^\([^:]*\): \(.*\)/\$_prefs\['\1'\]\['value'\] = '\2';/" >> /var/www/config/prefs.local.php

Tuesday, May 22, 2012

Getting CalDAV and CardDAV server capabilities within Horde

Would you like to sponsor CalDAV and CardDAV server capabilities within Horde? This has been a Horde feature wish for quite some time now. We would have liked to have it for Horde 4.0 already and it looked more feasible to get it with Horde 5.0. But ultimately we had to delay it again as the redesign had the priority.

We decided however that we would like to tackle the DAV topic with top priority right after Horde 5.0 has been released. And we already have some pledges of support to get the feature completed - but we would need a few more to get complete CalDAV and CardDAV support within Horde as the whole thing is a larger story.

If you have some interest in the functionality it would be great if you could contact us via info@horde.org. Or you could use the "Donate" button on our homepage.

Thanks for you support!

Friday, May 11, 2012

A sneak peek of the new Horde 5 user interface



To get an idea on how Horde 5 will look like: click the link or the image of this post.

Why does Horde 5 get a face lift? Simply because the current UI was mentioned often enough as an issue by many Horde users. And since the Horde 4 release had a very technical focus the switch from Horde 3 to Horde 4 last year did not help - it even degraded consistency between the applications. At the same time the competition does not sleep and there are more and more large installations that offer their user base two different webmails - one of them being Horde for the power users that feel they need a lot of features but that care less about the UI. Time to get our act together.

So what is the primary target of the redesign? First and foremost we want to unify the main user interfaces. At the moment we have the static application views, the dynamic webmailer, and the dynamic calender as the core parts. All looked somewhat different. These are the elements that we wish to give a consistent look. The special views such as the minimal webmailer or the smartphone UI will remain untouched.

We also hope the new design looks somewhat fresher than what we had before but please keep in mind that we are oriented towards people that use the interface for their daily work. We do not aim for a UI that looks like the last hype. It should be functional instead.

The Horde LLC has been the driving factor behind the redesign. At least financially. A subset of the Horde core developers started the LLC a while back as a contact point for people that want to pay for Horde support or feature development. A part of the money that such contracts pay goes to the developers dealing with the particular customer request. But another part of the money remains within the LLC. The idea is to use the latter to drive features that we consider to be important for Horde and its community. The redesign is the first project that has been financed this way. The Horde team tried finding designers interested in contributing to an Open Source project several times before. This was unsuccessful however and paying a designer for the work remained the only reasonable alternative.

We contracted No agency for the design. After several rounds of communication between them and all Horde developers we managed to end up with the draft displayed above. This has been converted to HTML and CSS this week and will be hammered into code during the next week by Jan Schneider. We do hope to present you with an alpha of Horde 5 - including the draft of the new design - on the 22nd May of 2012.

Feedback and comments - as usual - are welcome!

Thursday, March 08, 2012

Horde becomes biggest KDE sponsor!



Here at CeBIT we support our friendly neighbor project with a constant and vital support of gummy bears. As anyone knows these sweet animals can make the difference between one line of brilliant code and a dreadful spaghetti mess. Thus it is probably hard to deny that this fruitful collaboration turned Horde into one of the biggest KDE sponsors.


That being said: KDE, I'm already here, breakfast is ready ;)


Beside having fun in the open source area the second day was already packed with people here at CeBIT. We had plenty of Horde users which provided kind feedback. Some of them we could surprise with features they didn't know about. Others were happy to hear that "GPL" really means that they can use the software and modify it without being harassed with a lawsuit afterwards.


We had Horde newbies as well as free software newbies. Explaining how free software can result in a revenue was the easy part. Explaining why we have no strong interest in a product for obfuscating our code so that there is a decent protection against people trying to find security holes was ... sigh ... harder.


The most fascinating thing was a company that installed Horde and wants to run it from -25°C to 65°C - like putting Horde to the extreme. There were other extremes involved and I omit the details but it is always fascinating what people do with free software.

Wednesday, March 07, 2012

First day on CeBIT



Wonderful start at CeBIT yesterday. Meeting people with an interest in Horde face-to-face is a refreshing alternative to the work behind the screen at home or at work. The positive feedback helps tremendously in building up energy.


The most important part yesterday was talking to well known contacts, chatting about the progress of some projects which will hopefully result in a few interesting feature additions to Horde during this year.


I also visited ownCloud at their Univention booth to chat a bit about integrating the tool with a webmailer.


And last but not least it is always fun seeing Jan in person. Nothing against having a "distributed" type of project with people scattered in Germany and US. But it would be so much fun having all of you guys here. Can't really wait for the next hackathon ;)

Wednesday, February 22, 2012

Recap of the verification that there was no backdoor in the Horde 4 packages



When we discovered the successful attack on ftp.horde.org two weeks ago we were of course frantic to determine which packages had been affected in addition to the one Horde 3 archive Jan identified as modified initially.

For the Horde 4 packages we had no hashes to verify the file integrity though. While PEAR supports signing of packages via GPG that seems to be a feature which is virtually unused. For one thing not that many PHP based projects use PEAR packaging and in addition there is no way to automatically verify package integrity on the user side when installing via PEAR. So we also didn't consider signing our packages when switching to installing Horde via PEAR.

Obviously you gain a different perspective on that issue once a hacker implanted a backdoor in some of your packages. Of course we invested a lot of time into securing our infrastructure now to ensure that such an event never happens again. On our side the file integrity is constantly monitored now. But we will also have to investigate how we can improve the PEAR based installation procedure so that it also allows for the required amount of security on the user side.

But if we had no hashes how did we ensure the Horde 4 packages were indeed unmodified? Git to the rescue! As we tag all our releases it was a matter of creating a short script to automatically compare the current state of the packages on our PEAR server against the state we had within git.

Without further ado - here is the script I used:

#!/bin/bash

git reset --hard HEAD
git clean -f -d

STAMP=`date +%y%m%d-%H%M`
mkdir ../diffs-$STAMP
mkdir -p ../validate-$STAMP/pear.horde.org
mkdir -p ../validate-$STAMP/rebuild

for package in `cat ../pear-recovery-packages.txt | grep -v ".tar$"`
do
  TAG=${package/.tgz/}
  TAG=${TAG,,}
  PPATH=${package/-*/}
  if [ "x${PPATH/Horde_*/}" == "x" ]; then
      PPATH=framework/${PPATH/Horde_};
  fi
  if [ "x${PPATH/groupware*/}" == "x" ]; then
      PPATH=bundles/$PPATH;
  fi
  if [ "x${PPATH/webmail*/}" == "x" ]; then
      PPATH=bundles/$PPATH;
  fi
  PRESENT=`git tag -l $TAG`
  if [ "x$PRESENT" == "x" ]; then
      echo
      echo "======================================================================"
      echo "Tag $TAG for package $package is missing!"
      echo "======================================================================"
      echo
      echo "$package: TAG MISSING" >> ../status-$STAMP
  else
      rm *.tgz                                                                                                               
      rm -rf ../validate-$STAMP/pear.horde.org/*
      rm -rf ../validate-$STAMP/rebuild/*
      GIT=`git checkout $TAG`
      horde-components -z $PPATH --keep-version
      if [ -e $package ]; then
          cp *.tgz ../validate-$STAMP/pear.horde.org/
          cp ../pear.horde.org/get/$package ../validate-$STAMP/rebuild/
          tar -C ../validate-$STAMP/pear.horde.org/ -x -z -f ../validate-$STAMP/pear.horde.org/*.tgz
          tar -C ../validate-$STAMP/rebuild/ -x -z -f ../validate-$STAMP/rebuild/*.tgz
          DIFF=`diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/}`
          if [ "x$DIFF" != "x" ]; then
              echo
              echo "======================================================================"
              echo "Diff for package $package detected!"
              diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/} > ..$
              echo "======================================================================"
              echo
              echo "$package: DIFF" >> ../status-$STAMP
          else
              echo
              echo "======================================================================"
              echo "$package CLEAN!!!"
              echo "======================================================================"
              echo
              echo "$package: CLEAN" >> ../status-$STAMP
          fi
      else
          echo
          echo "======================================================================"
          echo "Failed rebuilding package $package!"
          echo "======================================================================"
          echo
          echo "$package: FAILED REBUILDING" >> ../status-$STAMP
      fi
  fi
done


The script walks through the list of packages we had on the PEAR
server, moves back in time within our git repository to the
appropriate tag, rebundles the package, extracts both the old and the
new package and compares them using diff.


The resulting status list looked quite okay. There were a few release glitches where the tag was not on the exact commit that was used for building the package. In that case usually a small diff resulted. The list was rechecked manually for any malicious traces - Horde_Imap_Client-1.4.3 was the only one were the diff was large but that turned out to be a result from a mishap while releasing that package version. A few times the diff led to a problem with rebuilding the package (indicated by "FAILED REBUILDING"). The package.xml was broken in those cases and needed a fix that was only committed after the tagging. Here I compared the git state directly against the extracted package. There was only a single case ("Horde_ActiveSync-1.1.11") where the tag was missing which required manually identifying the corresponding commit to verify that state against the old package contents.


Once the script was established the analysis ran for about two hours after which we were at least somewhat relieved. Having a backdoor in some Horde 3 packages was already bad enough - having that in Horde 4 would have been even more disastrous.

Thursday, February 02, 2012

Horde on CeBIT 2012

It is official now: Horde gets a booth on CeBIT 2012! Sponsored by Linux New Media the company behind the Linux Magazine. Thanks! Simply awesome!

Hope to see you there!