A sneak peek of the new Horde 5 user interface

To get an idea on how Horde 5 will look like: click the link or the image of this post.

Why does Horde 5 get a face lift? Simply because the current UI was mentioned often enough as an issue by many Horde users. And since the Horde 4 release had a very technical focus the switch from Horde 3 to Horde 4 last year did not help - it even degraded consistency between the applications. At the same time the competition does not sleep and there are more and more large installations that offer their user base two different webmails - one of them being Horde for the power users that feel they need a lot of features but that care less about the UI. Time to get our act together.

So what is the primary target of the redesign? First and foremost we want to unify the main user interfaces. At the moment we have the static application views, the dynamic webmailer, and the dynamic calender as the core parts. All looked somewhat different. These are the elements that we wish to give a consistent look. The special views such as the minimal webmailer or the smartphone UI will remain untouched.

We also hope the new design looks somewhat fresher than what we had before but please keep in mind that we are oriented towards people that use the interface for their daily work. We do not aim for a UI that looks like the last hype. It should be functional instead.

The Horde LLC has been the driving factor behind the redesign. At least financially. A subset of the Horde core developers started the LLC a while back as a contact point for people that want to pay for Horde support or feature development. A part of the money that such contracts pay goes to the developers dealing with the particular customer request. But another part of the money remains within the LLC. The idea is to use the latter to drive features that we consider to be important for Horde and its community. The redesign is the first project that has been financed this way. The Horde team tried finding designers interested in contributing to an Open Source project several times before. This was unsuccessful however and paying a designer for the work remained the only reasonable alternative.

We contracted No agency for the design. After several rounds of communication between them and all Horde developers we managed to end up with the draft displayed above. This has been converted to HTML and CSS this week and will be hammered into code during the next week by Jan Schneider. We do hope to present you with an alpha of Horde 5 - including the draft of the new design - on the 22nd May of 2012.

Feedback and comments - as usual - are welcome!

Horde becomes biggest KDE sponsor!

Here at CeBIT we support our friendly neighbor project with a constant and vital support of gummy bears. As anyone knows these sweet animals can make the difference between one line of brilliant code and a dreadful spaghetti mess. Thus it is probably hard to deny that this fruitful collaboration turned Horde into one of the biggest KDE sponsors.

That being said: KDE, I'm already here, breakfast is ready ;)

Beside having fun in the open source area the second day was already packed with people here at CeBIT. We had plenty of Horde users which provided kind feedback. Some of them we could surprise with features they didn't know about. Others were happy to hear that "GPL" really means that they can use the software and modify it without being harassed with a lawsuit afterwards.

We had Horde newbies as well as free software newbies. Explaining how free software can result in a revenue was the easy part. Explaining why we have no strong interest in a product for obfuscating our code so that there is a decent protection against people trying to find security holes was ... sigh ... harder.

The most fascinating thing was a company that installed Horde and wants to run it from -25°C to 65°C - like putting Horde to the extreme. There were other extremes involved and I omit the details but it is always fascinating what people do with free software.

First day on CeBIT

Wonderful start at CeBIT yesterday. Meeting people with an interest in Horde face-to-face is a refreshing alternative to the work behind the screen at home or at work. The positive feedback helps tremendously in building up energy.

The most important part yesterday was talking to well known contacts, chatting about the progress of some projects which will hopefully result in a few interesting feature additions to Horde during this year.

I also visited ownCloud at their Univention booth to chat a bit about integrating the tool with a webmailer.

And last but not least it is always fun seeing Jan in person. Nothing against having a "distributed" type of project with people scattered in Germany and US. But it would be so much fun having all of you guys here. Can't really wait for the next hackathon ;)

Recap of the verification that there was no backdoor in the Horde 4 packages

When we discovered the successful attack on ftp.horde.org two weeks ago we were of course frantic to determine which packages had been affected in addition to the one Horde 3 archive Jan identified as modified initially.

For the Horde 4 packages we had no hashes to verify the file integrity though. While PEAR supports signing of packages via GPG that seems to be a feature which is virtually unused. For one thing not that many PHP based projects use PEAR packaging and in addition there is no way to automatically verify package integrity on the user side when installing via PEAR. So we also didn't consider signing our packages when switching to installing Horde via PEAR.

Obviously you gain a different perspective on that issue once a hacker implanted a backdoor in some of your packages. Of course we invested a lot of time into securing our infrastructure now to ensure that such an event never happens again. On our side the file integrity is constantly monitored now. But we will also have to investigate how we can improve the PEAR based installation procedure so that it also allows for the required amount of security on the user side.

But if we had no hashes how did we ensure the Horde 4 packages were indeed unmodified? Git to the rescue! As we tag all our releases it was a matter of creating a short script to automatically compare the current state of the packages on our PEAR server against the state we had within git.

Without further ado - here is the script I used:

#!/bin/bash

git reset --hard HEAD
git clean -f -d

STAMP=`date +%y%m%d-%H%M`
mkdir ../diffs-$STAMP
mkdir -p ../validate-$STAMP/pear.horde.org
mkdir -p ../validate-$STAMP/rebuild

for package in `cat ../pear-recovery-packages.txt | grep -v ".tar$"`
do
  TAG=${package/.tgz/}
  TAG=${TAG,,}
  PPATH=${package/-*/}
  if [ "x${PPATH/Horde_*/}" == "x" ]; then
      PPATH=framework/${PPATH/Horde_};
  fi
  if [ "x${PPATH/groupware*/}" == "x" ]; then
      PPATH=bundles/$PPATH;
  fi
  if [ "x${PPATH/webmail*/}" == "x" ]; then
      PPATH=bundles/$PPATH;
  fi
  PRESENT=`git tag -l $TAG`
  if [ "x$PRESENT" == "x" ]; then
      echo
      echo "======================================================================"
      echo "Tag $TAG for package $package is missing!"
      echo "======================================================================"
      echo
      echo "$package: TAG MISSING" >> ../status-$STAMP
  else
      rm *.tgz                                                                                                               
      rm -rf ../validate-$STAMP/pear.horde.org/*
      rm -rf ../validate-$STAMP/rebuild/*
      GIT=`git checkout $TAG`
      horde-components -z $PPATH --keep-version
      if [ -e $package ]; then
          cp *.tgz ../validate-$STAMP/pear.horde.org/
          cp ../pear.horde.org/get/$package ../validate-$STAMP/rebuild/
          tar -C ../validate-$STAMP/pear.horde.org/ -x -z -f ../validate-$STAMP/pear.horde.org/*.tgz
          tar -C ../validate-$STAMP/rebuild/ -x -z -f ../validate-$STAMP/rebuild/*.tgz
          DIFF=`diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/}`
          if [ "x$DIFF" != "x" ]; then
              echo
              echo "======================================================================"
              echo "Diff for package $package detected!"
              diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/} > ..$
              echo "======================================================================"
              echo
              echo "$package: DIFF" >> ../status-$STAMP
          else
              echo
              echo "======================================================================"
              echo "$package CLEAN!!!"
              echo "======================================================================"
              echo
              echo "$package: CLEAN" >> ../status-$STAMP
          fi
      else
          echo
          echo "======================================================================"
          echo "Failed rebuilding package $package!"
          echo "======================================================================"
          echo
          echo "$package: FAILED REBUILDING" >> ../status-$STAMP
      fi
  fi
done

The script walks through the list of packages we had on the PEAR
server, moves back in time within our git repository to the
appropriate tag, rebundles the package, extracts both the old and the
new package and compares them using diff.

The resulting status list looked quite okay. There were a few release glitches where the tag was not on the exact commit that was used for building the package. In that case usually a small diff resulted. The list was rechecked manually for any malicious traces - Horde_Imap_Client-1.4.3 was the only one were the diff was large but that turned out to be a result from a mishap while releasing that package version. A few times the diff led to a problem with rebuilding the package (indicated by "FAILED REBUILDING"). The package.xml was broken in those cases and needed a fix that was only committed after the tagging. Here I compared the git state directly against the extracted package. There was only a single case ("Horde_ActiveSync-1.1.11") where the tag was missing which required manually identifying the corresponding commit to verify that state against the old package contents.

Once the script was established the analysis ran for about two hours after which we were at least somewhat relieved. Having a backdoor in some Horde 3 packages was already bad enough - having that in Horde 4 would have been even more disastrous.

Horde on CeBIT 2012

It is official now: Horde gets a booth on CeBIT 2012! Sponsored by Linux New Media the company behind the Linux Magazine. Thanks! Simply awesome!

Hope to see you there!

Wiki time

Tonight it has been wiki time. The article on IMP from the English wikipedia was a mere stub and I expanded it with a little bit of history. I will hopefully find the time to continue this later. I recently did the same for the Horde article.

In addition I updated the list of Horde deployments in our Horde wiki. I also added a list of Horde hosting providers and a list of alternate installation methods for Horde. I couldn't refrain from adding a short abstract on why we only offer PEAR with Horde 4 to the latter article.

Of course corrections and additions are very welcome!

Horde is calling you

Yes, it's late... and I should either sleep or try to work for a final hour before collapsing onto a pillow. But instead I feel like writing a few lines. And calling strangers in the US...

I don't know the people I call there. But it is fun, I train my english, and on top they are usually happy about my call.

What we talk about? I explain them what a provider is. And why I'm actually unable to help them. Which maybe helpful information in its own right.

Of course these people called the Horde LLC before. The Horde webmail is running on so many servers around the world that we get a constant stream of requests for help from users that mistake Horde for their service provider. Usually we get a handful of e-mails every day asking whether we could reset the password, restart the server, or in general just "HELP!!!". Once in a week the line

I can't log in. My username is ABC, my password is XYZ.

reminds us how easy phishing is.

E-Mails are simple: sending out a friendly response with a link to one of our most successful wiki pages is an easy thing to do.

Phone calls are a different matter though. The phone number for calling us in the US has been set up by MojoLingo based on Adhearsion a while back. And of course we get a certain number of "I can't get my mail!" calls as well. We usually didn't answer these until MojoLingo helped us with setting up Voip access so that calls into the US generate negligible costs.

So I can sit here, run SipDroid on my little droid and call into US when I need a break or have a minute to spare. A few more pleas for help that do not get lost unanswered.

In case you ever get called by a stranger with a weird German accent after your Horde webmail broke you probably didn't read this blog post.

The Horde release train

The Horde project did push out 906 releases in the last 8 months since the initial Horde 4 release. PEAR packages, better release management tools and continuous integration seem to pay off. The stream of bug fixes and improvements available to the users has switched to high speed.

At the same the time the use of PEAR packages and automatic DB migrations has lowered the effort of updates on the administrator side to an absolute minimum.

pear upgrade -c horde

And few clicks later you are up-to-date.

The code quality of the basic PEAR tools is an entirely different matter... BUT it is just soo damn cool anyway ;)

Accessing PEAR server information with PHP

The number of PEAR servers has increased dramatically in recent times. Maybe due to the availability of Pirum - a simple PEAR server. Whatever the cause: there are plenty of package lists and package datasets available on the net these days. Accessing this data is not always easy though.

PEAR servers provide a well defined REST API that is usually queried using the PEAR toolset available via the basic PEAR package. That package however does not provide anything that I would consider to be a decent developers API.

So let me provide you with an alternative that is available via the Horde framework libraries: The Horde_Pear library.

The package comes with the Horde_Pear_Remote class wich provides you with high-level access to the REST interface of a PEAR server.

Creating an instance of this class without providing any arguments to the constructor will allow you to access the PEAR server at pear.horde.org.

$pear = new Horde_Pear_Remote();
print(join("\n", $pear->listPackages()));

Horde_ActiveSync
Horde_Alarm
Horde_Argv
Horde_Auth
Horde_Autoloader
...

Alternative servers can be specified in the first argument to the constructor:

$pear = new Horde_Pear_Remote('pear.phpunit.de');
print(join("\n", $pear->listPackages()));

DbUnit
File_Iterator
Object_Freezer
PHPUnit
...

The full set of the functionality provided by Horde_Pear_Remote is detailed on our website.

The library section of the Horde website supports component documentation now.

Our new libraries section has been started a while ago to further push the PHP components we offer. Now this section supports publishing the component documentation as well. You can take a look at the documentation of the Cli_Modular package for example. In fact: it is pretty much the only package that fully uses the new system already. So there is still work ahead.

If you use any of the components: We are grateful if you start writing a bit of documenation or describe some examples on how to successfully use the package. It will help us and others tremendously. Thanks!

Let me try to explain how the system is currently intended to work - feedback and critical comments of course welcome:

1. When creating new documentation for one of the Horde components you start a new section on the developer documentation of our wiki. See the Horde_Cli_Modular link below Library components for example. Please note: It is not mandatory to write the documentation in the wiki. You can have static documentation files within a component as well. This is just meant as an option for those situations where it makes sense to work on documentation files together with people that do not have direct git commit access. Hopefully there will be many component consumers interested in updating and fixing the documentation.
2. How you structure that new section is up to you. For a simple package you might wish to keep all documentation in a single file but for the more complex one it might make sense to have several files. In that case the new section page should probably just be a link list to the various wiki pages that document the component. In case of Horde_Cli_Modular I used a single page.
3. Once the documentation has been written in the wiki format it is time to download it into the doc directory of the component. The horde-components helper is what you should use for that operation. The tool expects to find a DOCS_ORIGIN file within the doc or docs (for the applications) directory of the component. This file must conform to the reStructuredText format and map remote URLs to local file paths relative to the component root. The DOCS_ORIGIN from Cli_Modular links the wiki page exported as reStructuredText to the path doc/Horde/Cli/Modular/README. The links can of course point anywhere so you are in principle free to use any remote source. But you should ensure the page provides readable reStructuredText.
4. Now you can run horde-components fetchdocs and it should fetch the URLs into the respective local files. You can then run horde-components update to refresh the file list in the package.xml file if the documentation files were not included before.
5. Once a component was released we can now run something like this: horde-components Horde_Cli_Modular webdocs -D ~/git/horde-web/ --html-generator=git/horde-support/maintainer-tools/docutils/html.py --allow-remote in order to update the documentation on the website. Right now this does not happen automatically during a release but I plan to add this soon.

There are of course still a few problems with this approach:

• The reStructuredText exporter of our wiki engine "Wicked" is not yet complete. You may experience problems with the export if you use constructs it does not know yet. Please ping me if you experience problems with the export.
• The wiki syntax and the syntax required by reStructuredText is not 100% aligned. You can write a wiki page that looks just fine but leads to errors or formatting problems in the reStructuredText export. After creating / updating a new wiki documentation page it make sense to run the horde-components fetchdocs/horde-components webdocs sequence to check for problems.
• We still have some components with documentation files not in reStructuredText format. These will result in non-existing links on our website for now but I plan to fix the few faulty packages soon.