Tuesday, December 13, 2011

The Horde release train

The Horde project did push out 906 releases in the last 8 months since the initial Horde 4 release. PEAR packages, better release management tools and continuous integration seem to pay off. The stream of bug fixes and improvements available to the users has switched to high speed.

At the same the time the use of PEAR packages and automatic DB migrations has lowered the effort of updates on the administrator side to an absolute minimum.

pear upgrade -c horde

And few clicks later you are up-to-date.

The code quality of the basic PEAR tools is an entirely different matter... BUT it is just soo damn cool anyway ;)

Sunday, November 13, 2011

Accessing PEAR server information with PHP

The number of PEAR servers has increased dramatically in recent times. Maybe due to the availability of Pirum - a simple PEAR server. Whatever the cause: there are plenty of package lists and package datasets available on the net these days. Accessing this data is not always easy though.

PEAR servers provide a well defined REST API that is usually queried using the PEAR toolset available via the basic PEAR package. That package however does not provide anything that I would consider to be a decent developers API.

So let me provide you with an alternative that is available via the Horde framework libraries: The Horde_Pear library.

The package comes with the Horde_Pear_Remote class wich provides you with high-level access to the REST interface of a PEAR server.

Creating an instance of this class without providing any arguments to the constructor will allow you to access the PEAR server at pear.horde.org.

$pear = new Horde_Pear_Remote();
print(join("\n", $pear->listPackages()));

Horde_ActiveSync
Horde_Alarm
Horde_Argv
Horde_Auth
Horde_Autoloader
...

Alternative servers can be specified in the first argument to the constructor:

$pear = new Horde_Pear_Remote('pear.phpunit.de');
print(join("\n", $pear->listPackages()));

DbUnit
File_Iterator
Object_Freezer
PHPUnit
...

The full set of the functionality provided by Horde_Pear_Remote is detailed on our website.

Friday, November 04, 2011

The library section of the Horde website supports component documentation now.

Our new libraries section has been started a while ago to further push the PHP components we offer. Now this section supports publishing the component documentation as well. You can take a look at the documentation of the Cli_Modular package for example. In fact: it is pretty much the only package that fully uses the new system already. So there is still work ahead.

If you use any of the components: We are grateful if you start writing a bit of documenation or describe some examples on how to successfully use the package. It will help us and others tremendously. Thanks!

Let me try to explain how the system is currently intended to work - feedback and critical comments of course welcome:

  1. When creating new documentation for one of the Horde components you start a new section on the developer documentation of our wiki. See the Horde_Cli_Modular link below Library components for example. Please note: It is not mandatory to write the documentation in the wiki. You can have static documentation files within a component as well. This is just meant as an option for those situations where it makes sense to work on documentation files together with people that do not have direct git commit access. Hopefully there will be many component consumers interested in updating and fixing the documentation.
  2. How you structure that new section is up to you. For a simple package you might wish to keep all documentation in a single file but for the more complex one it might make sense to have several files. In that case the new section page should probably just be a link list to the various wiki pages that document the component. In case of Horde_Cli_Modular I used a single page.
  3. Once the documentation has been written in the wiki format it is time to download it into the doc directory of the component. The horde-components helper is what you should use for that operation. The tool expects to find a DOCS_ORIGIN file within the doc or docs (for the applications) directory of the component. This file must conform to the reStructuredText format and map remote URLs to local file paths relative to the component root. The DOCS_ORIGIN from Cli_Modular links the wiki page exported as reStructuredText to the path doc/Horde/Cli/Modular/README. The links can of course point anywhere so you are in principle free to use any remote source. But you should ensure the page provides readable reStructuredText.
  4. Now you can run horde-components fetchdocs and it should fetch the URLs into the respective local files. You can then run horde-components update to refresh the file list in the package.xml file if the documentation files were not included before.
  5. Once a component was released we can now run something like this: horde-components Horde_Cli_Modular webdocs -D ~/git/horde-web/ --html-generator=git/horde-support/maintainer-tools/docutils/html.py --allow-remote in order to update the documentation on the website. Right now this does not happen automatically during a release but I plan to add this soon.

There are of course still a few problems with this approach:

  • The reStructuredText exporter of our wiki engine "Wicked" is not yet complete. You may experience problems with the export if you use constructs it does not know yet. Please ping me if you experience problems with the export.
  • The wiki syntax and the syntax required by reStructuredText is not 100% aligned. You can write a wiki page that looks just fine but leads to errors or formatting problems in the reStructuredText export. After creating / updating a new wiki documentation page it make sense to run the horde-components fetchdocs/horde-components webdocs sequence to check for problems.
  • We still have some components with documentation files not in reStructuredText format. These will result in non-existing links on our website for now but I plan to fix the few faulty packages soon.

Jenkins on ci.horde.org got a major update

The Horde continuous integration setup saw a major upgrade and a number of fixes during the last week.

The take home message for everyone that does not wish to read the full log: ci.horde.org will now run horde/framework/bin/test_framework after a new commit has been pushed. This reduces the load on the system significantly and you will get results faster. If you develop Horde code you should run horde/framework/bin/test_framework as well in order to check if your recent commits could cause failures on our continuous integration server.

The full recap of the recent changes:

  • Update to the newest version of the Horde components tool. This allows to use the improved templating system.
  • New configuration and build templates that allow to reduce the job build time by avoiding to rebuild the set of package dependencies if this is not necessary. This has an important implication: A code change in one package (e.g. Horde_Imap_Client) will be tested against unchanged dependencies (e.g. Horde_Mime). Even if the commit also touched one or several of the dependencies (e.g. Horde_Mime). The packages should be backward compatible - so this should result in no error. The dependencies of one package will only get updated in case the dependency list in the package.xml of that package changes.
  • Running horde/framework/bin/test_framework has been integrated into the horde-git job. With the new "rebuild dependencies only when necessary" policy (see above) the component jobs do not fully test the integrity of the latest commit anymore. When the dependencies do not get updated the focus of shifts a bit more towards checking for backward compatibility. In order to not loose the integrity check for the "bleeding edge" our test_framework script is now executed right after updating the code from git.
  • Running test_framework also allows to run component jobs only if the code of the component has actually been touched. This significantly reduces the load on ci.horde.org as it removes the need for rebuilding all components for each and every commit.
  • An update to the newer CodeSniffer which included an update to the checklist for the Horde style. The new ruleset is available from our horde-support repository.
  • A customizable ruleset for the PHP mess detector has been added as well. We still need to tweak the exact configuration of PMD to match it with what we consider reasonable defaults for Horde code.
  • DocBlox has been added to allow generating experimental API documentation. DocBlox is significantly faster and less memory hungry than PHPDocumentor. It has already been adopted by large frameworks such as Zend. So I figured it is worth taking a look at it. Feedback welcome!
  • The Jenkins configuration has been updated with the latest fixes and improvements from the Jenkins-PHP project.
  • The setup procedure has been fixed so that it should be possible to generate a local setup - comparable to ci.horde.org - again.

Tuesday, October 18, 2011

demo.horde.org got updated

You might have noticed that the Horde homepage received a new Demo button within the main navigation recently. We did indeed manage to get a demo machine up and running at demo.horde.org. The installation got updated to the latest releases just yesterday and Ansel - the Horde photo management application - got added into the mix. Feel free to test drive this new application as well as the other components on demo.horde.org.

If you experience any problems please let us know.

In principle you shouldn't be able to break anything on the installation. It is a virtual EC2 machine and we will update the image every once in a while when there are new releases available. Any data you store on the machine will be resetted at that point.

You should also be able to mail to the two demo users demo@demo.horde.org and guest@demo.horde.org. Mailing to the outside is not possible - for obvious reasons.

Friday, October 14, 2011

Horde_Push now supports sending to Blogger.com

Horde_Push supports sending to blogger.com now.

The Horde_Push library is a tool to send content elements to various external services. Such as Twitter, Facebook, Blogger and others. Right now it only all
ows sending Tweets, publishing entries on blogger.com and sending e-mails.

The idea is to allow publishing content you curate on a Horde installation to the social networks out there. At the moment the code is not yet there as the
integration into the base horde package is still lacking. Right now there is only a small command line helper that I use for testing.

Tuesday, June 21, 2011

Anatomy of a Horde test suite - III

Ready for the next item on the test suite agenda? This time the topic is Autoloading. We use a rather simple autoloading setup for most component test suites. It requires no additional setup and works out of the box if you run php AllTests.php.

That is already nice and allows running the complete test suite without further ado. But I must admit that I want more. My default work mode looks like this:

  • open test case (and modify it)
  • hit <f3> <f8> to run phpunit on this single test case
  • hit <f4> <f4> to to jump to the code line that produced the first error

This allows me to add a new test definition and immediately run it so that I can check for problems. And I don't need to execute the full AllTests.php to get feedback on the new test. So I'm annoyed every time I hit a test case that does not allow me to do that. A working autoloading setup is the key for that.

Luckily not only my own preferences make using an Autoload.php file in a test suite attractive. There are a number of reasons why such a file can be useful. The Wiki page for the Horde_Test component details them and this is a copy of the relevant section:


The Autoload.php file is not required in a test suite but it is strongly recommended that you use it. It's purpose is to setup PHP autoloading so that all tests in the test suite automatically have access to all the classes required for executing the tests. The reason why it is not mandatory is that Horde_Test_AllTests already loads a basic autoloading definition that works for most framework components.

This means that running php AllTests.php usually does not hit any autoloading problems. Running a single test case (e.g. phpunit Horde/Xyz/Unit/UnitTest.php) is a different matter though.

The *Test.php files do not extend Horde_Test_AllTests and thus there is nothing that would magically setup autoloading if you try to run such a test suite in isolation. And running single test cases can be quite convenient if the whole test suite would take a long time to execute. Using an Autoload.php file alongside the AllTests.php file is the recommended way to provide a single test case with autoloading and thus enable commands such as phpunit Horde/Xyz/Unit/UnitTest.php. In addition the file is helpful for any case where you need slightly more complex loading patterns or want to pull in special files manually.

Once you created an Autoload.php file for your test suite it will also be heeded by Horde/Test/AllTests.php. The latter will avoid the basic autoloading setup if it detects the presence of an Autoload.php file for the current test suite. That one will be loaded and is assumed to contain the required autoloading setup.

The content of Autoload.php

You should at least require the Autoload.php from Horde_Test in this file. This is also what Horde_Test_AllTests would do when choosing the simple autoloading setup.

require_once 'Horde/Test/Autoload.php';

It also makes sense to adapt the error reporting level to the same standards as required in the AllTests.php wrapper:

error_reporting(E_ALL | E_STRICT);

If you derive your test cases from a central test case definition you should load this one in Autoload.php as well:

/** Load the basic test definition */
require_once dirname(__FILE__) . '/TestCase.php';

Sometimes it makes sense to pull in the definition of test helpers that may be used throughout the test suite. They are usually not available via autoloading and need to be pulled in explicitely:

/** Load stub definitions */
require_once dirname(__FILE__) . '/Stub/ListQuery.php';
require_once dirname(__FILE__) . '/Stub/DataQuery.php';

Real world examples for Autoload.php helpers can be found in the Horde_Date and the Kolab_Storage components.

Within the test cases you only need to load the Autoload.php file which usually looks like this (and obviously depends on the position of the test case in the directory hierarchy of the test suite):

require_once dirname(__FILE__) . '/../Autoload.php';

You'll find additional background information on autoloading within test suite runs on the Wiki page for the Horde_Test component.

Monday, June 20, 2011

Horde4 debian packages - Second round

Nearly half a year passed since I started my first attempt at Debian packages for Horde4. Back then it was just about snapshots which I generated via a continuous integration setup. But Horde4 has been released now, there are packages and it is time for the real thing.

I did get a first set of packages installable today but this is just the initial draft. The main point about it is that the majority of it is automated.

If you want more details and the installation steps then I would suggest to follow my discussion with Mathieu Parent on pkg-php-pear@lists.alioth.debian.org. These were the mails exchanged so far (with the last two detailing my current status):

  1. On PEAR packaging (25.5., Mathieu)
  2. On PEAR packaging (7.6., Gunnar)
  3. On PEAR packaging (7.7., Mathieu)
  4. On PEAR packaging (17.7., Gunnar)
  5. On PEAR packaging (20.7., Gunnar)

There is not much more to say right now. Any testing and feedback is welcome and there is obviously still a lot of work ahead until this pops up in your default package channels. But it is at least on the horizon and the variant that I ship on files.pardus.de should become useable very soon.

Tuesday, June 14, 2011

Anatomy of a Horde test suite - II

This morning I completed the next step on the journey through Horde's test suites and added the description of the AllTests.php file to the wiki page. I am not going to copy the complete text here but instead focus on the use cases for this file as I still have a few question to the audience below.


AllTests.php is the only mandatory requirement for a Horde test suite. Everything else is optional but there has to be an AllTests.php file which serves as an entry point into the test suite.

This is the functionality expected from the file:

  1. It must collect all tests of the test suite.
  2. It must allow to retrieve all tests of the suite via Horde_Xyz_AllTests::suite().
  3. It must allow running the test suite via phpunit AllTests.php.
  4. It must allow running the test suite via php AllTests.php.

The Horde_Test package already delivers a boilerplate AllTests.php class in framework/Test/lib/Horde/Test/AllTests.php and deriving an AllTests.php for a standard test suite becomes rather simple. The full code for this is presented on the wiki page and you can also look at an example from our repository.


Now I wonder if the items listed above are in fact all the requirements we have for this file.

Requirements (1) and (2) are obvious as this is functionality needed for our horde/framework/bin/test_framework helper that runs all framework tests. Though I assume nobody uses this one on a regular basis at the moment.

But I noticed that (3) does not work out of the box with the current PHPUnit. This led to a pull request as it definitely should (and can) work.

I usually run the tests with a rather long command line that ultimately boils down to phpunit Horde_Xyz_AllTests AllTests.php which is tied to a shortcut in Emacs. As the Lisp code I use for that extracts the class name automatically I never noticed that a plain phpunit AllTests.php does not work.

So are most people using php AllTests.php? How do you run the test suites or would like to run them? Can I get some feedback on this (either here, on IRC or via tweet)?

Anything additional I missed about the requirements for the AllTests.php file?

Next in the series will be on autoloading which should allow me to also look at the problems we still have with that in the application components.

Thursday, June 09, 2011

The Horde newsletter again

Time flies and it is soon time for the next issue of the Horde newsletter. Which reminds me to post the link to the last one in case you missed it. You can subscribe to the monthly newsletter here.

As Jan already mentioned in his blog he gave Radio Tux an interview on Horde 4. I just checked by roughly translating half of it to English that I should be able to include a transcript of it in the next newsletter. It delivers a very good overview on all things Horde 4.