Showing posts with label quality control. Show all posts
Showing posts with label quality control. Show all posts

Tuesday, June 21, 2011

Anatomy of a Horde test suite - III

Ready for the next item on the test suite agenda? This time the topic is Autoloading. We use a rather simple autoloading setup for most component test suites. It requires no additional setup and works out of the box if you run php AllTests.php.

That is already nice and allows running the complete test suite without further ado. But I must admit that I want more. My default work mode looks like this:

  • open test case (and modify it)
  • hit <f3> <f8> to run phpunit on this single test case
  • hit <f4> <f4> to to jump to the code line that produced the first error

This allows me to add a new test definition and immediately run it so that I can check for problems. And I don't need to execute the full AllTests.php to get feedback on the new test. So I'm annoyed every time I hit a test case that does not allow me to do that. A working autoloading setup is the key for that.

Luckily not only my own preferences make using an Autoload.php file in a test suite attractive. There are a number of reasons why such a file can be useful. The Wiki page for the Horde_Test component details them and this is a copy of the relevant section:


The Autoload.php file is not required in a test suite but it is strongly recommended that you use it. It's purpose is to setup PHP autoloading so that all tests in the test suite automatically have access to all the classes required for executing the tests. The reason why it is not mandatory is that Horde_Test_AllTests already loads a basic autoloading definition that works for most framework components.

This means that running php AllTests.php usually does not hit any autoloading problems. Running a single test case (e.g. phpunit Horde/Xyz/Unit/UnitTest.php) is a different matter though.

The *Test.php files do not extend Horde_Test_AllTests and thus there is nothing that would magically setup autoloading if you try to run such a test suite in isolation. And running single test cases can be quite convenient if the whole test suite would take a long time to execute. Using an Autoload.php file alongside the AllTests.php file is the recommended way to provide a single test case with autoloading and thus enable commands such as phpunit Horde/Xyz/Unit/UnitTest.php. In addition the file is helpful for any case where you need slightly more complex loading patterns or want to pull in special files manually.

Once you created an Autoload.php file for your test suite it will also be heeded by Horde/Test/AllTests.php. The latter will avoid the basic autoloading setup if it detects the presence of an Autoload.php file for the current test suite. That one will be loaded and is assumed to contain the required autoloading setup.

The content of Autoload.php

You should at least require the Autoload.php from Horde_Test in this file. This is also what Horde_Test_AllTests would do when choosing the simple autoloading setup.

require_once 'Horde/Test/Autoload.php';

It also makes sense to adapt the error reporting level to the same standards as required in the AllTests.php wrapper:

error_reporting(E_ALL | E_STRICT);

If you derive your test cases from a central test case definition you should load this one in Autoload.php as well:

/** Load the basic test definition */
require_once dirname(__FILE__) . '/TestCase.php';

Sometimes it makes sense to pull in the definition of test helpers that may be used throughout the test suite. They are usually not available via autoloading and need to be pulled in explicitely:

/** Load stub definitions */
require_once dirname(__FILE__) . '/Stub/ListQuery.php';
require_once dirname(__FILE__) . '/Stub/DataQuery.php';

Real world examples for Autoload.php helpers can be found in the Horde_Date and the Kolab_Storage components.

Within the test cases you only need to load the Autoload.php file which usually looks like this (and obviously depends on the position of the test case in the directory hierarchy of the test suite):

require_once dirname(__FILE__) . '/../Autoload.php';

You'll find additional background information on autoloading within test suite runs on the Wiki page for the Horde_Test component.

Tuesday, June 14, 2011

Anatomy of a Horde test suite - II

This morning I completed the next step on the journey through Horde's test suites and added the description of the AllTests.php file to the wiki page. I am not going to copy the complete text here but instead focus on the use cases for this file as I still have a few question to the audience below.


AllTests.php is the only mandatory requirement for a Horde test suite. Everything else is optional but there has to be an AllTests.php file which serves as an entry point into the test suite.

This is the functionality expected from the file:

  1. It must collect all tests of the test suite.
  2. It must allow to retrieve all tests of the suite via Horde_Xyz_AllTests::suite().
  3. It must allow running the test suite via phpunit AllTests.php.
  4. It must allow running the test suite via php AllTests.php.

The Horde_Test package already delivers a boilerplate AllTests.php class in framework/Test/lib/Horde/Test/AllTests.php and deriving an AllTests.php for a standard test suite becomes rather simple. The full code for this is presented on the wiki page and you can also look at an example from our repository.


Now I wonder if the items listed above are in fact all the requirements we have for this file.

Requirements (1) and (2) are obvious as this is functionality needed for our horde/framework/bin/test_framework helper that runs all framework tests. Though I assume nobody uses this one on a regular basis at the moment.

But I noticed that (3) does not work out of the box with the current PHPUnit. This led to a pull request as it definitely should (and can) work.

I usually run the tests with a rather long command line that ultimately boils down to phpunit Horde_Xyz_AllTests AllTests.php which is tied to a shortcut in Emacs. As the Lisp code I use for that extracts the class name automatically I never noticed that a plain phpunit AllTests.php does not work.

So are most people using php AllTests.php? How do you run the test suites or would like to run them? Can I get some feedback on this (either here, on IRC or via tweet)?

Anything additional I missed about the requirements for the AllTests.php file?

Next in the series will be on autoloading which should allow me to also look at the problems we still have with that in the application components.

Thursday, June 09, 2011

Anatomy of a Horde test suite - I

Just got issue 07/2011 of the the German Linux Magazine in the mailbox and on the final page there is this little abstract about 08/2011 saying...

"PHP Unit and Jenkins - There are two things guarding against programming errors: unit tests covering your code and continuous integration systems that automate the testing. The next issue will demonstrate this based on a real example from a PHP web project." [translated from German].

The "PHP web project" is actually named "Horde" and hm... I guess this means I have to write this thing - ;) . When agreeing to the article I immediately knew I wanted to combine it with an overview on how the Horde test suites are arranged. So far we have been lacking a summary in that area and it should help newcomers to the Horde project to get into testing mode as well.

My mind is currently fully tuned to unit testing and code quality and it is amazing how easy it is to write about this. The initial draft for the article already exceeded all limits when it comes to size. Though I got pretty positive feedback on it I will have to leave some stuff out. Those sections should make it to this blog instead so that I can link to it in the article.

Basically I will make this into a short series of blog entries on unit testing in Horde. I will include parts of the Horde_Test overview, personal musings, and stuff related to the article. Let's hope it is useful to some people out there.

Here we go with the introduction to the Horde_Test overview...


Introduction

The Horde Project has always had high standards when it comes to code quality. Of course these standards evolved with time and also with the progress the PHP community made. The code from IMP-1.0.0 (1998) didn't come with unit tests. And somehow it lacked classes. And there is an awful lot of code mixed with HTML. Somehow this looks horribly like PHP3.

Oh, it was PHP3.

Of course PHP development changed over time and so did the Horde project. Nowadays each and every commit into our repository leads to the automatic execution of thousands of unit tests written by the Horde developers and they check our code for failures. Night and day our continuous integration server broadcasts the current test status to us in particular but also to anyone else interested.

With the release of Horde 4 the test suites of the Horde components available via our PEAR server all show some common patterns. There are certain Do's and Don'ts and a lot of playground in between. Often the Horde_Test component is involved. So it makes sense to associate the overview on the anatomy of Horde test suites with this particular module.


I must admit that I really like the way the Horde project approaches unit testing. There is no way we could be unit test purists which would be too extreme given the fact that the project already exists for more than a decade. But at the same time there was also no one complaining when testing entered the equation. It just felt like continuing to adhere to the quality standards that seem so familiar when it comes to Horde code.

So much for now. More technical stuff to follow soon...

Tuesday, May 31, 2011

Horde continuous integration got updated

The Horde continuous integration server received an update to the most recent version of Jenkins. In addition two new jobs were added to the system: Kolab_Storage and Imap_Client. May they stay forever green! The total number of packages under CI surveillance is 48 by now.

Friday, January 21, 2011

Hudson Quickie

Keyboard

After my third Hudson repository using the same installation procedure it was definitely necessary to finally extract the whole Hudson specific part into its own repository.

It is nothing fancy as the installation procedure with Hudson is pretty straightforward anyhow. But maybe you want to get Hudson quickly running on your own Linux machine with a few standard plugins pre-installed. Then the repository might be exactly what you need.

Just clone the repository with

git clone git://github.com/wrobel/hudson-install.git

and follow the "Install" instructions in the README. Of course you can also just fork the repo in case you need your own predefined set of plugins installed into Hudson.

Monday, November 15, 2010

Horde continuous integration

With the increasing complexity of a huge codebase such as provided by Horde there is a significant need for quality control. As p@rdus accumulated a decent amount of experience with a variety of PHP QA tools during the past years it was only natural to apply this knowledge to the Horde codebase.

Many modules - including the Kolab_* packages - are now under constant surveillance of a continous integration server. The system is available for your inspection at ci.horde.org.

The service itself is run by Hudson - the standard CI server these days. The tools deployed on the server are a collection of the latest and greatest in the arena of quality control for PHP. PHPUnit certainly being the most important one as it delivers the framework for unit testing and code coverage analysis. In addition Hudson runs PEAR CodeSniffer (style check), PHP Depend (structural analysis), PHP Mess Detector, and the PHP Copy Paste Detector. On top of this the brand new PHP Code Browser integrates the analysis results into a view of the code.

With this system in place Horde is probably the only PHP based groupware client that can claim to validate its code base against current commercial grade software standards for PHP with each and every commit.

Monday, November 01, 2010

A new old PHP framework

With Horde4 the old "framework" block of code contained in the base Horde application will be split into its single component parts. This should help to underline the modular framework aspect of Horde itself.

The Kolab Server is one of the few systems that tried to use part of the Horde software stack even with Horde3. Both the free/busy system and the Resource management do not require more than a few packages from the Horde core framework.

With Horde3 this approach has always been somewhat awkward. The Horde release process made no use of the fact that the Horde framework was split into modules. With the Kolab Server we had to work around this limitation which resulted in the split packaging layout which will be available with Kolab-Server-2.3. But this is hand made on the Kolab side.

For Horde 4 p@rdus is pushing the use of PEAR with a specialized "component helper". The idea is to facilitate the handling of many small Horde components - each a PEAR package.

The tool currently allows automatic updates to Horde component manifests. It packages development snapshots. Builds continuous integration configuration for a component. It creates packaging specs for a distribution - and lists package dependencies.

Last but not least: It install components. And it does that primarily based on the code repository.

This is an essential feature for the continuous integration setup as all testing of the newest code will happen based on installed components. As a result the packaging will automatically be part of the quality control. Both the developers as well as the packagers will benefit from that.

Once the complete components setup is in place it will be very easy to derive a set of packages from it. Whether these packages are targeted for OpenPKG (Kolab Server 2.2.4 or 2.3) or some later native packaging will not really matter. p@rdus is of course actively working on such packaging and you can watch for commits on the horde4 branch of the Kolab server mercurial repository.

Monday, October 25, 2010

PHP Hudson tools

A while ago p@rdus published an easy to use toolset for PHP quality control purposes. Many PHP software tools left the good old hacky script era of PHP some years ago and nowadays quality control becomes increasingly important.

The toolset is primarily intended to provide a basis for the continuous integration setup of the upcoming Horde4 release. It helps to enforce commercial grade quality guidelines for the Horde4 code base and allows to publish this to the outside.

Since the Kolab Server bases a large part of it's functionality on PHP packages it is obvious that the toolset is likely to also play its role in the quality control of the server.

As the name implies the toolset is primarily oriented towards providing the standard PHP QA toolset for the Continuous Integration server Hudson.

The content of the package is a collection of PEAR based packages and an ant script for automated installation. Thus you can basically use the resulting toolset for any other purpose in the field of PHP QA, too.

The included software in detail:

The toolset hopefully helps to get you up and running with PHP QA in no time.