Showing posts with label unit testing. Show all posts
Showing posts with label unit testing. Show all posts

Tuesday, June 21, 2011

Anatomy of a Horde test suite - III

Ready for the next item on the test suite agenda? This time the topic is Autoloading. We use a rather simple autoloading setup for most component test suites. It requires no additional setup and works out of the box if you run php AllTests.php.

That is already nice and allows running the complete test suite without further ado. But I must admit that I want more. My default work mode looks like this:

  • open test case (and modify it)
  • hit <f3> <f8> to run phpunit on this single test case
  • hit <f4> <f4> to to jump to the code line that produced the first error

This allows me to add a new test definition and immediately run it so that I can check for problems. And I don't need to execute the full AllTests.php to get feedback on the new test. So I'm annoyed every time I hit a test case that does not allow me to do that. A working autoloading setup is the key for that.

Luckily not only my own preferences make using an Autoload.php file in a test suite attractive. There are a number of reasons why such a file can be useful. The Wiki page for the Horde_Test component details them and this is a copy of the relevant section:


The Autoload.php file is not required in a test suite but it is strongly recommended that you use it. It's purpose is to setup PHP autoloading so that all tests in the test suite automatically have access to all the classes required for executing the tests. The reason why it is not mandatory is that Horde_Test_AllTests already loads a basic autoloading definition that works for most framework components.

This means that running php AllTests.php usually does not hit any autoloading problems. Running a single test case (e.g. phpunit Horde/Xyz/Unit/UnitTest.php) is a different matter though.

The *Test.php files do not extend Horde_Test_AllTests and thus there is nothing that would magically setup autoloading if you try to run such a test suite in isolation. And running single test cases can be quite convenient if the whole test suite would take a long time to execute. Using an Autoload.php file alongside the AllTests.php file is the recommended way to provide a single test case with autoloading and thus enable commands such as phpunit Horde/Xyz/Unit/UnitTest.php. In addition the file is helpful for any case where you need slightly more complex loading patterns or want to pull in special files manually.

Once you created an Autoload.php file for your test suite it will also be heeded by Horde/Test/AllTests.php. The latter will avoid the basic autoloading setup if it detects the presence of an Autoload.php file for the current test suite. That one will be loaded and is assumed to contain the required autoloading setup.

The content of Autoload.php

You should at least require the Autoload.php from Horde_Test in this file. This is also what Horde_Test_AllTests would do when choosing the simple autoloading setup.

require_once 'Horde/Test/Autoload.php';

It also makes sense to adapt the error reporting level to the same standards as required in the AllTests.php wrapper:

error_reporting(E_ALL | E_STRICT);

If you derive your test cases from a central test case definition you should load this one in Autoload.php as well:

/** Load the basic test definition */
require_once dirname(__FILE__) . '/TestCase.php';

Sometimes it makes sense to pull in the definition of test helpers that may be used throughout the test suite. They are usually not available via autoloading and need to be pulled in explicitely:

/** Load stub definitions */
require_once dirname(__FILE__) . '/Stub/ListQuery.php';
require_once dirname(__FILE__) . '/Stub/DataQuery.php';

Real world examples for Autoload.php helpers can be found in the Horde_Date and the Kolab_Storage components.

Within the test cases you only need to load the Autoload.php file which usually looks like this (and obviously depends on the position of the test case in the directory hierarchy of the test suite):

require_once dirname(__FILE__) . '/../Autoload.php';

You'll find additional background information on autoloading within test suite runs on the Wiki page for the Horde_Test component.

Tuesday, June 14, 2011

Anatomy of a Horde test suite - II

This morning I completed the next step on the journey through Horde's test suites and added the description of the AllTests.php file to the wiki page. I am not going to copy the complete text here but instead focus on the use cases for this file as I still have a few question to the audience below.


AllTests.php is the only mandatory requirement for a Horde test suite. Everything else is optional but there has to be an AllTests.php file which serves as an entry point into the test suite.

This is the functionality expected from the file:

  1. It must collect all tests of the test suite.
  2. It must allow to retrieve all tests of the suite via Horde_Xyz_AllTests::suite().
  3. It must allow running the test suite via phpunit AllTests.php.
  4. It must allow running the test suite via php AllTests.php.

The Horde_Test package already delivers a boilerplate AllTests.php class in framework/Test/lib/Horde/Test/AllTests.php and deriving an AllTests.php for a standard test suite becomes rather simple. The full code for this is presented on the wiki page and you can also look at an example from our repository.


Now I wonder if the items listed above are in fact all the requirements we have for this file.

Requirements (1) and (2) are obvious as this is functionality needed for our horde/framework/bin/test_framework helper that runs all framework tests. Though I assume nobody uses this one on a regular basis at the moment.

But I noticed that (3) does not work out of the box with the current PHPUnit. This led to a pull request as it definitely should (and can) work.

I usually run the tests with a rather long command line that ultimately boils down to phpunit Horde_Xyz_AllTests AllTests.php which is tied to a shortcut in Emacs. As the Lisp code I use for that extracts the class name automatically I never noticed that a plain phpunit AllTests.php does not work.

So are most people using php AllTests.php? How do you run the test suites or would like to run them? Can I get some feedback on this (either here, on IRC or via tweet)?

Anything additional I missed about the requirements for the AllTests.php file?

Next in the series will be on autoloading which should allow me to also look at the problems we still have with that in the application components.

Thursday, June 09, 2011

Anatomy of a Horde test suite - I

Just got issue 07/2011 of the the German Linux Magazine in the mailbox and on the final page there is this little abstract about 08/2011 saying...

"PHP Unit and Jenkins - There are two things guarding against programming errors: unit tests covering your code and continuous integration systems that automate the testing. The next issue will demonstrate this based on a real example from a PHP web project." [translated from German].

The "PHP web project" is actually named "Horde" and hm... I guess this means I have to write this thing - ;) . When agreeing to the article I immediately knew I wanted to combine it with an overview on how the Horde test suites are arranged. So far we have been lacking a summary in that area and it should help newcomers to the Horde project to get into testing mode as well.

My mind is currently fully tuned to unit testing and code quality and it is amazing how easy it is to write about this. The initial draft for the article already exceeded all limits when it comes to size. Though I got pretty positive feedback on it I will have to leave some stuff out. Those sections should make it to this blog instead so that I can link to it in the article.

Basically I will make this into a short series of blog entries on unit testing in Horde. I will include parts of the Horde_Test overview, personal musings, and stuff related to the article. Let's hope it is useful to some people out there.

Here we go with the introduction to the Horde_Test overview...


Introduction

The Horde Project has always had high standards when it comes to code quality. Of course these standards evolved with time and also with the progress the PHP community made. The code from IMP-1.0.0 (1998) didn't come with unit tests. And somehow it lacked classes. And there is an awful lot of code mixed with HTML. Somehow this looks horribly like PHP3.

Oh, it was PHP3.

Of course PHP development changed over time and so did the Horde project. Nowadays each and every commit into our repository leads to the automatic execution of thousands of unit tests written by the Horde developers and they check our code for failures. Night and day our continuous integration server broadcasts the current test status to us in particular but also to anyone else interested.

With the release of Horde 4 the test suites of the Horde components available via our PEAR server all show some common patterns. There are certain Do's and Don'ts and a lot of playground in between. Often the Horde_Test component is involved. So it makes sense to associate the overview on the anatomy of Horde test suites with this particular module.


I must admit that I really like the way the Horde project approaches unit testing. There is no way we could be unit test purists which would be too extreme given the fact that the project already exists for more than a decade. But at the same time there was also no one complaining when testing entered the equation. It just felt like continuing to adhere to the quality standards that seem so familiar when it comes to Horde code.

So much for now. More technical stuff to follow soon...

Tuesday, May 17, 2011

Is Horde PHP?

LinuxTag was really nice for the Horde project because of all that positive feedback to our recent release and the valuable suggestions for stuff that would be nice to have in Horde. Might be worth another blog post. Here I just wanted to log one of the funnier conversations I had ...

"Is Horde PHP?"

"Yes."

"Do you use unit tests?"

"Absolutely. We got about 3000 of those."

"Oh, great. Mind writing an article about PHP Unit testing in Linux Magazine?"

"Deal!"

Monday, October 25, 2010

PHP Hudson tools

A while ago p@rdus published an easy to use toolset for PHP quality control purposes. Many PHP software tools left the good old hacky script era of PHP some years ago and nowadays quality control becomes increasingly important.

The toolset is primarily intended to provide a basis for the continuous integration setup of the upcoming Horde4 release. It helps to enforce commercial grade quality guidelines for the Horde4 code base and allows to publish this to the outside.

Since the Kolab Server bases a large part of it's functionality on PHP packages it is obvious that the toolset is likely to also play its role in the quality control of the server.

As the name implies the toolset is primarily oriented towards providing the standard PHP QA toolset for the Continuous Integration server Hudson.

The content of the package is a collection of PEAR based packages and an ant script for automated installation. Thus you can basically use the resulting toolset for any other purpose in the field of PHP QA, too.

The included software in detail:

The toolset hopefully helps to get you up and running with PHP QA in no time.

Wednesday, July 09, 2008

Generating a PEAR test environment for releasing PEAR packages

As a developer you will often have a high number of different libraries installed. This will usually not match the users situation that will be nearer to the minimal number of required libraries. This may turn out to be a problem when releasing packages as missing dependencies can be easily overlooked.

For releasing PHP PEAR packages it makes sense to get a separate PEAR environment for testing if the package to be release has a correct set of dependencies. If PEAR is already installed on the system, the steps for that are straight forward.

Setup the new repository

Create the test environment with

# mkdir ~/pear-test

A separate PEAR configuration will be needed there:

# pear config-create ~/pear-test ~/pear-test/.pearrc

CONFIGURATION (CHANNEL PEAR.PHP.NET):
=====================================
...

All that is required to complete the test environment is the installation of PEAR itself:

# pear -c ~/pear-test/.pearrc install -o PEAR

WARNING: channel "pear.php.net" has updated its protocols, use "channel-update pear.php.net" to update
Did not download optional dependencies: pear/XML_RPC, use --alldeps to download automatically
pear/PEAR can optionally use package "pear/XML_RPC" (version >= 1.4.0)
downloading PEAR-1.7.2.tgz ...
Starting to download PEAR-1.7.2.tgz (302,744 bytes)
..........................done: 302,744 bytes
downloading Archive_Tar-1.3.2.tgz ...
Starting to download Archive_Tar-1.3.2.tgz (17,150 bytes)
...done: 17,150 bytes
downloading Structures_Graph-1.0.2.tgz ...
Starting to download Structures_Graph-1.0.2.tgz (30,947 bytes)
...done: 30,947 bytes
downloading Console_Getopt-1.2.3.tgz ...
Starting to download Console_Getopt-1.2.3.tgz (4,011 bytes)
...done: 4,011 bytes
install ok: channel://pear.php.net/Archive_Tar-1.3.2
install ok: channel://pear.php.net/Structures_Graph-1.0.2
install ok: channel://pear.php.net/Console_Getopt-1.2.3
install ok: channel://pear.php.net/PEAR-1.7.2
PEAR: Optional feature webinstaller available (PEAR's web-based installer)
PEAR: Optional feature gtkinstaller available (PEAR's PHP-GTK-based installer)
PEAR: Optional feature gtk2installer available (PEAR's PHP-GTK2-based installer)
PEAR: To install optional features use "pear install pear/PEAR#featurename"

A real example

As I am currently working on releasing the Kolab modules in Horde as PEAR packages I'll provide this process as an example.

In order to wrap a PEAR package the appropriate channel needs to be known for pear. For horde packages this is pear.horde.org. This channel needs to be discovered first:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc channel-discover pear.horde.org
Adding Channel "pear.horde.org" succeeded
Discovery of channel "pear.horde.org" succeeded

The initial package to be released will be Kolab_Format. PEAR packaging happens with

# cd Kolab_Format
# ~/pear-test/pear/pear -c ~/pear-test/.pearrc package package.xml

Analyzing lib/Horde/Kolab/Format/XML/contact.php
Analyzing lib/Horde/Kolab/Format/XML/distributionlist.php
Analyzing lib/Horde/Kolab/Format/XML/event.php
Analyzing lib/Horde/Kolab/Format/XML/hprefs.php
Analyzing lib/Horde/Kolab/Format/XML/note.php
Analyzing lib/Horde/Kolab/Format/XML/task.php
Analyzing lib/Horde/Kolab/Format/Date.php
Analyzing lib/Horde/Kolab/Format/XML.php
Analyzing lib/Horde/Kolab/Format.php
Package Horde_Kolab_Format-0.9.0.tgz done
Tag the released code with `pear cvstag package.xml'
(or set the CVS tag RELEASE_0_9_0 by hand)

As most Horde PEAR packages have not yet been marked stable PEAR will still refuse to install the new package:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install Horde_Kolab_Format-0.9.0.tgz 

Failed to download horde/Horde_DOM within preferred state "stable", latest release is version 0.1.0, stability "alpha", use "channel://pear.horde.org/Horde_DOM-0.1.0" to install
Failed to download horde/Horde_NLS within preferred state "stable", latest release is version 0.0.2, stability "alpha", use "channel://pear.horde.org/Horde_NLS-0.0.2" to install
Failed to download horde/Horde_Util within preferred state "stable", latest release is version 0.0.2, stability "alpha", use "channel://pear.horde.org/Horde_Util-0.0.2" to install
Did not download optional dependencies: horde/Horde_Prefs, use --alldeps to download automatically
horde/Horde_Kolab_Format requires package "horde/Horde_DOM" (version >= 0.1.0)
horde/Horde_Kolab_Format requires package "horde/Horde_NLS"
horde/Horde_Kolab_Format requires package "horde/Horde_Util"
horde/Horde_Kolab_Format can optionally use package "horde/Horde_Prefs"
No valid packages found
install failed

The required packages must be installed manually first:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install channel://pear.horde.org/Horde_DOM-0.1.0

downloading Horde_DOM-0.1.0.tgz ...
Starting to download Horde_DOM-0.1.0.tgz (4,256 bytes)
.....done: 4,256 bytes
install ok: channel://pear.horde.org/Horde_DOM-0.1.0

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install channel://pear.horde.org/Horde_Util-0.0.2

Did not download optional dependencies: horde/Horde_Browser, use --alldeps to download automatically
horde/Horde_Util can optionally use package "horde/Horde_Browser"
downloading Horde_Util-0.0.2.tgz ...
Starting to download Horde_Util-0.0.2.tgz (16,603 bytes)
......done: 16,603 bytes
install ok: channel://pear.horde.org/Horde_Util-0.0.2

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install channel://pear.horde.org/Horde_NLS-0.0.2

downloading Horde_NLS-0.0.2.tgz ...
Starting to download Horde_NLS-0.0.2.tgz (75,779 bytes)
.................done: 75,779 bytes
install ok: channel://pear.horde.org/Horde_NLS-0.0.2

This time installation should suceed:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc  install Horde_Kolab_Format-0.9.0.tgz 

Did not download optional dependencies: horde/Horde_Prefs, use --alldeps to download automatically
horde/Horde_Kolab_Format can optionally use package "horde/Horde_Prefs"
install ok: channel://pear.horde.org/Horde_Kolab_Format-0.9.0

All that is missing now is to tell PEAR that it should provide PHP only with one include directory: the one we just setup. Ensure that you replace USER with the name of the current user.

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc config-set php_bin "`~/pear-test/pear/pear -c ~/pear-test/.pearrc config-get php_bin` -d include_path=/home/USER/pear-test/pear/php"

After installation succeeded the unit tests should be run in order to validate the dependencies. If the tests are based on PHPUnit then this tool will have to be installed first:

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc channel-discover pear.phpunit.de

Adding Channel "pear.phpunit.de" succeeded
Discovery of channel "pear.phpunit.de" succeeded

# ~/pear-test/pear/pear -c ~/pear-test/.pearrc install phpunit/PHPUnit

And now we can finally run the last check before releasing the PEAR package:

# cd ~/pear-test/pear/tests/Horde_Kolab_Format/Horde/Kolab/Format/
# ~/pear-test/pear/pear -c ~/pear-test/.pearrc run-tests -u 

Thursday, May 15, 2008

A first positive experience with ruby: Patching puppet

So far I didn't have much experience with ruby. The few lines of code I've written in that language reminded me too much of perl. And I'm not really a fan of the perl syntax. But today ruby managed to convince me in the area of unit testing.

The problem

I'm bound to stick to ruby as I decided that ruby-based puppet will provide a central element of the next Kolab2/Gentoo version. While it provides some nice LDAP integration features these are not quite sufficient for Kolab. Puppet can grab some host parameters from LDAP and integrate these into the host configuration. The problem for Kolab2/Gentoo is the limitation to some LDAP parameters. Actually these have to be real LDAP attributes that have been defined in a schema.

As I have already argued on the Kolab mailing list it does not make much sense to define attributes in a schema if you want to use such parameters for configuration of a large set of possible applications (postfic, openldap, cyrus, ...). In this case it makes more sense to use the approach also used by the Horde LDAP schema: specifying a single attribute that uses a string value to specify parameters with arbitrary names. E.g. ldapAttribute:"one=two" in order to define parameter "one". Only the "ldapAttribute" will have to be defined in a schema while the code using this parameter handles converting the string into the final paramter.

I wrote a short patch for puppet to implement this. After a short while I got a positive response but the patch was considered insufficient as it lacked any tests.

A simple solution

I admit I was slightly worried because learning to handle just another test framework in a language I have nearly no clue about was something I did not fancy at all. And that was the first really positive surprise about ruby: Using the test framework and successfully writing unit tests in it was a matter of half an hour. Even though it required mocking the LDAP connection.

The testing allowed me to reconsider my expectations concerning the patch and to fix a problem of my initial version. I submitted the new version shortly afterwards and hope it will find its way into the repository now.

Well done, ruby. Let me see what else you can do in order to convince me that you are indeed a good thing...