p@rdus: May 2008

Puppet is a tool for managing your system configuration. It provides a complete language for expressing and realizing system settings. After some introductory words this post will focus on a Gentoo specific puppet module for managing package installations.

If you have no clue about puppet you might wish to read the introduction if you are interested in managing the configurations of your system in an efficient way. The discussion about the gentoo specific module will only be of interest to you if you already know the basics of writing puppet modules.

Introduction

What are the advantages of using puppet rather than editing all files in /etc by hand?

Using puppet means you create a repository of your configuration knowledge
You can replicate all of or part of the settings to another host
In addition you can version control and share your knowledge in a repository

Mind you: If you are only managing a single host you might not find much value in the items listed above. Indeed puppet only becomes useful if you really wish to apply a complex configuration over many hosts.

But of course this is true for any groupware server and in particular the Kolab Server. Porting Kolab to Gentoo is a project I have been working on for more than three years now.

The initial version (Kolab2/Gentoo-2.1) failed to make me really happy. One central reason for that has been the configuration tool provided by Kolab. While it works fine for the original version of the Kolab Server it simply fails to cope with the amount of options users have on Gentoo.

I always wanted to merge my own crappy tool for configuration management with the code from the Kolab Server. But a kind anonymous voice answered to the blog post linked in the previous setence, telling me that this is a stupid idea and I should use puppet. He was right.

So I'm establishing the Kolab2/Gentoo groupware server configuration based on puppet at the moment. As this includes generating some Gentoo specific modules for puppet it is now time to stop the introductory words and get down to some puppet code.

Installing packages for generic distributions

In order to tell puppet that you wish to have a single package installed you would use a construct like this:

package { openldap:
  ensure   => 'latest',
}

This works fine on most distributions but on Gentoo you might ask about support for use flags, keywords and masking.

Installing packages on Gentoo

My solution is the puppet module os_gentoo.

This module is mainly concerned with management of the files/directories you find at /etc/portage/package.* in your Gentoo system. In order for puppet to manage these paths it makes sense to convert these into directories.

The module provides four central parts:

Backup of the original contents of /etc/portage/package.* if these were files.
Converting the paths into directories.
Restoring the original file contents as /etc/portage/package.*/package.*.original.
Providing functions to easily manage use flags, keywords and masking for other packages.

Backup of /etc/portage/package.*

If the user managed /etc/portage/package.* as files we need to grab the content and store it. Puppet provides the file() function for that but that function will fail if it sees a directory. So we need to determine if the path already is a directory. We need to write some ruby code at this point and create a new fact:

# Determine if these are regular files
 
package_use = '/etc/portage/package.use'
 
Facter.add('use_isfile') do
  setcode do
    if FileTest.file?(package_use)
      true
    else
      false
    end
  end
end

...

Facts are little pieces of system information that puppet determines automatically using the tool dev-ruby/facter. The code given above checks if /etc/portage/package.use is a file and places that information in the variable use_isfile. We will shortly meet that variable again.

This fact is something we store as a plugin at os_gentoo/plugins/facter/portage_dirs.rb within the module.

The code actually performing the backup is packaged in a puppet class:

# Class gentoo::etc::portage::backup
#
# Stores user settings in the /etc/portage/package.* files.
#
# @author Gunnar Wrobel 
# @version 1.0
# @package os_gentoo
#
class gentoo::etc::portage::backup
{
  if $use_isfile {
    $use = file('/etc/portage/package.use')
  } else {
    $use = false
  }
  if $keywords_isfile {
    $keywords = file('/etc/portage/package.keywords')
  } else {
    $keywords = false
  }
  if $mask_isfile {
    $mask = file('/etc/portage/package.mask')
  } else {
    $mask = false
  }
  if $unmask_isfile {
    $unmask = file('/etc/portage/package.unmask')
  } else {
    $unmask = false
  }
}

Here we meet the variables again. In case $use_isfile is true the file contents will be parsed into $use. Otherwise the variable is set to false. We return to our backup two sections further down.

Converting /etc/portage/package.* into directories

Now that we have saved the file contents we can safely convert the files into directories. Puppet would not destroy the original files but instead store them in an archive. But recovering them from there would be cumbersome for the user. Automating the conversion seems to be a better solution.

Requiring a path to be a directory is easy in puppet:

# Class gentoo::etc::portage
#
# Ensure that all /etc/portage/package.* locations are actually
# handled as directories. This allows to easily manage the package
# specific settings for Gentoo.
#
# @author Gunnar Wrobel 
# @version 1.0
# @package os_gentoo
#
class gentoo::etc::portage
{
  # Check that we are able to handle /etc/portage/package.* as
  # directories
 
  file { 'package.use::directory':
    path => '/etc/portage/package.use',
    ensure => 'directory',
    tag => 'buildhost'
  }
 
  file { 'package.keywords::directory':
    path => '/etc/portage/package.keywords',
    ensure => 'directory',
    tag => 'buildhost'
  }
 
  file { 'package.mask::directory':
    path => '/etc/portage/package.mask',
    ensure => 'directory',
    tag => 'buildhost'
  }
 
  file { 'package.unmask::directory':
    path => '/etc/portage/package.unmask',
    ensure => 'directory',
    tag => 'buildhost'
  }
}

Again the four actions have been packaged into a single puppet class. The different actions all have a buildhost tag. This is only required if you really use a build host structure with your servers and plays no role otherwise.

Restoring the original /etc/portage/package.*

Now that puppet converted /etc/portage/package.* to directories we lost the original file contents. Another class will rescue them:

# Class gentoo::etc::portage::restore
#
# Restores user settings from the /etc/portage/package.* files.
#
# @author Gunnar Wrobel 
# @version 1.0
# @package os_gentoo
#
class gentoo::etc::portage::restore
{
  if $gentoo::etc::portage::backup::use {
    file { '/etc/portage/package.use/package.use.original':
      content => $gentoo::etc::portage::backup::use,
      tag => 'buildhost'
    }
  }
  if $gentoo::etc::portage::backup::keywords {
    file { '/etc/portage/package.keywords/package.keywords.original':
      content => $gentoo::etc::portage::backup::keywords,
      tag => 'buildhost'
    }
  }
  if $gentoo::etc::portage::backup::mask {
    file { '/etc/portage/package.mask/package.mask.original':
      content => $gentoo::etc::portage::backup::mask,
      tag => 'buildhost'
    }
  }
  if $gentoo::etc::portage::backup::unmask {
    file { '/etc/portage/package.unmask/package.unmask.original':
      content => $gentoo::etc::portage::backup::unmask,
      tag => 'buildhost'
    }
  }
}

For each of the four paths the original backup variable (e.g. $gentoo::etc::portage::backup::use) is checked for content. We need to use the full class path here to access the variable content. If it contains content it will be written to the corresponding new path (e.g. /etc/portage/package.use/package.use.original).

Handling /etc/portage/package.* with puppet

Now the management of /etc/portage/package.* becomes easy as puppet can place new files for every package or set of packages that requires special use flags, keywords or masking.

This is an example for the use flags:

# Function gentoo_use_flags
#
# Specify use flags for a package.
#
# @param context A unique context for the package
# @param package The package atom
# @param use The use flags to apply
#
define gentoo_use_flags ($context = '',
                         $package = '',
                         $use = '')
{
 
  file { "/etc/portage/package.use/${context}":
    content => "$package $use",
    require => File['package.use::directory'],
    tag => 'buildhost'
  }
 
}

The function takes a context which must be unique and will be used as path component. In addition the package atom needs to be specified including the use flags to be set. Puppet will then create a new file within /etc/portage/package.use using the file type (This is something different than the file function mentioned above).

The only new thing here is the require argument that specifies that puppet must ensure that the file operation with the name package.use::directory has been executed before creating this new file. In other words we ensure that /etc/portage/package.use is indeed a directory.

Managing package installations on Gentoo

Taking all these definitions together we can now express a package installation in the following way:

# Package installation
  case $operatingsystem {
    gentoo:
    {
      gentoo_unmask { openldap:
        context => 'service_openldap',
        package => '=net-nds/openldap-2.4.7',
        tag => 'buildhost'
      }
      gentoo_keywords { openldap:
        context => 'service_openldap',
        package => '=net-nds/openldap-2.4.7',
        keywords => "~$keyword",
        tag => 'buildhost'
      }
      gentoo_use_flags { openldap:
        context => 'service_openldap',
        package => 'net-nds/openldap',
        use => 'berkdb crypt overlays perl ssl syslog -sasl',
        tag => 'buildhost'
      }
      package { openldap:
        category => 'net-nds',
        ensure => 'latest',
        require => [ Gentoo_unmask['openldap'],
                       Gentoo_keywords['openldap'],
                       Gentoo_use_flags['openldap'] ],
        tag => 'buildhost'
      }
    }
    default:
    {
      package { openldap:
        ensure => 'installed',
      }
    }
  }
}

The example installs the experimental net-nds/openldap-2.4.7 package. We differentiate between Gentoo and other distributions using the $operatingsystem variable automatically provided by puppet.

Of course the Gentoo installation looks much more complex than the standard installation on other systems but we have a lot more flexibility on Gentoo. And the idea of the module is to allow us to use this flexibility within puppet.

The first three sections (gentoo_unmask,gentoo_keywords, and gentoo_use_flags) handle the settings in /etc/portage/package.* and the actual installation happens in the fourth section. We use the standard package type here but require that all the settings in /etc/portage/package.* have been executed before puppet runs emerge

A final note on the variable $keyword that is being used in the section above. This is another fact that prevents us from specifying keywords like ~x86 while we actually want ~amd64. It simply reads ACCEPT_KEYWORDS and assumes that the user has the stable keyword selected there. This probably still needs fixing.

Conclusion

It is not too difficult to map the full power of package installations on Gentoo into the puppet way of installing packages. I'm pretty certain that some of the methods I implemented in os_gentoo are still bound to evolve and do not yet represent the best way of handling installations on Gentoo. The module does for example not solve any of the issues mentioned on the Gentoo page in the puppet wiki. So there is still work to be done.

But for now I'm happy to have the central aspects of use flags, keywords and masking available within puppet.

So far I didn't have much experience with ruby. The few lines of code I've written in that language reminded me too much of perl. And I'm not really a fan of the perl syntax. But today ruby managed to convince me in the area of unit testing.

The problem

I'm bound to stick to ruby as I decided that ruby-based puppet will provide a central element of the next Kolab2/Gentoo version. While it provides some nice LDAP integration features these are not quite sufficient for Kolab. Puppet can grab some host parameters from LDAP and integrate these into the host configuration. The problem for Kolab2/Gentoo is the limitation to some LDAP parameters. Actually these have to be real LDAP attributes that have been defined in a schema.

As I have already argued on the Kolab mailing list it does not make much sense to define attributes in a schema if you want to use such parameters for configuration of a large set of possible applications (postfic, openldap, cyrus, ...). In this case it makes more sense to use the approach also used by the Horde LDAP schema: specifying a single attribute that uses a string value to specify parameters with arbitrary names. E.g. ldapAttribute:"one=two" in order to define parameter "one". Only the "ldapAttribute" will have to be defined in a schema while the code using this parameter handles converting the string into the final paramter.

I wrote a short patch for puppet to implement this. After a short while I got a positive response but the patch was considered insufficient as it lacked any tests.

A simple solution

I admit I was slightly worried because learning to handle just another test framework in a language I have nearly no clue about was something I did not fancy at all. And that was the first really positive surprise about ruby: Using the test framework and successfully writing unit tests in it was a matter of half an hour. Even though it required mocking the LDAP connection.

The testing allowed me to reconsider my expectations concerning the patch and to fix a problem of my initial version. I submitted the new version shortly afterwards and hope it will find its way into the repository now.

Well done, ruby. Let me see what else you can do in order to convince me that you are indeed a good thing...

Sunday, May 18, 2008

Using puppet on Gentoo