Recap of the verification that there was no backdoor in the Horde 4 packages

When we discovered the successful attack on ftp.horde.org two weeks ago we were of course frantic to determine which packages had been affected in addition to the one Horde 3 archive Jan identified as modified initially.

For the Horde 4 packages we had no hashes to verify the file integrity though. While PEAR supports signing of packages via GPG that seems to be a feature which is virtually unused. For one thing not that many PHP based projects use PEAR packaging and in addition there is no way to automatically verify package integrity on the user side when installing via PEAR. So we also didn't consider signing our packages when switching to installing Horde via PEAR.

Obviously you gain a different perspective on that issue once a hacker implanted a backdoor in some of your packages. Of course we invested a lot of time into securing our infrastructure now to ensure that such an event never happens again. On our side the file integrity is constantly monitored now. But we will also have to investigate how we can improve the PEAR based installation procedure so that it also allows for the required amount of security on the user side.

But if we had no hashes how did we ensure the Horde 4 packages were indeed unmodified? Git to the rescue! As we tag all our releases it was a matter of creating a short script to automatically compare the current state of the packages on our PEAR server against the state we had within git.

Without further ado - here is the script I used:

#!/bin/bash

git reset --hard HEAD
git clean -f -d

STAMP=`date +%y%m%d-%H%M`
mkdir ../diffs-$STAMP
mkdir -p ../validate-$STAMP/pear.horde.org
mkdir -p ../validate-$STAMP/rebuild

for package in `cat ../pear-recovery-packages.txt | grep -v ".tar$"`
do
  TAG=${package/.tgz/}
  TAG=${TAG,,}
  PPATH=${package/-*/}
  if [ "x${PPATH/Horde_*/}" == "x" ]; then
      PPATH=framework/${PPATH/Horde_};
  fi
  if [ "x${PPATH/groupware*/}" == "x" ]; then
      PPATH=bundles/$PPATH;
  fi
  if [ "x${PPATH/webmail*/}" == "x" ]; then
      PPATH=bundles/$PPATH;
  fi
  PRESENT=`git tag -l $TAG`
  if [ "x$PRESENT" == "x" ]; then
      echo
      echo "======================================================================"
      echo "Tag $TAG for package $package is missing!"
      echo "======================================================================"
      echo
      echo "$package: TAG MISSING" >> ../status-$STAMP
  else
      rm *.tgz                                                                                                               
      rm -rf ../validate-$STAMP/pear.horde.org/*
      rm -rf ../validate-$STAMP/rebuild/*
      GIT=`git checkout $TAG`
      horde-components -z $PPATH --keep-version
      if [ -e $package ]; then
          cp *.tgz ../validate-$STAMP/pear.horde.org/
          cp ../pear.horde.org/get/$package ../validate-$STAMP/rebuild/
          tar -C ../validate-$STAMP/pear.horde.org/ -x -z -f ../validate-$STAMP/pear.horde.org/*.tgz
          tar -C ../validate-$STAMP/rebuild/ -x -z -f ../validate-$STAMP/rebuild/*.tgz
          DIFF=`diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/}`
          if [ "x$DIFF" != "x" ]; then
              echo
              echo "======================================================================"
              echo "Diff for package $package detected!"
              diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/} > ..$
              echo "======================================================================"
              echo
              echo "$package: DIFF" >> ../status-$STAMP
          else
              echo
              echo "======================================================================"
              echo "$package CLEAN!!!"
              echo "======================================================================"
              echo
              echo "$package: CLEAN" >> ../status-$STAMP
          fi
      else
          echo
          echo "======================================================================"
          echo "Failed rebuilding package $package!"
          echo "======================================================================"
          echo
          echo "$package: FAILED REBUILDING" >> ../status-$STAMP
      fi
  fi
done

The script walks through the list of packages we had on the PEAR
server, moves back in time within our git repository to the
appropriate tag, rebundles the package, extracts both the old and the
new package and compares them using diff.

The resulting status list looked quite okay. There were a few release glitches where the tag was not on the exact commit that was used for building the package. In that case usually a small diff resulted. The list was rechecked manually for any malicious traces - Horde_Imap_Client-1.4.3 was the only one were the diff was large but that turned out to be a result from a mishap while releasing that package version. A few times the diff led to a problem with rebuilding the package (indicated by "FAILED REBUILDING"). The package.xml was broken in those cases and needed a fix that was only committed after the tagging. Here I compared the git state directly against the extracted package. There was only a single case ("Horde_ActiveSync-1.1.11") where the tag was missing which required manually identifying the corresponding commit to verify that state against the old package contents.

Once the script was established the analysis ran for about two hours after which we were at least somewhat relieved. Having a backdoor in some Horde 3 packages was already bad enough - having that in Horde 4 would have been even more disastrous.