When we discovered the successful attack on ftp.horde.org two weeks ago we were of course frantic to determine which packages had been affected in addition to the one Horde 3 archive Jan identified as modified initially.
For the Horde 4 packages we had no hashes to verify the file integrity though. While PEAR supports signing of packages via GPG that seems to be a feature which is virtually unused. For one thing not that many PHP based projects use PEAR packaging and in addition there is no way to automatically verify package integrity on the user side when installing via PEAR. So we also didn't consider signing our packages when switching to installing Horde via PEAR.
Obviously you gain a different perspective on that issue once a hacker implanted a backdoor in some of your packages. Of course we invested a lot of time into securing our infrastructure now to ensure that such an event never happens again. On our side the file integrity is constantly monitored now. But we will also have to investigate how we can improve the PEAR based installation procedure so that it also allows for the required amount of security on the user side.
But if we had no hashes how did we ensure the Horde 4 packages were indeed unmodified? Git to the rescue! As we tag all our releases it was a matter of creating a short script to automatically compare the current state of the packages on our PEAR server against the state we had within git.
Without further ado - here is the script I used:
#!/bin/bash
git reset --hard HEAD
git clean -f -d
STAMP=`date +%y%m%d-%H%M`
mkdir ../diffs-$STAMP
mkdir -p ../validate-$STAMP/pear.horde.org
mkdir -p ../validate-$STAMP/rebuild
for package in `cat ../pear-recovery-packages.txt | grep -v ".tar$"`
do
TAG=${package/.tgz/}
TAG=${TAG,,}
PPATH=${package/-*/}
if [ "x${PPATH/Horde_*/}" == "x" ]; then
PPATH=framework/${PPATH/Horde_};
fi
if [ "x${PPATH/groupware*/}" == "x" ]; then
PPATH=bundles/$PPATH;
fi
if [ "x${PPATH/webmail*/}" == "x" ]; then
PPATH=bundles/$PPATH;
fi
PRESENT=`git tag -l $TAG`
if [ "x$PRESENT" == "x" ]; then
echo
echo "======================================================================"
echo "Tag $TAG for package $package is missing!"
echo "======================================================================"
echo
echo "$package: TAG MISSING" >> ../status-$STAMP
else
rm *.tgz
rm -rf ../validate-$STAMP/pear.horde.org/*
rm -rf ../validate-$STAMP/rebuild/*
GIT=`git checkout $TAG`
horde-components -z $PPATH --keep-version
if [ -e $package ]; then
cp *.tgz ../validate-$STAMP/pear.horde.org/
cp ../pear.horde.org/get/$package ../validate-$STAMP/rebuild/
tar -C ../validate-$STAMP/pear.horde.org/ -x -z -f ../validate-$STAMP/pear.horde.org/*.tgz
tar -C ../validate-$STAMP/rebuild/ -x -z -f ../validate-$STAMP/rebuild/*.tgz
DIFF=`diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/}`
if [ "x$DIFF" != "x" ]; then
echo
echo "======================================================================"
echo "Diff for package $package detected!"
diff -Naur ../validate-$STAMP/pear.horde.org/${package/.tgz/} ../validate-$STAMP/rebuild/${package/.tgz/} > ..$
echo "======================================================================"
echo
echo "$package: DIFF" >> ../status-$STAMP
else
echo
echo "======================================================================"
echo "$package CLEAN!!!"
echo "======================================================================"
echo
echo "$package: CLEAN" >> ../status-$STAMP
fi
else
echo
echo "======================================================================"
echo "Failed rebuilding package $package!"
echo "======================================================================"
echo
echo "$package: FAILED REBUILDING" >> ../status-$STAMP
fi
fi
done
The script walks through the list of packages we had on the PEAR
server, moves back in time within our git repository to the
appropriate tag, rebundles the package, extracts both the old and the
new package and compares them using diff.
The resulting status list looked quite okay. There were a few release glitches where the tag was not on the exact commit that was used for building the package. In that case usually a small diff resulted. The list was rechecked manually for any malicious traces - Horde_Imap_Client-1.4.3 was the only one were the diff was large but that turned out to be a result from a mishap while releasing that package version. A few times the diff led to a problem with rebuilding the package (indicated by "FAILED REBUILDING"). The package.xml was broken in those cases and needed a fix that was only committed after the tagging. Here I compared the git state directly against the extracted package. There was only a single case ("Horde_ActiveSync-1.1.11") where the tag was missing which required manually identifying the corresponding commit to verify that state against the old package contents.
Once the script was established the analysis ran for about two hours after which we were at least somewhat relieved. Having a backdoor in some Horde 3 packages was already bad enough - having that in Horde 4 would have been even more disastrous.
