Past halfway there: history of autopkgtest support in Debian

The Release of Debian 13 ("Trixie") last month marked another milestone on the effort to provide automated test support for Debian packages in their installed form. We have achieved the mark of 57% of the source packages in the archive declaring support for autopkgtest.

Release	Packages with tests	Total number of packages	% of packages with tests
wheezy	5	17175	0%
jessie	1112	20596	5%
stretch	5110	24845	20%
buster	9966	28501	34%
bullseye	13949	30943	45%
bookworm	17868	34341	52%
trixie	21527	37670	57%

The code that generated this table is provided at the bottom.

The growth rate has been consistently decreasing at each release after stretch. That probably means that the low hanging fruit -- adding support en masse for large numbers of similar packages, such as team-maintained packages for a given programming language -- has been picked, and from now on the work gets slightly harder. Perhaps there is a significant long tail of packages that will never get autopkgtest support.

Looking for common prefixes among the packages missing a Testsuite: field gives me us the largest groups of packages missing autopkgtest support:

$ grep-dctrl -v -F Testsuite --regex -s Package -n . trixie | cut -d - -f 1 | uniq -c | sort -n| tail -20
     50 apertium
     50 kodi
     51 lomiri
     53 maven
     55 libjs
     57 globus
     66 cl
     67 pd
     72 lua
     79 php
     88 puppet
     91 r
    111 gnome
    124 ruby
    140 ocaml
    152 rust
    178 golang
    341 fonts
    557 python
   1072 haskell

There seems to be a fair amount of Haskell and Python. If someone could figure out a way of testing installed fonts in a meaningful way, this would a be a good niche where we can cover 300+ packages.

There is a another analysis that can be made, which I didn't: which percentage of new packages introduced in a given release have declared autopkgtest support, compared with the total of new packages in that release? My data only counts the totals, so we start with the technical debt of the almost all of the 17,000 packages with no tests in wheezy, which was the stable at the time I started Debian CI. How many of those got tests since then?

Note that not supporting autopkgtest does not mean that a package is not tested at all: it can run build-time tests, which are also useful. Not supporting autopkgtest, though, means that their binaries in the archive can't be automatically tested in their installed, but then there is a entire horde of volunteers running testing and unstable on a daily basis who test Debian and report bugs.

This is the script that produced the table in the beginning of this post:

#!/bin/sh

set -eu

extract() {
  local release
  local url
  release="$1"
  url="$2"

  if [ ! -f "${release}" ]; then
    rm -f "${release}.gz"
    curl --silent -o ${release}.gz "${url}"
    gunzip "${release}.gz"
  fi

  local with_tests
  local total
  with_tests="$(grep-dctrl -c -F Testsuite --regex . $release)"
  total="$(grep-dctrl -c -F Package --regex . $release)"

  echo "| ${release} | ${with_tests} | ${total} | $((100*with_tests/total))% |"
}

echo "| **Release** | **Packages with tests** | **Total number of packages** | **% of packages with tests** |"
echo "|-------------|-------------------------|------------------------------|------------------------------|"
for release in wheezy jessie stretch buster; do
  extract "${release}" "http://archive.debian.org/debian/dists/${release}/main/source/Sources.gz"
done
for release in bullseye bookworm trixie; do
  extract "${release}" "http://ftp.br.debian.org/debian/dists/${release}/main/source/Sources.gz"
done