Bits from the Debian Continuous Integration project

December 11, 2015
Tags: ,

It’s been almost 2 years since the Debian Continuous Integration project has been launched, and it has proven to be a useful resource for the development of Debian.

I have previously made a an introductory post, and this this is an update on the latest developments.

Infrastructure upgrade

Back in early 2014 when Debian CI was launched, there were less than 200 source packages with declared test suite metadata, and using a single worker machine polling the archive for updates and running tests sequentially in an infinite loop (“the simplest thing that could possibly work”) was OK-ish.

Then our community started an incredible, slow and persistent effort to prepare source packages for automated testing, and we now have almost 5,000 of them. The original, over-simplistic design had to be replaced.

The effort of transforming debci in a distributed system was started by Martin Pitt, who did an huge amount of work. In the latest months I was able to complete that work, to a point where I am confident in letting it run (mostly) unatended. We also had lots of contributions to the web UI from Brandon Fairchild, who was a GSOC intern in 2014, and continues to contribute to this date.

All this work culminated in the migration from a single-worker model to a master/workers setup, currently with 10 worker nodes. On busy periods all of those worker nodes will go on for days with full utilization, but even then the turnaround between package upload and a test run is now a lot faster than it used to.

Debian members can inspect the resource usage on those systems, as well as the length of the processing queue, by browsing to the corresponding munin instance (requires authentication via a SSL client certificated issued by sso.debian.org).

The system is currenly being hosted on a Amazon EC2 account sponsored by Amazon.

The setup is fully automated and reproducible. It is not fully (or at all) documented yet, but those interested should feel free to get in touch on IRC (OFTC, #debci)

Testing backend changed from schroot to lxc

Together with the infrastructure updates, we also switched to using lxc instead of schroot as backend. Most test suites should not be affected by this, but the default lxc settings might cause some very specific issues in a few packages. See for example #806542 (“liblinux-prctl-perl: autopkgtest failures: seccomp, capbset”)

Adding support for KVM is also in the plans, and we will get to that at some point.

Learn more

If you want to learn more on how you can add tests for your package, a good first start is the debci online documentation (which is also available locally if you install `debci`).

You might also be interested in watching the live tutorial (WebM, 469 MB!) that has been presented at Debconf 15 earlier this year, full of tips and real examples from the archive. It would be awesome if someone wanted to transcribe that into a text tutorial ;-)

How to get involved

There are a few ways you can contribute:

autodep8. if you are knowledgeable on a subset of packages that are very similar and can have their tests executed in a similar way, such as “\$Language libraries”, you might consider writing a test metadata generator so that each package does not need to declare a debian/tests/control file explicitly, requiring only The `Testsuite:` header in debian/control.

Ruby and Perl are already covered, and there is initial support for NodeJS. Adding support for new types of packages is very easy. See the source repository.

If you manage to add support for your favorite language, please get in touch so we can discuss whitelisting the relavant packages in ci.debian.net so that they will get their tests executed even before being uploaded with the proper `Testsuite:` control field.

autopkgtest. autopkgtest is responsible for actually running your tests, and you can use it to reproduce test runs locally.

debci. debci is the system running in ci.debian.net (version 1.0, currently in testing, is exactly what is running up there, minus a version number and a changelog entry).

It can also be used to have private clones of ci.debian.net, e.g. for derivatives or internal Debian-related development. See for example the Ubuntu autopkgtest site.

Getting in touch

For maintainer queries and general discussion:

  • mailing list: debian-qa\@lists.debian.org
  • IRC: #debian-qa on OFTC. Feel free to highlight `terceiro`

For the development of debci/autopkgtest/autodep8

  • mailing list: autopkgtest-devel\@lists.alioth.debian.org
  • IRC: #debci on OFTC