https://terceiro.xyz/Antonio Terceiro2023-12-28T15:00:00ZAntonio Terceirohttps://terceiro.xyztag:terceiro.xyz,2023-12-28:/2023/12/28/debian-ci-10-years-later/Debian CI: 10 years later2023-12-28T15:00:00Z2023-12-28T15:00:00Z<p>It was 2013, and I was on a break from work between Christmas and New Year of
2013. I had been working at <a href="https://www.linaro.org/">Linaro</a> for well over a year, on the <a href="https://www.lavasoftware.org/">LAVA
project</a>. I was living and breathing automated testing infrastructure,
mostly for testing low-level components such as kernels and bootloaders, on
real hardware.</p>
<p>At this point I was also a Debian contributor for quite some years, and had
become an official project members two years prior. Most of my involvement was
in the Ruby team, where we were already consistently running upstream test
suites during package builds.</p>
<p>During that break, I put these two contexts together, and came to the
conclusion that Debian needed a dedicated service that would test the contents
of the Debian archive. I was aware of the existance of
<a href="https://packages.debian.org/sid/autopkgtest">autopkgtest</a>, and started working on a very simple service that
would later become <a href="https://ci.debian.net/">Debian CI</a>.</p>
<p>In January 2014, <code>debci</code> was initially announced on that month's <a href="https://lists.debian.org/debian-devel-announce/2014/01/msg00007.html">Misc
Developer News</a>, and later <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=736416">uploaded to Debian</a>. It's been
continuously developed for the last 10 years, evolved from a single shell
script running tests in a loop into a distributed system with 47
geographically-distributed machines as of writing this piece, became part of
the official Debian release process gating migrations to testing, had 5 Summer
of Code and Outrechy interns working on it, and processed beyond 40 million
test runs.</p>
<p>In there years, Debian CI has received contributions from a lot of people, but
I would like to give special credits to the following:</p>
<ul>
<li>Ian Jackson - created autopkgtest.</li>
<li>Martin Pitt - was the maintainer of autopkgtest when Debian CI launched and
helped a lot for some time.</li>
<li>Paul Gevers - decided that he wanted Debian CI test runs to control testing
migration. While at it, became a member of the Debian Release Team and the
other half of the permanent Debian CI team together with me.</li>
<li>Lucas Kanashiro - Google Summer of Code intern, 2014.</li>
<li>Brandon Fairchild - Google Summer of Code intern, 2014.</li>
<li>Candy Tsai - Outreachy intern, 2019.</li>
<li>Pavit Kaur - Google Summer of Code intern, 2021</li>
<li>Abiola Ajadi - Outreachy intern, December 2021-2022.</li>
</ul>
tag:terceiro.xyz,2021-10-12:/2021/10/12/triaging-debian-build-failure-logs-with-collab-qa-tools/Triaging Debian build failure logs with collab-qa-tools2021-10-12T08:30:00Z2021-10-12T08:30:00Z<p>The Ruby team is working now on transitioning to ruby 3.0. Even though most
packages will work just fine, there is substantial amount of packages that
require some work to adapt. We have been doing test rebuilds for a while during
transitions, but usually triaged the problems manually.</p>
<p>This time I decided to try
<a href="https://salsa.debian.org/lucas/collab-qa-tools">collab-qa-tools</a>, a set of
scripts Lucas Nussbaum uses when he does archive-wide rebuilds. I'm really glad
that I did, because those tols save a lot of time when processing a large
number of build failures. In this post, I will go through how to triage a set
of build logs using collab-qa-tools.</p>
<p>I have
<a href="https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/13">made</a>
<a href="https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/14">some</a>
<a href="https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/15">improvements</a>
to the code. Given my last merge request is very new and was not merged yet,
a few of the things I mention here may apply only to
<a href="https://salsa.debian.org/terceiro/collab-qa-tools/-/tree/ruby3.0">my own <code>ruby3.0</code> branch</a>.</p>
<p>collab-qa-tools also contains a few tools do perform the builds in the
cloud, but since we already had the builds done, I will not be mentioning that
part and will write exclusively about the triaging tools.</p>
<h2>Installing collab-qa-tools</h2>
<p>The first step is to clone the git repository. Make sure you have the
dependencies from <code>debian/control</code> installed (a few Ruby libraries).</p>
<p>One of the patches I sent, and was already accepted, is the ability to run it
without the need to install:</p>
<pre><code>source /path/to/collab-qa-tools/activate.sh
</code></pre>
<p>This will add the tools to your $PATH.</p>
<h2>Preparation</h2>
<p>The first think you need to do is getting all your build logs in a directory.
The tools assume <code>.log</code> file extension, and they can be named
<code>${PACKAGE}_*.log</code> or just <code>${PACKAGE}.log</code>.</p>
<h2>Creating a TODO file</h2>
<pre><code>cqa-scanlogs | grep -v OK > todo
</code></pre>
<p><code>todo</code> will contain one line for each log with a summary of the failure, if
it's able to find one. collab-qa-tools has a large set of regular expressions
for finding errors in the build logs</p>
<p>It's a good idea to split the TODO file in multiple ones. This can easily be
done with <code>split(1)</code>, and can be used to delimit triaging sessions, and/or to
split the triaging between multiple people. For example this will create
<code>todo</code> into <code>todo00</code>, <code>todo01</code>, ..., each containing 30 lines:</p>
<pre><code>split --lines=30 --numeric-suffixes todo todo
</code></pre>
<h2>Triaging</h2>
<p>You can now do the triaging. Let's say we split the TODO files, and will start
with <code>todo01</code>.</p>
<p>The first step is calling <code>cqa-fetchbugs</code> (it does what it says on the tin):</p>
<pre><code>cqa-fetchbugs --TODO=todo01
</code></pre>
<p>Then, <code>cqa-annotate</code> will guide you through the logs and allow you to report
bugs:</p>
<pre><code>cqa-annotate --TODO=todo01
</code></pre>
<p>I wrote myself a <code>process.sh</code> wrapper script for <code>cqa-fetchbugs</code> and
<code>cqa-annotate</code> that looks like this:</p>
<pre><code class="language-bash"><span class="c">#!/bin/sh</span>
<span class="nb">set</span> <span class="nt">-eu</span>
<span class="k">for </span>todo <span class="k">in</span> <span class="nv">$@</span><span class="p">;</span> <span class="k">do</span>
<span class="c"># force downloading bugs</span>
<span class="nb">awk</span> <span class="s1">'{print(".bugs." $1)}'</span> <span class="s2">"</span><span class="k">${</span><span class="nv">todo</span><span class="k">}</span><span class="s2">"</span> | xargs <span class="nb">rm</span> <span class="nt">-f</span>
cqa-fetchbugs <span class="nt">--TODO</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">todo</span><span class="k">}</span><span class="s2">"</span>
cqa-annotate <span class="se">\</span>
<span class="nt">--template</span><span class="o">=</span>template.txt.jinja2 <span class="se">\</span>
<span class="nt">--TODO</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">todo</span><span class="k">}</span><span class="s2">"</span>
<span class="k">done</span></code></pre>
<p>The <code>--template</code> option is a recent contribution of mine. This is a template
for the bug reports you will be sending. It uses
<a href="https://github.com/Shopify/liquid/wiki/Liquid-for-Designers">Liquid templates</a>,
which is very similar to Jinja2 for Python. You will notice that I am even
pretending it <em>is</em> Jinja2 to trick vim into doing syntax highlighting
for me. The template I'm using looks like this:</p>
<pre><code class="language-jinja">From: <span class="cp">{{</span> <span class="nv">fullname</span> <span class="cp">}}</span> <span class="nt"><</span><span class="cp">{{</span> <span class="nv">email</span> <span class="cp">}}</span><span class="nt">></span>
To: submit@bugs.debian.org
Subject: <span class="cp">{{</span> <span class="nv">package</span> <span class="cp">}}</span>: FTBFS with ruby3.0: <span class="cp">{{</span> <span class="nv">summary</span> <span class="cp">}}</span>
Source: <span class="cp">{{</span> <span class="nv">package</span> <span class="cp">}}</span>
Version: <span class="cp">{{</span> <span class="nv">version</span> <span class="o">| </span><span class="nf">split</span><span class="err">:</span><span class="s1">'+rebuild'</span> <span class="o">| </span><span class="nf">first</span> <span class="cp">}}</span>
Severity: serious
Justification: FTBFS
Tags: bookworm sid ftbfs
User: debian-ruby@lists.debian.org
Usertags: ruby3.0
Hi,
We are about to enable building against ruby3.0 on unstable. During a test
rebuild, <span class="cp">{{</span> <span class="nv">package</span> <span class="cp">}}</span> was found to fail to build in that situation.
To reproduce this locally, you need to install ruby-all-dev from experimental
on an unstable system or build chroot.
Relevant part (hopefully):
<span class="cp">{%</span> <span class="k">for</span> <span class="nv">line</span> <span class="ow">in</span> <span class="nv">extract</span> <span class="cp">%}</span>> <span class="cp">{{</span> <span class="nv">line</span> <span class="cp">}}</span>
<span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span>
The full build log is available at
https://people.debian.org/~kanashiro/ruby3.0/round2/builds/3/<span class="cp">{{</span> <span class="nv">package</span> <span class="cp">}}</span>/<span class="cp">{{</span> <span class="nv">filename</span> <span class="o">| </span><span class="nf">replace</span><span class="err">:</span><span class="s2">".log"</span><span class="p">,</span><span class="s2">".build.txt"</span> <span class="cp">}}</span></code></pre>
<h2>The cqa-annotate loop</h2>
<p><code>cqa-annotate</code> will parse each log file, display an extract of what it found as
possibly being the relevant part, and wait for your input:</p>
<pre><code>######## ruby-cocaine_0.5.8-1.1+rebuild1633376733_amd64.log ########
--------- Error:
Failure/Error: undef_method :exitstatus
FrozenError:
can't modify frozen object: pid 2351759 exit 0
# ./spec/support/unsetting_exitstatus.rb:4:in `undef_method'
# ./spec/support/unsetting_exitstatus.rb:4:in `singleton class'
# ./spec/support/unsetting_exitstatus.rb:3:in `assuming_no_processes_have_been_run'
# ./spec/cocaine/errors_spec.rb:55:in `block (2 levels) in <top (required)>'
Deprecation Warnings:
Using `should` from rspec-expectations' old `:should` syntax without explicitly enabling the syntax is deprecated. Use the new `:expect` syntax or explicitly enable `:should` with `config.expect_with(:rspec) { |c| c.syntax = :should }` instead. Called from /<<PKGBUILDDIR>>/spec/cocaine/command_line/runners/backticks_runner_spec.rb:19:in `block (2 levels) in <top (required)>'.
If you need more of the backtrace for any of these deprecations to
identify where to make the necessary changes, you can configure
`config.raise_errors_for_deprecations!`, and it will turn the
deprecation warnings into errors, giving you the full backtrace.
1 deprecation warning total
Finished in 6.87 seconds (files took 2.68 seconds to load)
67 examples, 1 failure
Failed examples:
rspec ./spec/cocaine/errors_spec.rb:54 # When an error happens does not blow up if running the command errored before execution
/usr/bin/ruby3.0 -I/usr/share/rubygems-integration/all/gems/rspec-support-3.9.3/lib:/usr/share/rubygems-integration/all/gems/rspec-core-3.9.2/lib /usr/share/rubygems-integration/all/gems/rspec-core-3.9.2/exe/rspec --pattern ./spec/\*\*/\*_spec.rb --format documentation failed
ERROR: Test "ruby3.0" failed:
----------------
ERROR: Test "ruby3.0" failed: Failure/Error: undef_method :exitstatus
----------------
package: ruby-cocaine
lines: 30
------------------------------------------------------------------------
s: skip
i: ignore this package permanently
r: report new bug
f: view full log
------------------------------------------------------------------------
Action [s|i|r|f]:
</code></pre>
<p>You can then choose one of the options:</p>
<ul>
<li>
<code>s</code> - skip this package and do nothing. You can run <code>cqa-annotate</code> again
later and come back to it.</li>
<li>
<p><code>i</code> - ignore this package completely. New runs of <code>cqa-annotate</code> won't ask
about it again.</p>
<p>This is useful if the package only fails in your rebuilds due to another
package, and would just work when that other package gets fixes. In the Ruby
transition this happens when A depends on B, while B builds a C extension and
failed to build against the new Ruby. So once B is fixed, A should
just work (in principle). But even if A would even have problems of its own,
we can't really know before B is fixed so we can retry A.</p>
</li>
<li>
<p><code>r</code> - report a bug. <code>cqa-annotate</code> will expand the template with the data
from the current log, and feed it to mutt. This is currently a limitation:
you have to use mutt to report bugs.</p>
<p>After you report the bug, <code>cqa-annotate</code> will ask if it should edit the TODO
file. In my opinion it's best to not do this, and annotate the package with a
bug number when you have one (see below).</p>
</li>
<li><p><code>f</code> - view the full log. This is useful when the extract displayed doesn't
have enough info, or you want to inspect something that happened earlier (or
later) during the build.</p></li>
</ul>
<p>When there are existing bugs in the package, <code>cqa-annotate</code> will list them
among the options. If you choose a bug number, the TODO file will be annotated
with that bug number and new runs of <code>cqa-annotate</code> will not ask about that
package anymore. For example after I reported a bug for <code>ruby-cocaine</code> for the
issue listed above, I aborted with a <code>ctrl-c</code>, and when I run my <code>process.sh</code>
script again I then get this prompt:</p>
<pre><code>----------------
ERROR: Test "ruby3.0" failed: Failure/Error: undef_method :exitstatus
----------------
package: ruby-cocaine
lines: 30
------------------------------------------------------------------------
s: skip
i: ignore this package permanently
1: 996206 serious ruby-cocaine: FTBFS with ruby3.0: ERROR: Test "ruby3.0" failed: Failure/Error: undef_method :exitstatus ||
r: report new bug
f: view full log
------------------------------------------------------------------------
Action [s|i|1|r|f]:
</code></pre>
<p>Chosing <code>1</code> will annotate the TODO file with the bug number, and I'm done with
this package. Only a few other hundreds to go.</p>
tag:terceiro.xyz,2021-07-19:/2021/07/19/getting-help-with-autopkgtest-for-your-package/Getting help with autopkgtest for your package2021-07-19T21:00:00Z2021-07-19T21:00:00Z<p>If you have been involved in Debian packaging at all in the last few years, you
are probably aware that autopkgtest is now an important piece of the Debian
release process. Back in 2018, the automated testing migration process started
<a href="https://lists.debian.org/msgid-search/4cb42bdc-ea03-57ca-1623-435d562f05ff@debian.org">considering autopkgtest test results</a> as part of its decision
making.</p>
<p>Since them, this process has received several improvements. For example, during
the <a href="https://lists.debian.org/msgid-search/a3ce6189-5306-9019-d978-dba4419d1520@debian.org">bullseye freeze</a>, non-key packages with a non-trivial
autopkgtest test suite could migrate automatically to testing without their
maintainers needing to open unblock requests, provided there was no regression
in theirs autopkgtest (or those from their reverse dependencies).</p>
<p><a href="/2014/06/01/an-introduction-to-the-debian-continuous-integration-project/">Since 2014 when ci.debian.net was first introduced</a>, we have seen an
amazing increase in the number of packages in Debian that can be automatically
tested. We went from around 100 to 15,000 today. This means not only happier
maintainers because their packages get to testing faster, but also <strong>improved
quality assurance for Debian as a whole</strong>.</p>
<p><img src="/images/ci-stats-2021.png" title="Packages tested by ci.debian.net on amd64" alt="Chart showing the number of packages tested by ci.debian.net. Starts from close to 0 in 2014, up to 15,000 in 2021. The growth tendency seems to slow down in the last year"></p>
<p>However, the growth rate seems to be decreasing. Maybe the low hanging fruit
have all been picked, or maybe we just need to help more people jump in the
automated testing bandwagon.</p>
<p>With that said, we would like to encourage and help more maintainers to add
autopkgtest to their packages. To that effect, I just created the
<a href="https://salsa.debian.org/ci-team/autopkgtest-help/">autopkgtest-help repository on salsa</a>, where we will take
help requests from maintainers working on autopkgtest for their packages.</p>
<p>If you want help, please go ahead and <a href="https://salsa.debian.org/ci-team/autopkgtest-help/-/issues/new">create an issue in there</a>.
To quote the repository README:</p>
<blockquote>
<p>Valid requests:</p>
<ul>
<li>
<p><em>"I want to add autopkgtest to package X. X is a tool that [...] and it works
by [...]. How should I approach testing it?"</em></p>
<p>It's OK if you have no idea where to start. But at least try to describe your
package, what it does and how it works so we can try to help you.</p>
</li>
<li>
<p><em>"I started writing autopkgtest for X, here is my current work in progress
[link]. But I encountered problem Y. How to I move forward?"</em></p>
<p>If you already have an autopkgtest but is having trouble making it work as
you think it should, you can also ask here.</p>
</li>
</ul>
<p>Invalid requests:</p>
<ul>
<li>
<p><em>"Please write autopkgtest for my package X for me"</em>.</p>
<p>As with anything else in free software, please show appreciation for other
people's time, and do your own research first. If you pose your question with
enough details (see above) and make it interesting, it <em>may</em> be that whoever
answers will write at least a basic structure for you, but as the maintainer
you are still the expert in the package and what tests are relevant.</p>
</li>
</ul>
</blockquote>
<p>If you ask your question soon, you might get your answer recorded in video: we
are going to have a <a href="https://debconf21.debconf.org/talks/1-autopkgtest-office-hours/">DebConf21 talk</a> next month, where we I and
Paul Gevers (elbrus) will answer a few autopkgtest questions in video for
posterity.</p>
<p>Now, if you have experience enabling autopkgtest for you own packages, please
consider watching that repository there to help us help our fellow maintainers.</p>
tag:terceiro.xyz,2021-06-27:/2021/06/27/debian-continuous-integration-now-using-salsa-logins/Debian Continuous Integration now using Salsa logins2021-06-27T13:00:00Z2021-06-27T13:00:00Z<p>I have just updated the <a href="https://ci.debian.net">Debian Continuous Integration
platform</a> with <a href="https://tracker.debian.org/news/1243579/accepted-debci-31-source-into-experimental/">debci 3.1</a>.</p>
<p>This update brings a few database performance improvements, courtesy of adding
indexes to very important columns that were missing them. And boy, querying a
table with 13 million rows without the proper indexes is bad! :-)</p>
<p>Now, the most user visible change in this update is the change from Debian SSO
to Salsa Logins, which is part of <a href="https://pavitkaur05.github.io/">Pavit Kaur</a>'s GSoC work. She has been
working with me and Paul Gevers for a few weeks, and this was the first
official task in the internship.</p>
<p>For users, this means that you now can only log in via Salsa. If you have an
existing session where you logged in with an SSO certificate, it will still be
valid. When you log in with Salsa, your username will be changed to match the
one in Salsa. This means that if your account on salsa gets renamed, it will
automatically be renamed on Debian CI when you log in the next time.
Unfortunately we don't have a logout feature yet, but in the meantime you can
use the developer toolbar to delete any existing cookies you might have for
<code>ci.debian.net</code>.</p>
<p>Migrating to Salsa logins was in my TODO list for a while. I had the impression
that it could do it pretty quick to do by using pre-existing libraries that
provide gitlab authentication integration for Rack (Ruby's de facto standard web
application interface, like uwsgi for Python). But in reality, the devil was in
the details.</p>
<p>We went through <a href="https://salsa.debian.org/ci-team/debci/-/merge_requests/179">several rounds of reviews</a> to get it right. During
the entire process, Pavit demonstrated an excelent capacity for responding to
feedback, and overall I'm very happy with her performance in the internship so
far.</p>
<p>While we were discussing the Salsa logins, we noted a limitation in the existing
database structure, where we stored usernames directly as the test <code>requestor</code>
field, and decided it was better to normalize that relationship with a proper
foreign key to the users table, which she <a href="https://salsa.debian.org/ci-team/debci/-/merge_requests/181">also worked on</a>.</p>
<p>This update also include the very first (and non-user visible) step of her next
task, which is adding support for having private tests. Those will be useful
for implementing testing for embargoed security updates, and other use cases.
This was broken up into 7 or 8 seperate steps, so there is still some
work to do there. I'm looking forward to the continuation of this work.</p>
tag:terceiro.xyz,2021-03-27:/2021/03/27/migrating-from-chef-to-itamae/Migrating from Chef™ to itamae2021-03-27T20:00:00Z2021-03-27T20:00:00Z<p>The <a href="https://ci.debian.net/">Debian CI</a> platform is comprised of 30+ (virtual)
machines. Maintaining this many machines, and being able to add new ones with
some degree of reliability requires one to use some sort of configuration
management.</p>
<p>Until about a week ago, we were using Chef for our <a href="https://salsa.debian.org/ci-team/debian-ci-config/">configuration
management</a>. I was, for
several years, the main maintainer of Chef in Debian, so using it was natural
to me, as I had used it before for personal and work projects. But last year I
decided to <a href="https://bugs.debian.org/963750">request the removal of Chef from
Debian</a>, so that it won't be shipped with
Debian 11 (<em>bullseye</em>).</p>
<p>After evaluating a few options, I believed that the path of least resistance
was to migrate to <a href="https://itamae.kitchen/">itamae</a>. itamae was inspired by
chef, and uses a DSL that is very similar to the Chef one. Even though the
itamae team claim it's not compatible with Chef, the changes that I needed to
do were relatively limited. The <a href="https://salsa.debian.org/ci-team/debian-ci-config/-/merge_requests/6">necessary code
changes</a>
might look like a lot, but a large part of them could be automated or done in
bulk, like doing simple search and replace operations, and moving entire
directories around.</p>
<p>In the rest of this post, I will describe the migration process, starting with
the infrastructure changes, the types of changes I needed to make to the
configuration management code, and my conclusions about the process.</p>
<h2>Infrastructure changes</h2>
<p>The first step was to add support for itamae to
<a href="https://gitlab.com/terceiro/chake">chake</a>, a configuration management wrapper
tool that I wrote. chake was originally written as a serverless remote executor
for Chef, so this involved a bit of a redesign. I thought it was worth it to
do, because at that point chake had gained several interesting managements
features that we no directly tied to Chef. This work was done a bit slowly over
the course of the several months, starting almost exactly one year ago, and was
completed 3 months ago. I wasn't in a hurry and knew I had time before Debian
11 is released and I had to upgrade the platform.</p>
<p>After this was done, I started the work of migrating the then Chef cookbooks to
itamae, and the next sections present the main types of changes that were
necessary.</p>
<p>During the entire process, I sent a few patches out:</p>
<ul>
<li><a href="https://github.com/itamae-kitchen/itamae/pull/317">itamae #317 - Make output unbuffered</a></li>
<li><a href="https://github.com/itamae-kitchen/itamae/pull/325">itamae #325 - file: add support for sensitive files</a></li>
<li><a href="https://github.com/itamae-kitchen/itamae/pull/327">itamae #327 - file: improve diff output</a></li>
<li>
<a href="https://github.com/itamae-kitchen/itamae/pull/337">itamae #337 - package: add :upgrade action</a>.
This is work in progress at the moment, and wasn't merged yet.</li>
<li><a href="https://github.com/mizzy/specinfra/pull/727">serverspec #727 - debian: add support for Debian testing and unstable</a></li>
</ul>
<h2>Code changes</h2>
<p>These are the main types of changes that were necessary in the configuration
code to accomplish the migration to itamae.</p>
<h3>Replace <code>cookbook_file</code> with <code>remote_file</code>.</h3>
<p>The resource known as <code>cookbook_file</code> in Chef is called <code>remote_file</code> in itamae.
Fixing this is just a matter of search and replace, e.g.:</p>
<pre><code class="language-diff"><span class="gd">-cookbook_file '/etc/apt/apt.conf.d/00updates' do
</span><span class="gi">+remote_file '/etc/apt/apt.conf.d/00updates' do</span></code></pre>
<h3>Changed file locations</h3>
<p>The file structure assumed by itamae is a lot simpler than the one in Chef. The
needed changes were:</p>
<ul>
<li>static files and templates moved from <code>cookbooks/${cookbook}/{files,templates}/default</code> to <code>cookbooks/${cookbook}/{files,templates}</code>
</li>
<li>recipes moved from <code>cookbooks/${cookbook}/recipes/*.rb</code> to <code>cookbooks/${cookbook}/*.rb</code>
</li>
<li>
<p>host-specific files and templates are not supported directly, but can be
implemented just by using an explicit source statement, like this:</p>
<pre><code class="language-ruby"><span class="n">remote_file</span> <span class="s2">"/etc/foo.conf"</span> <span class="k">do</span>
<span class="n">source</span> <span class="s2">"files/host-</span><span class="si">#{</span><span class="n">node</span><span class="p">[</span><span class="s1">'fqdn'</span><span class="p">]</span><span class="si">}</span><span class="s2">/foo.conf"</span>
<span class="k">end</span></code></pre>
</li>
</ul>
<h3>Explicit file ownership and mode</h3>
<p>Chef is usually design to run as root on the nodes, and files created are owned
by root and have move <code>0644</code> by default. With itamae, files are by default
owned by the user that was used to SSH into the machine. Because of this, I had
to review all file creation resources and add owner, group and mode explicitly:</p>
<pre><code class="language-diff"><span class="gd">-cookbook_file '/etc/apt/apt.conf.d/00updates' do
- source 'apt.conf'
</span><span class="gi">+remote_file '/etc/apt/apt.conf.d/00updates' do
+ source 'files/apt.conf'
+ owner 'root'
+ group 'root'
+ mode "0644"
</span> end</code></pre>
<p>In the end, I guess being explicit make the configuration code more
understandable, so I take that as a win.</p>
<h3>Different execution context</h3>
<p>One of the major differences between Chef itamae comes down the execution
context of the recipes. In both Chef and itamae, the configuration is written
in DSL embedded in Ruby. This means that the recipes are just Ruby code, and
difference here has to do with where that code is executed. With Chef, the
recipes are always execute on the machine you are configuring, while with
itamae the recipe is executed on the workstation where you run itamae, and that
gets translated to commands that need to be executed on the machine being
configured.</p>
<p>For example, if you need to configure a service based on how much RAM the
machine has, with Chef you could do something like this:</p>
<pre><code class="language-ruby"><span class="n">total_ram</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">readlines</span><span class="p">(</span><span class="s2">"/proc/meminfo"</span><span class="p">).</span><span class="nf">find</span> <span class="k">do</span> <span class="o">|</span><span class="n">l</span><span class="o">|</span>
<span class="n">l</span><span class="p">.</span><span class="nf">split</span><span class="p">.</span><span class="nf">first</span> <span class="o">==</span> <span class="s2">"MemTotal:"</span>
<span class="k">end</span><span class="p">.</span><span class="nf">split</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">file</span> <span class="s2">"/etc/service.conf"</span> <span class="k">do</span>
<span class="c1"># use 20% of the total RAM</span>
<span class="n">content</span> <span class="s2">"cache_size = </span><span class="si">#{</span><span class="n">ram</span> <span class="o">/</span> <span class="mi">5</span><span class="si">}</span><span class="s2">KB"</span>
<span class="k">end</span></code></pre>
<p>With itamae, all that Ruby code will run on the client, so <code>total_ram</code> will
contain the wrong number. In the Debian CI case, I worked around that by
explicitly declaring the amount of RAM in the static host configuration, and
the above construct ended up as something like this:</p>
<pre><code class="language-ruby"><span class="n">file</span> <span class="s2">"/etc/service.conf"</span> <span class="k">do</span>
<span class="c1"># use 20% of the total RAM</span>
<span class="n">content</span> <span class="s2">"cache_size = </span><span class="si">#{</span><span class="n">node</span><span class="p">[</span><span class="s1">'total_ram'</span><span class="p">]</span> <span class="o">/</span> <span class="mi">5</span><span class="si">}</span><span class="s2">KB"</span>
<span class="k">end</span></code></pre>
<h2>Lessons learned</h2>
<p>This migration is now complete, and there are a few points that I take away
from it:</p>
<ul>
<li>The migration is definitely viable, and I'm glad I picked itamae after all.</li>
<li>Of course, itamae is way simpler than Chef, and has less features. On the
other hand, this means that it a simple package to maintain, with less
dependencies and keeping it up to date is <a href="https://metadata.ftp-master.debian.org/changelogs//main/i/itamae/itamae_1.11.1-1_changelog">a lot easier</a>.</li>
<li>itamae is considerably slower than Chef. On my local tests, a <em>noop</em>
execution (e.g. re-applying the configuration a second time) against local
VMs with itamae takes 3x the time it takes with Chef.</li>
</ul>
<p>All in all, the system is working just fine, and I consider this to have been a
successful migration. I'm happy it worked out.</p>