Bug#727708: systemd and upstart, a view from a daemon upstream

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Firstly: a warning note.  Throughout this exercise I have been
persistently typing "upstart" when I mean "userv" and "userv" when I
mean "upstart".  If something I say make no sense, please try reading
it the other way :-).  (At least it gave me some extra bugs to
experience debugging...)

I have spent a while with the systemd and upstart reference manuals,
and written and tested integration for both systems for userv.
I didn't tackle OpenRC because AFAICT it has no reference manual.

Both systems seemed to have the key facilities I would have expected,
and I think (with minor enhancements) should be able to replace
sysvinit at least on Linux, and allow us to do away with unsupervised
(double-)forking daemons.

It's clear that systemd is a lot more sophisticated - or, to put it
another way, a lot more complex.  It took longer with systemd to see
the wood for the trees.

We have already discussed here the merits of systemd's daemon
readiness protocol.  I won't go into that again.  As I say, I
concluded that I wasn't willing to take any of the three plausible
routes to providing that in userv.

Conveniently, though, userv could be set up to use socket activation.
So this allowed me to experiment with that.  I did the upstart
integration with raise(SIGSTOP) and the systemd integration with
socket activation.

In both cases I invented a new command line option to specify the
startup protocol.  userv already had an option to request non-daemonic
startup, so this was a good fit.


In terms of the amount of code in uservd itself:

 * The upstart SIGSTOP readiness protocol was utterly trivial: exactly
   the one line raise(SIGSTOP), plus the command-line option handling.

   This was trivial to debug in an ad-hoc fashion, on my netbook which
   is running squeeze and sysvinit.

 * I had to write about 50 lines of environment-laundering code.  This
   is particularly because systemd's choice of environment variables
   for socket activation have very generic names: LISTEN_FDS and
   LISTEN_PID.

   systemd's theory AFAICT is that LISTEN_PID will prevent any process
   other than the intended one from honouring the LISTEN_* variables.
   That's of course not true of any other program which might already
   be using these variable names for a different purpose.

   I chose to also launder UPSTART_*, which is what made the
   environment laundering process more complicated.  If I had only
   wanted to unsetenv the two systemd variables it would have been
   much shorter, and arguably that would have been sufficient.

 * socket activation according to the systemd protocol was very short;
   the actual code to extract the socket fd was a mere 6 loc (I ignore
   LISTEN_PID, in violation of the nominal systemd protcol).
   Laundering the environment could be cut down to around 4 loc.  Plus
   command-line option handling, of course.

   I didn't attempt to debug this on my local workstation.  Doing so
   would probably have involved pratting about with socat plus a
   wrapper script to massage the environment etc.

   I didn't implement socket activation according to the upstart
   protocol but I expect it would be nearly as easy.  UPSTART_FDS is a
   bit more tricky because one has to tokenise it but userv only ever
   has one socket so actually that would be a trivial sanity check.

So overall my conclusions at this level are:

 * socket activation is an attractive implementation target for an
   upstream daemon author.

 * upstart's SIGSTOP protocol is an attractive implementation target
   for an upstream daemon author.

 * systemd's readiness protocol is an unattractive implementation
   target for an upstream daemon author.  I think this is an important
   weakness, if it remains unaddressed.