Bug#930446: popularity-contest: unable to submit report, impossible to debug

[Bill Allombert <ballombe@debian.org>]

On Sun, Sep 01, 2019 at 05:52:37PM +0200, Ludovic Rousseau wrote:
> So the cron job is executed at 7h57.
> 
> > > > In which case, could you check what is the issue with the timestamp
> > > > (see the full buildlog) ?
> > > 
> > > Can you be more specific about what you want me to check?
> > 
> > What gives
> > ls -l /var/log/popularity-contest*
> 
> $ LANG=C ls -l /var/log/popularity-contest*
> -rw-r--r-- 1 root root 24169 Aug 29 07:57 /var/log/popularity-contest
> -rw-r--r-- 1 root root     0 Aug 29 07:57 /var/log/popularity-contest.0
> -rw-r--r-- 1 root root  5563 Aug 22 07:57 /var/log/popularity-contest.1.gz
> -rw-r--r-- 1 root root    41 Aug 22 07:57 /var/log/popularity-contest.2.gz
> -rw-r--r-- 1 root root  5541 Aug 15 07:57 /var/log/popularity-contest.3.gz
> -rw-r--r-- 1 root root    41 Aug 15 07:57 /var/log/popularity-contest.4.gz
> -rw-r--r-- 1 root root  5583 Aug  8 07:57 /var/log/popularity-contest.5.gz
> -rw-r--r-- 1 root root    41 Aug  8 07:57 /var/log/popularity-contest.6.gz
> -rw-r--r-- 1 root root  8451 Aug  1 07:57 /var/log/popularity-contest.new.gpg

This seems good, except there might be an issue with logrotate or
whoever create the empty /var/log/popularity-contest.0.

> > Do you run some program that change the timestamp of the file
> > /var/log/popularity-contest ? maybe logrotate ?
> 
> Not that I am aware of.
> 
> > > > In you have an unrelated problem, please open a separate bug report.
> > > > 
> > > > The debug output you got just means that the server failed to answer.
> > > 
> > > Sure.
> > > Why did the server failed to answer?
> > 
> > Probably too many people are trying to submit at the same time. Hence the move
> > to cron.d with a user specific submission time.
> 
> popcon.debian.org is hosted by pinel.debian.org.
> I could not find a real load issue from munin graphs at https://munin.debian.org/debian.org/pinel.debian.org/index.html
> (login dsa-guest, password dsa-guest)

It is too thin to show on munin, but doing external measurement show the
problem.

> Ah OK.
> popularity-contest has TWO cron configurations:
> - /etc/cron.daily/popularity-contest
> - /etc/cron.d/popularity-contest
> 
> Should I just ignore the messages generated by the cron.daily job?

For the time being, yes.

I finally found the root of the problem, by writing a submission simulator.

There was a logic error in /etc/cron.daily/popularity-contest that
leads to the cron.daily to have higher priority than the cron.d.
Maybe it makes sense when cron.d was introduced, but not now.

I will fix it in the next update.
Thanks for following up on this painful issue and sorry for the delay.

Cheers,
-- 
Bill. <ballombe@debian.org>

Imagine a large red swirl here.