GitLab

We used to drop any mail we tried to send, regardless of whether
sending it actually succeeded or not.  This could result in lost mails
if the server failed to accept some mails for any temporary reason.

Fix this by re-queuing emails that failed to send and wait a few
moments before retrying.

The code was moved recently to be accessible by the watchdog but
a variable name was not changed…

Prints redirected to files are heavily cached, so we need to flush the
stream if we want to be able to read the output in log files as soon as
we return from signal handler.

Since we never reclaim finished tasks, exceptions raised during thread
run are never seen. Add a runner call in between to display them.

As the user configuration is now loaded after the mails module, there
is a race in the mail thread that fetches configuration options, which
may or may not already have the user's value.

So, remove caching of those value, which leads for them to be read much
later after everything has properly been initialized.  The timeout
setting might still have the default value on the first loop run, but
this doesn't matter as the server isn't started at this point anyway.

Now the 'every' check parameter can have a default value set globally and the
former global 'error_every' parameter can be overridden on a per-check
basis, so that the two parameters' configuration is now homogenous
and much more flexible.

Don't account for a task in the `finally` handler as we actually got a
task only if we didn't receive an exception.  This cleans up the code
and possibly fixes it if we ever get any other exception than a
timeout.

Don't explicitly handle KeyboardInterrupt in the mail thread as it
doesn't seem it ever was of of any use, and could actually lead to
a deadlock as it would terminate the in-thread loop without emptying
the queue, leading to `queue.join()` in `quit()` to wait forever for
the queue to empty.

A stupid typo lead for the mail thread not to be a daemon thread, thus
preventing Python from quitting before that thread quits by itself.

So properly make it a daemon thread, and as it would now actually get
interrupted abruptly, register an atexit handler to cleanly quit it.

This fixes ThreadedSMTP.quit() when the connection to the server is
down as Queue.join() will wait for all tasks to be popped and accounted
for, but we wouldn't account for the fake (empty) task because the
connection to the server would not be open.

So, properly differentiate a closed connection from a queue timeout.

Don't attempt to write a report after an arbitrary init time
but after a regular report interval.

AttrTree failed to create correct sub-attributes of itself because the
sub-attributes were not transmitted to the child.
In fact, str.split()'s 2nd argument is the number of split, not
the number of parts so '2' will split in 3 parts.