|Pjotr Prins 2e80389b72 README||5 months ago|
|bin||5 months ago|
|etc||5 months ago|
|sheepdog/monitor||5 months ago|
|.gitignore||5 months ago|
|COPYING||6 months ago|
|README.org||5 months ago|
|sheepdog-guix-manifest.scm||5 months ago|
-*- mode: org -*-
NOTICE: Sheepdog software is in early stages of development. YMMV.
A shepherd needs a sheepdog. If you have a herd of machines sheepdog helps you keep sane. Sheepdog does not replace the likes of Nagios or Zabbix though I am happy not to run those systems because they target (complex) deployments that have people watching these systems. Sheepdog can interact with other machines and, for example, schedule jobs or reconfigure a firewall during a hacking attempt.
Sheepdog is much simpler than Nagios because it does not require watching systems. Sheepdog gives a cursory idea of health of machines out there. It notifies you if things go really wrong, for example when backups fail. I used to have scripts for that that would mail/text me. But that was all a bit ad hoc and I got tired of maintaining them and I got tired of repeating notifications... Always an itch to scratch.
Also, sheepdog is a real helper for the shepherd. It will be designed to use logic programming. And allows questions like:
This means storing state of machines in a persistent database that gets updated by messages (SQLite). It means a good message broker (e.g., Redis or RabbitMQ). It means that every time you write a monitoring service, you'll have to write a receiver to turn it into a data structure. A key feature of sheepdog is to make creating such small reporter/receiver tools really easy. Sheepdog is pluggable - that means you can write a plugin for a monitor and you can write a plugin for the reporter/receiver which has hooks in the sheepdog daemon that runs on a server somewhere.
Visualisations are less important - though it is quite possible to build them.
In other words, much in the spirit of GNU Guix and GNU Shepherd, sheepdog is a different type of systems monitor: a minimalistic system that is hackable and can work out of the box for Guix systems and are really easy to extend.
Many tools run like CRON jobs and send output to some file. What happens when the job fails? You would like a notification! We can write a script wrapper that tests for the state of the backup job and puts it in the message queue when there is a problem. Say we have a backup job that backups a the /export/ipfs directory and is run with
borg create --stats /export/backup/borg::stamp /export/ipfs
Similar to most Unix tools
borg returns error code
0 on success.
sheepdog has to check the return code and should send a message
on failure. The message should include the error code and the command
that was executed. Ideally it should also capture stdout and stderr
for downstream trouble shooting.
sheepdog -c ~/.sheepdog.conf -m monitor-error \ -e "borg create --stats /export/backup/borg::stamp /export/ipfs"
here we can tell
sheepdog to only send a message on
monitor-error is the client-side module that handles the
checking. On the client we can also configure it to connect to a
certain host (and queue) and pass in an optional E-mail address. I.e.:
(config (host "localhost") (port 7777) (mailto "email@example.com") )
The receiver sits on the server and is a default plugin that is part of the sheepdog-queue-daemon. Receiving textual output is a standard message that is added in the database by the sheepdog-message plugin. If you want to change behaviour on the server you can write your own plugins.
There are hundreds of possible system indicators. The sheepdog-daemon can run on a system and monitors resources by default. Start the monitor daemon with
sheepdog-daemon -c server-config.scm
In the configuration you can specify what to watch. All monitors are (configurable) plugins.
The receiver is part of sheepdog-queue-daemon and consists of matching plugins (in fact, monitor and receiver with hooks live in the same source file).
The implementation starts in the most simple way. A message broker on a server. The actual queues are managed by Redis (a choice which may change later). As a choice redis is common, safe and fast. Later when we get to federated/distributed servers there may be other alternatives. On the server runs sheepdog-handler which attaches itself to a notification queue. When the sheepdog barks, a message goes into the queue and gets processed by the handler:
sheepdog -> bark -> queue -> handler
On the sheepdog end we create a module that reports on system error and pushes a commond when an error occurs. On the handler end we create a matching module which gets invoked when a message comes in.
This way both modules understand the
content of the message that
gets passed around. The handler module can decide what to do
with the message:
That is all up to the implementor! Key to understanding sheepdog is the flexible design and the hackability of message broker service.
Sheepdog is written in GNU Guile because it is small and fast and comes with almost every Linux installation. Sheepdog/Guile installs even on the smallest routers and mobile platforms. GNU Guile is a very powerful Scheme Lisp dialect. Even so, tools that use the sheepdog can be written in any language.
(when (require 'rainbow-delimiters nil 'noerror) (add-hook 'scheme-mode-hook #'rainbow-delimiters-mode))
In emacs M-x run-geiser to get a REPL. See also geiser docs.
Sheepdog is published under the GPLv3 License. See COPYING.