Supervision
Startup
Once the system is started, the system.init
actor starts all actors marked as entrypoint()
in the topology. Usually, it's only the system.configurers
group. Entry points must have empty configs.
Then, system.configurers
loads the config file and sends UpdateConfig
to all actor groups. Thus, if you need to start actors, the UpdateConfig
message must be routed to the necessary keys.
Note that the actor system terminates immediately if the config file is invalid at startup (elfo::start()
returns an error).
Reconfiguration
Reconfiguration can be caused in several cases:
- The configurer receives
ReloadConfigs
orTryRealodConfigs
. - The process receives
SIGHUP
, it's an equivalent to receivingReloadConfigs::default()
. - The process receives
SIGUSR2
, it's an equivalent to receivingReloadConfigs::default().with_force(true)
.
What's the difference between default behavior and with_force(true)
one? By default, group's up-to-date configs aren't sent across the system. In the force mode, all configs are updated, even unchanged ones.
The reconfiguration process consists of several stages:
- Config validation: the configurer sends the
ValidateConfig
request to all groups and waits for responses. - If all groups respond
Ok(_)
or ignore the request, the config is considered valid, and the configurer sendsUpdateConfig
to all groups. If at least one actor respondsErr(_)
, reconfiguration is aborted, and a new config isn't used.
More information about configs and the reconfiguration process is available on the corresponding page.
Restart
Three restart policies are implemented now:
RestartPolicy::on_failures
(by default) — actors are restarted only after failures (theexec
function returnedErr
or panicked).RestartPolicy::always
— actors are restarted after termination (theexec
function returnedOk
or()
) and failures.RestartPolicy::never
— actors are never restarted. However, they can be started again on incoming messages.
If the actor is scheduled to be restarted, incoming messages cannot spawn another actor for his key.
Repetitive restarts are limited by a linear backoff mechanism:
- If there are no restarts for 5s, restart an actor immediately.
- Otherwise, increase restart time by 5s and schedule restarting.
- Maximum restart time is limited by 30s.
These constants aren't configurable for now (elfo#62).
The restart policy can be chosen while creating ActorGroup
:
use elfo::group::{RestartPolicy, ActorGroup};
ActorGroup::new().restart_policy(RestartPolicy::always())
ActorGroup::new().restart_policy(RestartPolicy::on_failures())
ActorGroup::new().restart_policy(RestartPolicy::never())
Termination
System termination is started by the system.init
actor in several cases:
- Received
SIGTERM
orSIGINT
signals, Unix only. - Received
CTRL_C_EVENT
orCTRL_BREAK_EVENT
events, Windows only. - Too high memory usage, Unix only. Now it's 90% (ratio of RSS to total memory) and not configurable (elfo#60).
The process consists of several stages:
system.init
sends to all user-defined groups "polite"Terminate::default()
message.- Groups' supervisors stop spawning new actors.
Terminate
is routed as a usual message withOutcome::Broadcast
by default.- For
TerminationPolicy::closing
(by default): the mailbox is closed instantly,ctx.recv()
returnsNone
after already stored messages. - For
TerminationPolicy::manually
: the mailbox isn't closed,ctx.recv()
returnsTerminate
in order to handle in on your own. Usectx.close()
to close the mailbox.
- If some groups haven't terminated after 30s,
system.init
sendsTerminate::closing()
message.Terminate
is routed as a usual message withOutcome::Broadcast
by default.- For any policy, the mailbox is closed instantly,
ctx.recv()
returnsNone
after already stored messages.
- If some groups haven't terminated after another 15s,
system.init
stops waiting for that groups. - Repeat the above stages for system groups.
Timeouts above aren't configurable for now (elfo#61).
The termination policy can be chosen while creating ActorGroup
:
use elfo::group::{TerminationPolicy, ActorGroup};
ActorGroup::new().termination_policy(TerminationPolicy::closing())
ActorGroup::new().termination_policy(TerminationPolicy::manually())