Satellites and failover
One server is enough for most teams. When you want to spread client connections and delivery across machines, add satellites: extra servers that join your first server, the hub. Clients are assigned to servers automatically and fail over to another server by themselves when theirs stops answering, without losing data. No external database is needed.
How it fits together
- The hub keeps the settings, the client list, and the full history in its embedded database. It assigns clients to servers and serves clients like any server.
- A satellite accepts clients, checks them against a copy of the hub's client list, forwards telemetry to your destinations itself, and sends every record to the hub for storage.
- Clients connect to one server at a time: their assigned primary, with an ordered fallback list.
Settings are edited on the hub only. A satellite's dashboard shows its own status, its connected clients and live activity, with a note that settings are managed by the hub.
Add a satellite
- On the hub: open Settings → Scaling → Add server (or Add satellite on the Cluster page) and copy the join code. It works once, for 30 minutes.
- On the new machine: install SpanDock and choose Join it as a satellite on the first launch, then paste the code.
On a Linux machine without a screen
Install it as a service with the join code. One command does it all:
curl -fsSL https://spandock.com/deploy/install-server.sh | sudo SPANDOCK_JOIN_CODE='…' bash
Or install first, then answer the questions instead. The join code is asked for and hidden as you type, so it never appears in your shell history:
curl -fsSL https://spandock.com/deploy/install-server.sh | sudo bash
sudo spandock onboard
Either way the code goes in the environment, never on the command line, so it stays out of the process list. The script runs SpanDock as a spandock service that starts again after every restart, keeps the satellite's settings and cluster credentials in its own folder, and uses the join code only the first time. Cloud-init, Terraform and Ansible do the same: see Deploy a server in the cloud.
Why a satellite started by hand goes offline after a restart
Running SPANDOCK_JOIN_CODE='…' spandock -role=server … in a terminal joins the hub, but nothing keeps it running:
- Nothing starts it again. It runs only as long as that terminal session. After a reboot, or when you log out, it's gone, and the hub shows it offline.
- Its settings belong to the user who ran it. They're kept in that user's home folder. A service or another user starting SpanDock later finds no satellite settings there, so it starts as a new, unjoined server.
- Its credentials may live in that login session's keychain. On a machine without a desktop that keychain disappears when you log out. Run headless servers with
SPANDOCK_SECRET_STORE=file, as the install script does, to keep credentials in a private file next to the settings.
To fix a satellite that went offline this way:
- On the hub, open Cluster, choose Disconnect for the old satellite, then Remove its record.
- Create a new join code (Settings → Scaling → Add server, or Add satellite on the Cluster page).
- On the machine, run
sudo spandock onboard. When the last join failed, it asks for the new code first. Or run the install command above with the new code.
A join code works once. If the machine was ever started by hand, that run used the code, and the service then can't join with it: it stops and logs join the hub: this join code is invalid, expired, or already used (older versions keep retrying). A new code fixes it.
Once a satellite has joined, Clients → Add client on the hub asks which server the new client connects to.
Server assignment
The hub gives every client a primary server and a fallback list, and pushes them to the client over the encrypted tunnel.
- Sticky: a client whose primary is healthy keeps it. Moving costs a reconnect, so SpanDock avoids it.
- Spread: a client without a healthy primary goes to the least loaded server. Fallback lists are rotated per client, so when a server fails its clients spread across the others instead of all landing on one.
- Gradual rebalancing: if one server becomes much busier than the average, a small share of clients is moved at a time, spread out, so there is never a reconnect stampede.
- Pin to server: in a client's details you can pin it to one server whatever the load. A pinned client still fails over if that server stops answering.
- Drain: choose Drain for a satellite under Settings → Scaling to stop sending it clients and remove it from fallback lists. Its clients move within a minute. Disconnect it once it has none.
Automatic failover
When a client's server stops answering, the client first tries to heal the connection to the same server, so a short restart or update doesn't bounce it around. If that hasn't worked after about 20 seconds, it moves to the next server in its fallback list, and on down the list.
- No data loss: while the client is moving, telemetry goes into its on-disk queue. After the switch, the queue is replayed to the new server. Replays are deduplicated, so nothing is stored twice.
- Going back: a client on a fallback checks its primary every 30 seconds and returns only after a minute of continuous health, with a small random delay so a recovered server doesn't receive all its clients at once.
- When nothing answers (for example, the laptop itself is offline), the client keeps queueing and waits longer between attempts, up to 5 minutes.
- Long outages: if a server stays down, the hub makes each of its clients' current server their new primary.
The client's Settings → Server connection shows which server it is on, whether that is its primary or a fallback, and when it last failed over.
Updates
Update the hub, and it rolls its version out to the cluster: satellites first, then clients.
The cluster diagram
The Cluster page on the hub is a live diagram of your servers and clients. Servers are coloured by state: green online, orange draining, red down. A star marks the hub. Lines from each client to its current server move with its live traffic, and when a client fails over you see its line move to the new server. An event log lists satellites joining, going offline and coming back, and clients failing over or returning. A table view lists the same information. The Clients page also tags each client with the server it is on.
Limits
- The hub is a single point of failure for settings and history. It is not replaced automatically.
- While the hub is down, satellites keep accepting telemetry from their clients and keep forwarding it to your destinations. Records meant for the hub's history are held on each satellite and sent in order when the hub returns. Settings are read-only in the meantime: no new pairings, approvals or revocations. Client history pages show that history is on the hub.
- Storage throughput is limited to what the hub's embedded database can take. Connections and destination delivery scale out across satellites.
The Cluster page shows how many batches satellites are holding for the hub. A backlog that keeps growing is the signal that you have outgrown one hub.
High availability
If you need a high-availability cluster with no single point of failure, it is part of the Enterprise plan. Contact sales@spandock.com to discuss it.