Restore is not recovery until the cluster agrees

Restoring etcd data is one step in Kubernetes recovery; control-plane identity, manifests, endpoints, certificates, and reconciliation still have to agree.

  • etcd
  • Kubernetes
  • Recovery
Site plan · OPS-044

A snapshot answers a narrow question

An etcd snapshot proves that a point-in-time keyspace was captured. A successful restore proves that those keys can be written into a new data directory. Neither result proves that the Kubernetes API can start, authenticate clients, reach the restored member, or reconcile a functioning control plane.

Recovery crosses ownership and configuration boundaries that backups do not: file permissions, member identity, peer and client URLs, static pod manifests, certificates, and the API server's etcd arguments all have to describe the same topology.

Bring components back into agreement

After restoring, verify the data directory owner and the effective manifest mounts before starting components. Confirm endpoint health with the credentials the control plane actually uses. Then verify API availability, authentication, nodes, system workloads, and controllers—not just an etcd health response.

The restored state may include objects whose external dependencies changed after the snapshot: load balancer addresses, certificates, tokens, volumes, or cloud resources. Controllers will act on that older desired state as soon as they return, so recovery includes understanding what they will reconcile.

Test outcomes, not commands

A useful drill defines recovery assertions in advance: the API is reachable through its normal path, privileged and ordinary identities behave correctly, core controllers are healthy, workloads can be scheduled, and critical application data is coherent. The restore command is a procedure; recovery is the demonstrated return of system behavior.

Further down

Keep digging.

  1. NET-017

    When the tunnel is up but traffic is dead

    A WireGuard handshake proves that peers can exchange encrypted control traffic. It does not prove that application packets have a working path.

    • WireGuard
    • nftables
    • Linux networking
    2 min read
  2. PKI-009

    Building trust from the root up

    PKI becomes easier to reason about when it is treated as trust-chain and lifecycle design, not as a sequence of certificate-generation commands.

    • OpenSSL
    • X.509
    • CRL
    • TLS
    2 min read
  3. K8S-031

    Delivery as reconciliation

    GitOps is most useful when desired state, reconciliation order, and ownership boundaries are explicit enough to operate under failure.

    • FluxCD
    • Helm
    • Kustomize
    • GitOps
    2 min read