Kubernetes ETCD Backup And Restore — Part 1: Creating and Validating the Backup

Question

This is Part 1 of a two-part series.

In this part, the focus is entirely on the backup side of the disaster-recovery exercise:

• understanding what etcd does;
• verifying the Kubernetes control plane;
• locating the static etcd manifest;
• understanding the etcd endpoint and TLS certificates;
• installing etcdctl and etcdutl;
• dealing with command-line mistakes and permission errors;
• creating a valid etcd snapshot;
• validating the snapshot;
• creating an nginx Deployment;
• taking a second snapshot containing that workload;
• preparing the cluster for the restore exercise in Part 2.

The actual restoration, static Pod reconfiguration, kubelet troubleshooting, and recovery are covered in Part 2.

A beginner-friendly, hands-on walkthrough of backing up and restoring etcd in a kubeadm-based Kubernetes cluster — including the mistakes, errors, debugging steps, and the terminal commands that actually happened during the lab.

Introduction

If you are learning Kubernetes, you will eventually hear a simple but important statement:

etcd is the database of a Kubernetes cluster.

That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster.

In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes.

The exercise covered:

• identifying the etcd pod and confirming that the control plane was healthy;
• locating the static etcd manifest;
• inspecting the etcd configuration;
• installing etcdctl and etcdutl;
• creating an etcd snapshot;
• validating the snapshot;
• deliberately creating an nginx Deployment so there was something meaningful to restore;
• deleting the Deployment;
• restoring the etcd snapshot into a separate data directory;
• changing the static etcd manifest to use the restored data;
• troubleshooting an etcd pod that remained Pending;
• restarting kubelet;
• confirming that etcd eventually returned to Running.

The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal.

The terminal session showed the etcd pod running before the backup, with the control plane components also healthy. Later, after changing the etcd static pod configuration, etcd became Pending; the investigation eventually led to restarting kubelet, after which etcd became 1/1 Running. The original terminal output is the source for the commands and results reproduced throughout this article.

---

1. First: What is etcd?

Before touching commands, it helps to understand what we are backing up.

Kubernetes has many components, but the important relationship for this exercise is roughly:

kubectl
   |
   v
kube-apiserver
   |
   v
  etcd
   |
   +---- Kubernetes cluster state

The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state.

That state includes information such as:

• Kubernetes objects;
• Deployments;
• Pods and their specifications;
• Services;
• ConfigMaps;
• Secrets;
• cluster configuration and metadata.

A useful mental model for a beginner is:

The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted.

This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML.

---

2. Verify that the cluster is healthy first

Before taking a backup, I first checked the cluster.

Terminal

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-74c68c8864-fjtnv   0/1     ContainerCreating   0          3m10s
kube-system   calico-node-dzlcp                          0/1     Init:2/3             0          3m10s
kube-system   coredns-589f44dc88-l522w                   0/1     ContainerCreating   0          8m56s
kube-system   coredns-589f44dc88-nzsh9                   0/1     ContainerCreating   0          8m56s
kube-system   etcd-osboxes                               1/1     Running             0          9m6s
kube-system   kube-apiserver-osboxes                     1/1     Running             0          9m9s
kube-system   kube-controller-manager-osboxes            1/1     Running             0          9m5s
kube-system   kube-proxy-d8krd                           1/1     Running             0          8m56s
kube-system   kube-scheduler-osboxes                     1/1     Running             0          9m8s

The most important observation here is:

etcd-osboxes  1/1  Running

The API server, controller manager, scheduler and kube-proxy were also running.

Some networking components were still starting, so I watched the pods:

┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A -w

Eventually CoreDNS and Calico became ready.

I then checked the node:

┌──(osboxes㉿osboxes)-[~]
└─$ k get nodes
NAME      STATUS   ROLES           AGE   VERSION
osboxes   Ready    control-plane   10m   v1.36.3

At this point the single control-plane node was Ready.

The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy.

---

3. An important distinction: Kubernetes YAML backup vs etcd backup

Before working directly with etcd, I also ran:

┌──(osboxes㉿osboxes)-[~]
└─$ k get all -A -o yaml > backup.yaml

This produced:

┌──(osboxes㉿osboxes)-[~]
└─$ ls
backup.yaml  Documents  kubectl  Pictures  Public  Videos
Desktop      Downloads  Music    Projects  Templates  VM-Share

This is useful, but it is not the same thing as an etcd snapshot.

The command:

k get all -A -o yaml > backup.yaml

exports Kubernetes resources returned by kubectl get all.

An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself.

For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact.

---

4. Finding the etcd static pod

This cluster was created with kubeadm, so etcd was running as a static pod.

I initially made a typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kkubenetes/manifests
bash: cd: /etc/kkubenetes/manifests: No such file or directory

The problem was simply the spelling:

kkubenetes

instead of:

kubernetes

I then made another typo:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubenetes/manifests
bash: cd: /etc/kubenetes/manifests: No such file or directory

Again, the correct directory is:

/etc/kubernetes/manifests

Eventually:

┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubernetes/manifests

and:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml

This directory is extremely important on a kubeadm control-plane node.

The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running.

That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched.

---

5. Another wrong assumption: trying to use Docker

I also tried:

┌──(osboxes㉿osboxes)-[~]
└─$ docker exec -it osboxes bash
Error response from daemon: No such container: osboxes

This failed because the Kubernetes node name osboxes is not necessarily a Docker container name.

In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with:

containerd://

So the important lesson is:

Do not assume that a Kubernetes node or Pod name is a Docker container name.

If you need to inspect a Kubernetes container, first determine which container runtime the node is using.

---

6. Inspecting the etcd manifest

The first attempt to read the manifest without elevated privileges also failed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cat etcd.yaml
cat: etcd.yaml: Permission denied

The solution was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo cat etcd.yaml

The manifest revealed several important values.

For example:

- --advertise-client-urls=https://10.0.2.15:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379

It also showed that the etcd data directory was originally:

/var/lib/etcd

and the TLS certificates were under:

/etc/kubernetes/pki/etcd/

This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates.

---

7. Install etcdctl

The machine did not initially have the etcd client installed, so I installed it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo apt install etcd-client

The package installed successfully:

Installing:
  etcd-client

Summary:
  Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205

I then checked the snapshot command:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ETCDCTL_API=3 etcdctl snapshot

The output showed:

COMMANDS:
    restore  Restores an etcd member snapshot to an etcd directory
    save     Stores an etcd node backend snapshot to a given file
    status   [deprecated] Gets backend snapshot status of a given file

The key command for the backup is:

etcdctl snapshot save

---

8. ETCDCTL_API=3

I also explicitly exported:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ export ETCDCTL_API=3

For the lab, this made it explicit that etcdctl should use the v3 API.

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl snapshot

again showed the snapshot subcommands.

For a beginner, the important idea is:

ETCDCTL_API=3

selects the etcd API version used by the client.

---

9. Creating the first etcd snapshot — and making a typo

I attempted the backup with:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/etcd/server.crt \
> --key=/etc/kubernetes/pki/etcd/server.key \
> snapshot save /opt/etcd-backup.db

The error was:

Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379

The mistake was the number of hyphens.

I typed:

---endpoint

instead of:

--endpoints

There were actually two problems worth noticing:

1. There were three hyphens.
2. The correct flag is --endpoints, plural.

This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser.

---

10. The next error: permission denied

I corrected the flag and tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The command reached the certificate files but failed with:

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

This made sense.

The etcd private key is protected, and the regular user did not have permission to read it.

The solution was to run the command with sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

This time the backup succeeded.

---

11. The successful etcd backup

The successful command produced:

{"level":"info","ts":"2026-08-13T12:11:59.649233-0400","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/opt/etcd-backup.db.part"}
{"level":"info","ts":"2026-08-13T12:11:59.660384-0400","logger":"client","caller":"v3/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-13T12:11:59.660467-0400","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-08-13T12:11:59.831074-0400","logger":"client","caller":"v3/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-13T12:11:59.894020-0400","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2026-08-13T12:11:59.894158-0400","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/opt/etcd-backup.db"}
Snapshot saved at /opt/etcd-backup.db

The resulting backup file was:

/opt/etcd-backup.db

The snapshot was approximately:

3.9 MB

This was the first major milestone.

---

12. Validating the snapshot

I initially tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --write-out=table snapshot status snapshot.db

The client responded:

Deprecated: Use `etcdutl snapshot status` instead.

Error: stat snapshot.db: no such file or directory

There were two lessons here.

Lesson 1: The filename was wrong

The backup was saved as:

/opt/etcd-backup.db

not:

snapshot.db

Lesson 2: The command was deprecated

The output explicitly recommended:

etcdutl snapshot status

So I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db

but etcdutl was not installed.

The system suggested:

sudo apt install etcd-server

I installed the package.

After installation, I still used the wrong filename once:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Error: stat snapshot.db: no such file or directory

Then I used the correct path:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db

This produced:

Error: open /opt/etcd-backup.db: permission denied

Again, the issue was file permissions.

So I used sudo:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

The result was:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ a633855e │     5973 │       1298 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This confirmed that the snapshot was readable and contained etcd data.

---

13. Why create an nginx Deployment?

A backup is much more meaningful if we have something we can verify after restoring it.

At this point, I created an nginx Deployment.

My first attempt was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx
error: required flag(s) "image" not set

The command was missing the container image.

I then tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy --image=nginx
error: exactly one NAME is required, got 0
See 'kubectl create deployment -h' for help and examples

Now the image was present, but the Deployment name was missing.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx --image=nginx
deployment.apps/nginx created

This is a good beginner lesson:

kubectl create deployment <NAME> --image=<IMAGE>

So:

kubectl create deployment nginx --image=nginx

means:

• create a Deployment;
• call it nginx;
• use the nginx container image.

---

14. Waiting for the nginx Pod

Immediately after creation:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS              RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   0/1     ContainerCreating   0          4s

The Pod remained in:

ContainerCreating

I watched it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -w

After waiting, I checked again.

The Pod eventually became:

nginx-7f8fbb96d-9f7wm   1/1   Running   0   3m16s

The describe output explained why it initially took time:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ---   ----               -------
  Normal  Scheduled  83s   default-scheduler  Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes
  Normal  Pulling    82s   kubelet            spec.containers{nginx}: Pulling image "nginx"

The Pod had been scheduled successfully. The kubelet was simply pulling the image.

This is an important troubleshooting principle:

ContainerCreating does not automatically mean something is broken. Check the Pod events before changing anything.

Eventually:

k get po
NAME                    READY   STATUS    RESTARTS   AGE
nginx-7f8fbb96d-9f7wm   1/1     Running   0          3m16s

---

15. Take another backup after the nginx Deployment exists

This step is important because the first snapshot was taken before the nginx Deployment was created.

After nginx was running, I created another snapshot:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db

The snapshot was again:

3.9 MB

and the command ended with:

Snapshot saved at /opt/etcd-backup.db

I validated it:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Result:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

Notice that the revision changed.

The backup now represented a later state of the cluster.

---

16. Verify the state we want to recover

At this point:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-7f8fbb96d-9f7wm   1/1     Running   0          4m1s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1            4m2s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-7f8fbb96d   1         1         1       4m2s

The important object is:

deployment.apps/nginx

It was healthy.

Now the test was simple:

If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken?

---