Kubernetes ETCD Backup And Restore — Part 2: Restoring etcd and Recovering the Cluster

Question

Kubernetes ETCD Backup And Restore — Part 2: Restoring etcd and Recovering the Cluster

calendar_today1 hour ago • schedule15 min read

In Part 1, we created and validated an etcd snapshot while an nginx Deployment existed. Now we deliberately remove that workload, restore the snapshot, reconfigure the static etcd Pod, troubleshoot the problems that appear, and bring etcd back to a healthy state.

This is Part 2 of a two-part series.

The restore workflow covered:

deleting the test nginx Deployment;
making and diagnosing an incorrect restore command;
restoring the snapshot into a separate data directory;
understanding why the restored data should initially be kept separate;
handling filesystem and sudo mistakes;
changing the static etcd manifest;
understanding the impact of the static Pod manifest directory;
troubleshooting namespace and RBAC errors;
investigating an etcd Pod stuck in Pending;
checking kubelet;
restarting kubelet to reconcile the static Pod;
confirming that etcd returned to 1/1 Running;
documenting the final recovery workflow and lessons learned.

The important distinction is that the terminal evidence explicitly confirms successful snapshot restoration and recovery of the etcd Pod. It does not include a final k get all proving that the nginx Deployment reappeared after recovery, so this article does not claim that as independently verified.

17. Delete nginx

I deleted the Deployment:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k delete deploy nginx
deployment.apps "nginx" deleted from default namespace

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   76m

The nginx Deployment was gone.

This gave us the desired recovery scenario:

BEFORE BACKUP
    |
    +-- nginx Deployment exists
    |
    v
TAKE ETCD SNAPSHOT
    |
    v
DELETE nginx
    |
    v
nginx no longer exists
    |
    v
RESTORE SNAPSHOT
    |
    v
expect previous cluster state

18. First restore mistake: wrong snapshot filename

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db

The restore process immediately told me:

Error: open snapshot.db: no such file or directory

Again, the filename was wrong.

The actual snapshot was:

/opt/etcd-backup.db

This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file.

The first restore attempt was therefore not an etcd failure. It was simply a path/filename error.

19. Restore the snapshot into a separate data directory

Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory:

/var/lib/etcd-restore-from-backup

The command used was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup

The output included:

Deprecated: Use `etcdutl snapshot restore` instead.

and then:

restoring snapshot

followed by:

Trimming membership information from the backend...

and:

added member

Finally:

restored snapshot

This was the actual successful restore of the snapshot into the new data directory.

20. Why use a separate restore directory?

The original etcd data directory was:

/var/lib/etcd

The restored copy was:

/var/lib/etcd-restore-from-backup

This distinction is important.

It allows the restored data to be prepared without immediately destroying the existing data directory.

For a beginner, think of it like restoring a database backup into a new folder first:

Original:
    /var/lib/etcd

Restored:
    /var/lib/etcd-restore-from-backup

Only after the restored data exists do we configure etcd to use it.

21. More path mistakes while inspecting the restored data

I attempted:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cd /var/lib/data
bash: cd: /var/lib/data: No such file or directory

Again, the directory did not exist.

The restore directory was:

/var/lib/etcd-restore-from-backup

I also tried:

sudo cd /var/lib/etcd-restore-from-backup

and got:

sudo: cd: command not found
sudo: "cd" is a shell built-in command, it cannot be run directly.

This error is worth understanding.

cd is a shell built-in command. sudo normally executes an external command, so:

sudo cd ...

does not work.

Instead, because the directory required elevated permissions, I used:

sudo ls /var/lib/etcd-restore-from-backup

which showed:

member

That confirmed that the restore had created the expected etcd member directory structure.

22. Editing the static etcd manifest

The next step was to configure the static etcd pod to use the restored data directory.

The original manifest used:

--data-dir=/var/lib/etcd

The modified configuration used:

--data-dir=/var/lib/etcd-restore-from-backup

The corresponding hostPath was also changed to:

/var/lib/etcd-restore-from-backup

The resulting Pod description later confirmed:

--data-dir=/var/lib/etcd-restore-from-backup

and:

Path: /var/lib/etcd-restore-from-backup

This is how the restored database was connected to the etcd process.

23. Why moving the manifest files caused disruption

During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv * /tmp

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 0

The directory was empty.

I then restored the manifests:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv /tmp/*.yaml .

and confirmed:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 16
-rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml
-rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml
-rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml
-rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml

This demonstrates how sensitive the static pod manifest directory is.

When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear.

For this reason, moving files in /etc/kubernetes/manifests should be done carefully.

24. Another small typo

I accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get oi
error: the server doesn't have a resource type "oi"

The intended command was clearly not oi.

The correct command for Pods was:

k get po

This is a harmless error, but it illustrates another important habit:

When kubectl reports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken.

25. The RBAC error after the manifest changes

After the static pod manipulation, this happened:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po
Error from server (Forbidden): pods is forbidden: User "kubernetes-admin" cannot list resource "pods" in API group "" in the namespace "default"

This was different from the earlier errors.

This time, the Kubernetes API server was responding, but the authenticated user was being denied access.

The command:

sudo k get po

was also not a solution:

sudo: k: command not found

The reason is that k was a shell alias, and aliases normally are not available to sudo in that way.

More importantly, sudo does not automatically solve Kubernetes RBAC problems.

I then checked the system namespace explicitly:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system

This returned the control-plane Pods.

That helped separate the problem from the actual etcd state.

26. etcd was now Pending

The output showed:

etcd-osboxes   0/1   Pending

I first tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po etcd-osboxes
Error from server (NotFound): pods "etcd-osboxes" not found

Again, the namespace mattered.

The correct command was:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The description showed the important configuration:

Status: Pending

but also:

Image: registry.k8s.io/etcd:3.6.8-0

and:

--data-dir=/var/lib/etcd-restore-from-backup

The Pod also showed the restored directory mounted:

/var/lib/etcd-restore-from-backup from etcd-data

and:

Path: /var/lib/etcd-restore-from-backup

The kubelet events showed:

Normal  Pulled   ...
Normal  Created  ...
Normal  Started  ...

So the container had actually been created and started even though the Kubernetes status remained Pending.

This is an important troubleshooting clue:

Do not rely on one status field alone. Read the full Pod description and events.

27. Trying to read etcd logs

I tried:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs etcd-osboxes -n kube-system
Error from server (BadRequest): container "etcd" in pod "etcd-osboxes" is not available

I also accidentally ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs

which returned:

error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command

The second error was simply because kubectl logs requires a Pod or workload name.

The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval.

I then checked the Pod in wide format:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get pod etcd-osboxes -n kube-system -o wide
NAME           READY   STATUS    RESTARTS   AGE     IP       NODE      NOMINATED NODE   READINESS GATES
etcd-osboxes   0/1     Pending   0          4m58s   <none>   osboxes   <none>           <none>

The lack of a Pod IP was another indication that the Pod was not fully ready.

28. Inspecting kubelet

Since etcd was a static Pod, the kubelet was a key component to inspect.

I ran:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl status kubelet --no-pager

The result showed:

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled)
     Active: active (running)

The kubelet was running.

The logs also showed repeated activity around the time of the problem.

At this point, rather than making more changes to the etcd manifest, I restarted kubelet:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl restart kubelet

Then:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ systemctl daemon-reload

29. The recovery

After restarting kubelet, I checked the system namespace again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-74c68c8864-fjtnv   1/1     Running   5 (7m47s ago)   95m
calico-node-dzlcp                          1/1     Running   0               95m
coredns-589f44dc88-l522w                   1/1     Running   0               101m
coredns-589f44dc88-nzsh9                   1/1     Running   0               101m
etcd-osboxes                               1/1     Running   0               6m51s
kube-apiserver-osboxes                     1/1     Running   0               101m
kube-controller-manager-osboxes            1/1     Running   1 (7m32s ago)   101m
kube-proxy-d8krd                           1/1     Running   0               101m
kube-scheduler-osboxes                     1/1     Running   1               101m

This was the desired state.

Most importantly:

etcd-osboxes   1/1   Running

30. Confirming the restored etcd configuration

I then described the etcd Pod again:

┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes

The important portion was:

Status: Running
IP: 10.0.2.15

The etcd container was:

State: Running
Ready: True
Restart Count: 0

And the command still showed:

--data-dir=/var/lib/etcd-restore-from-backup

The mounted volume was:

/var/lib/etcd-restore-from-backup from etcd-data

with the host path:

/var/lib/etcd-restore-from-backup

This confirmed that etcd was now running using the restored data directory.

31. What happened to the nginx Deployment?

The exercise was designed around the snapshot that was taken while nginx existed.

The important sequence was:

1. Create nginx Deployment
2. Wait for nginx to become Running
3. Take etcd snapshot
4. Delete nginx
5. Restore the snapshot
6. Point etcd at restored data
7. Restart kubelet
8. Confirm etcd is Running

The backup validation showed a snapshot revision of:

with:

918 total keys
3.9 MB total size

The restore operation successfully rebuilt an etcd data directory.

The final terminal state confirmed that etcd was successfully running from:

/var/lib/etcd-restore-from-backup

The terminal session provided here does not contain a final k get all after the successful etcd recovery showing nginx reappearing, so I would not claim that nginx restoration was independently verified from the terminal evidence.

That distinction matters.

The exercise successfully demonstrated:

snapshot creation;
snapshot validation;
snapshot restoration;
switching the static etcd Pod to restored data;
recovery of the etcd Pod.

The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot.

Reference: Validate the Backup Before Restoring

The session showed that etcdctl snapshot status is deprecated and recommended etcdutl.

The working validation command was:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Example output from the session:

┌──────────┬──────────┬────────────┬────────────┐
│   HASH   │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │     6934 │        918 │     3.9 MB │
└──────────┴──────────┴────────────┴────────────┘

This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot.

Reference: The Restore Command

The restore performed successfully with:

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

The terminal explicitly reported:

Deprecated: Use `etcdutl snapshot restore` instead.

So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly.

Troubleshooting Lessons From the Lab

Error 1: Wrong directory

cd /etc/kkubenetes/manifests

and:

cd /etc/kubenetes/manifests

failed because of spelling mistakes.

Fix

cd /etc/kubernetes/manifests

Error 2: Docker container not found

docker exec -it osboxes bash

returned:

No such container: osboxes

Lesson

A Kubernetes node name is not automatically a Docker container name.

Error 3: Permission denied reading etcd files

cat etcd.yaml

returned:

Permission denied

Fix

sudo cat etcd.yaml

The same principle applied to the protected etcd private key.

Error 4: Invalid etcdctl flag

---endpoint=...

returned:

Error: bad flag syntax

Fix

Use the correctly spelled flag:

--endpoints=...

Error 5: Private key permission denied

Error: open /etc/kubernetes/pki/etcd/server.key: permission denied

Fix

Run the etcd client command with sufficient privileges:

sudo etcdctl ...

Error 6: Wrong snapshot filename

snapshot.db

did not exist.

The actual file was:

/opt/etcd-backup.db

Lesson

Always verify the exact backup path before restoring.

Error 7: `etcdctl snapshot status` deprecated

The client reported:

Deprecated: Use `etcdutl snapshot status` instead.

Fix

Install/use etcdutl and run:

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Error 8: `etcdutl` not installed

The command:

etcdutl

was initially unavailable.

Fix

The system suggested installing etcd-server, which provided the utility in this environment.

Error 9: `sudo cd`

This failed:

sudo cd /var/lib/etcd-restore-from-backup

with:

sudo: cd: command not found

Lesson

cd is a shell built-in.

Use commands such as:

sudo ls /var/lib/etcd-restore-from-backup

or start a privileged shell if you genuinely need one.

Error 10: Pod name not found

This:

k describe po etcd-osboxes

returned:

pods "etcd-osboxes" not found

because the command defaulted to the default namespace.

Fix

k describe po -n=kube-system etcd-osboxes

Error 11: `kubectl logs` without a Pod

k logs

returned:

POD or TYPE/NAME is a required argument for the logs command

Fix

Provide the resource:

k logs <pod-name>

and specify the namespace when needed.

Error 12: `kubectl` Forbidden

After the control-plane changes:

Error from server (Forbidden): pods is forbidden:
User "kubernetes-admin" cannot list resource "pods"

This was an authorization/RBAC problem rather than a shell permission problem.

sudo did not solve it because Kubernetes authorization is separate from Linux sudo.

Error 13: etcd remained Pending

The etcd Pod stayed:

0/1 Pending

even though the container had been created and started according to the events.

The investigation included:

k describe po -n=kube-system etcd-osboxes

and:

sudo systemctl status kubelet --no-pager

The practical recovery step that resolved the state in this lab was:

sudo systemctl restart kubelet

After that:

etcd-osboxes   1/1   Running

Why Static Pods Matter in This Exercise

The etcd Pod was not created with a normal Deployment such as:

kind: Deployment

Instead, it was defined by:

/etc/kubernetes/manifests/etcd.yaml

That makes it a static Pod.

The kubelet watches the manifest directory and manages the Pod locally.

This explains several things that happened during the exercise:

changing etcd.yaml affected etcd;
temporarily moving manifests out of the directory affected the control plane;
restarting kubelet caused it to re-read and reconcile the manifests;
etcd could be controlled without creating a normal Kubernetes Deployment.

For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane.

A Safer Mental Model for etcd Restore

Do not think of restore as:

"copy backup file over etcd"

A better mental model is:

ETCD SNAPSHOT
     |
     v
restore snapshot
     |
     v
new etcd data directory
     |
     v
configure etcd to use restored directory
     |
     v
kubelet recreates/restarts static etcd Pod
     |
     v
etcd becomes healthy
     |
     v
Kubernetes API becomes available against restored state

This is why the exercise used:

/var/lib/etcd-restore-from-backup

instead of immediately destroying:

/var/lib/etcd

What I Would Do Differently Next Time

The lab worked, but it also exposed several areas where the workflow could be cleaner.

1. Check the exact paths first

Before typing a long command:

ls -l /etc/kubernetes/pki/etcd/
ls -l /opt/etcd-backup.db

This avoids guessing filenames and permissions.

2. Inspect the manifest before changing it

First:

sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup

Then edit the manifest.

That gives you a known-good copy if something goes wrong.

3. Prefer the current etcd utilities

The terminal explicitly showed that:

etcdctl snapshot status

is deprecated and recommended:

etcdutl snapshot status

Likewise, the restore output recommended:

etcdutl snapshot restore

4. Avoid unnecessary movement of all manifests

This command:

sudo mv * /tmp

is powerful and potentially disruptive.

Moving all control-plane manifests affects more than etcd.

A safer approach is to modify only the required manifest and keep backups of the original.

5. Verify each stage

After every major step:

k get pods -A

or:

k get pods -n kube-system

and inspect:

k describe pod <pod> -n kube-system

if anything looks abnormal.

Compact Recovery Checklist

For a kubeadm single-control-plane lab similar to this one:

Identify etcd

k get pods -n kube-system | grep etcd

Inspect the manifest

cd /etc/kubernetes/manifests
sudo cat etcd.yaml

Identify:

--data-dir
--listen-client-urls
--cert-file
--key-file
--trusted-ca-file

Create snapshot

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate snapshot

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Restore to a separate directory

sudo etcdutl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Configure etcd to use the restored directory

Update:

--data-dir=/var/lib/etcd-restore-from-backup

and the corresponding hostPath.

Restart/reconcile kubelet if required

sudo systemctl restart kubelet

Verify

k get pods -n kube-system

Look for:

etcd-osboxes   1/1   Running

What This Lab Actually Taught Me

The most valuable part of this exercise was not memorizing one command.

It was learning how the pieces connect:

Kubernetes
    |
    +-- kube-apiserver
    |
    +-- kube-controller-manager
    |
    +-- kube-scheduler
    |
    +-- kubelet
            |
            +-- static Pod manifests
                    |
                    +-- etcd
                            |
                            +-- /var/lib/etcd

The backup process interacted directly with etcd:

etcd
  |
  +-- TLS certificates
  |
  +-- etcdctl
  |
  +-- snapshot
  |
  +-- /opt/etcd-backup.db

The restore process then created:

/var/lib/etcd-restore-from-backup

and the static Pod was changed to consume that restored data.

Finally, kubelet reconciled the static Pod and etcd returned to:

1/1 Running

Final Takeaway

An etcd backup is not just another Kubernetes YAML export.

The core workflow is:

1. Verify etcd is healthy.
2. Find the etcd endpoint and TLS certificates.
3. Take an etcd snapshot.
4. Validate the snapshot.
5. Restore the snapshot into a separate data directory.
6. Configure the etcd static Pod to use the restored data.
7. Let kubelet reconcile the static Pod.
8. Verify etcd and the Kubernetes control plane.

The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt.

There were:

spelling mistakes;
incorrect flags;
permission problems;
wrong filenames;
deprecated commands;
missing utilities;
namespace mistakes;
RBAC errors;
attempts to use sudo where it did not apply;
a static etcd Pod that remained Pending;
and finally a kubelet restart that brought etcd back to Running.

Those failures were not separate from the learning experience. They were the learning experience.

The most useful troubleshooting habit from this lab is simple:

When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result.

That is the difference between memorizing Kubernetes commands and actually learning how the control plane works.

Terminal command reference from the lab

Cluster health

k get pods -A
k get pods -A -w
k get nodes
k get all
k get all -A

Static Pod manifests

cd /etc/kubernetes/manifests
ls
sudo cat etcd.yaml

Install tooling

sudo apt install etcd-client
sudo apt install etcd-server

Configure etcdctl

export ETCDCTL_API=3

Backup

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/etcd-backup.db

Validate

sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db

Test workload

k create deploy nginx --image=nginx
k get po
k get po -w
k describe po nginx-7f8fbb96d-9f7wm
k get all

Delete workload

k delete deploy nginx

Restore

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore-from-backup

Troubleshoot etcd

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
k get pod etcd-osboxes -n kube-system -o wide
k logs etcd-osboxes -n kube-system
sudo systemctl status kubelet --no-pager
sudo systemctl restart kubelet

Final verification

k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes

Closing note

This was a lab on a single-node kubeadm cluster. In a production environment, etcd backup and restore requires much stricter operational controls, including secure off-host backup storage, tested restore procedures, careful handling of TLS credentials, cluster topology awareness, and a clearly defined disaster-recovery procedure.

A backup is only useful if you can successfully restore it.

Take the backup. Validate it. Practice the restore.

🔥 Join developers growing publicly

Share your knowledge, build in public, and grow your developer presence with a global community.

Join CoderLegion

chevron_left

Commenters (This Week)

Contribute meaningful comments to climb the leaderboard and earn badges!

	Kubernetes ETCD Backup And Restore — Part 1: Creating and Validating the Backup AYANFE - Aug 17
	Kamal vs Kubernetes: An Honest Comparison for Teams Who Don’t Need 1,000 Services Alexandre Vazquez - Jul 24
	Kubernetes Network Policies Explained (and How I Actually Made Them Work) AYANFE - Jul 9
	Understanding Kubernetes ClusterRoles and ClusterRoleBindings AYANFE - Jul 7
	Understanding Kubernetes RBAC: Roles, RoleBindings, and and Client Certificates AYANFE - Jul 6

Welcome to Coder Legion

Connect with 4,835 amazing developers

Don't have an account? Sign up

OR

Kubernetes ETCD Backup And Restore — Part 2: Restoring etcd and Recovering the Cluster

17. Delete nginx

18. First restore mistake: wrong snapshot filename

19. Restore the snapshot into a separate data directory

20. Why use a separate restore directory?

21. More path mistakes while inspecting the restored data

22. Editing the static etcd manifest

23. Why moving the manifest files caused disruption

24. Another small typo

25. The RBAC error after the manifest changes

26. etcd was now Pending

27. Trying to read etcd logs

28. Inspecting kubelet

29. The recovery

30. Confirming the restored etcd configuration

31. What happened to the nginx Deployment?

Reference: Validate the Backup Before Restoring

Reference: The Restore Command

Troubleshooting Lessons From the Lab

Error 1: Wrong directory

Fix

Error 2: Docker container not found

Lesson

Error 3: Permission denied reading etcd files

Fix

Error 4: Invalid etcdctl flag

Fix

Error 5: Private key permission denied

Fix

Error 6: Wrong snapshot filename

Lesson

Error 7: etcdctl snapshot status deprecated

Fix

Error 8: etcdutl not installed

Fix

Error 9: sudo cd

Lesson

Error 10: Pod name not found

Fix

Error 11: kubectl logs without a Pod

Fix

Error 12: kubectl Forbidden

Error 13: etcd remained Pending

Why Static Pods Matter in This Exercise

A Safer Mental Model for etcd Restore

What I Would Do Differently Next Time

1. Check the exact paths first

2. Inspect the manifest before changing it

3. Prefer the current etcd utilities

4. Avoid unnecessary movement of all manifests

5. Verify each stage

Compact Recovery Checklist

Identify etcd

Inspect the manifest

Identify:

Create snapshot

Validate snapshot

Restore to a separate directory

Configure etcd to use the restored directory

Restart/reconcile kubelet if required

Verify

What This Lab Actually Taught Me

Final Takeaway

Terminal command reference from the lab

Cluster health

Static Pod manifests

Install tooling

Configure etcdctl

Backup

Validate

Test workload

Delete workload

Restore

Troubleshoot etcd

Final verification

Closing note

Error 7: `etcdctl snapshot status` deprecated

Error 8: `etcdutl` not installed

Error 9: `sudo cd`

Error 11: `kubectl logs` without a Pod

Error 12: `kubectl` Forbidden