当前位置：首页 > news >正文

k8s的数据库etcd报 etcdserver: mvcc: database space exceeded的处理办法

news 2024/12/4 8:57:42

一.问题现象

公司的k8s集群的etcd配置是默认配置，其磁盘配置为2GB的配额，目前出现了数据写入失败的情况，报错Error: etcdserver: mvcc: database space exceeded。

二.排查步骤

以下排查路径涉及到的证书，根据自己的实际情况指定。

1.查看etcd的endpoint

 etcdctl --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key --endpoints=https://10.20.12.101:2379,https://10.20.12.102:2379,https://10.20.12.103:2379 endpoint status  -w table

2.备份数据

 etcdctl --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key --endpoints=https://10.20.12.101:2379,https://10.20.12.102:2379,https://10.20.12.103:2379 snapshot save /var/lib/etcd/my-snapshot.db

3.获取当前版本

 etcdctl --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key --endpoints={$endpoints} endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*')

4.压缩所有旧版本

 etcdctl --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key --endpoints={$endpoints} compact $rev

4.执行碎片清理

 etcdctl --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key --endpoints=https://10.20.12.101:2379,https://10.20.12.102:2379,https://10.20.12.103:2379 defrag

5.解除告警

 etcdctl --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt --key=/var/lib/rancher/rke2/server/tls/etcd/client.key --endpoints={$endpoints} alarm disarm

以上操作可以减少etcd的数据文件大小，解决mvcc: database space exceeded。但etcd的磁盘配额的大小还是2G。修改etcd磁盘配额的方式如下：

在etcd配置文件中找到或添加--quota-backend-bytes参数，其值设置为：8589934592，这是8GB的字节数。

比如通过kubeadm安装的k8s，其etcd的配置文件在/etc/kubernetes/manifests/etcd.yaml,添加- --quota-backend-bytes=8589934592，添加之后重启服务

apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: etcd
tier: control-plane
name: etcd
namespace: kube-system
spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://10.10.19.63:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --initial-advertise-peer-urls=https://10.10.19.63:2380
- --initial-cluster=k8s-master-01=https://10.10.19.61:2380,k8s-master-02=https://10.10.19.62:2380,k8s-master-03=https://10.10.19.63:2380
- --initial-cluster-state=existing
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.10.19.63:2379
- --listen-peer-urls=https://10.10.19.63:2380
- --name=k8s-master-03
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

- --quota-backend-bytes=8589934592
image: k8s.gcr.io/etcd:3.3.10
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /bin/sh
- -ec
- ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
get foo
failureThreshold: 8
initialDelaySeconds: 15
timeoutSeconds: 15
name: etcd
resources: {}
volumeMounts:
- mountPath: /var/lib/etcd
name: etcd-data
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs
- hostPath:
path: /var/lib/etcd
type: DirectoryOrCreate
name: etcd-data
status: {}