Kubernetes服务注册到consul流程实践

---

---

前言

在云原生架构中，服务注册与发现是微服务架构的核心组件之一。随着Kubernetes成为容器编排的事实标准，如何将传统服务注册中心与Kubernetes原生服务发现机制相结合，成为企业级应用面临的重要课题。本文详细介绍如何利用HashiCorp Consul实现Kubernetes服务的自动注册与发现，通过开发定制化工具解决实际生产环境中的服务治理难题,并结合Prometheus实现监控

---

目的: 通过开发一个轻量级的 consulctl 工具，配合Kubernetes的生命周期钩子preStop函数+sidecar边车容器实现pod启动时自动注册到consul、pod销毁时自动从consul剔除

架构图示意

一、环境准备

名称	作用
k8s1.28集群	用于部署consul及测试pod
docker	镜像构建
harbor	镜像存储

二、consul部署

以NFS为底层存储,通过storageclass实现consul的动态pv

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: consul-storage #要与statefulset文件中的名称一致
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:archiveOnDelete: "true"
reclaimPolicy: Retai

1.yaml示例

如下（示例）：consul开启ACL token,避免后续未授权登录漏洞

生成一个64位字符串用做于consul的master token

[root@k8s-master consul]# uuidgen
9bfbe81f-2648-4673-af14-d13e0a170050

准备consul的配置文件,开启acl、配置token,生成configmap

[root@k8s-master consul]# cat consul-config.json 
{"datacenter":"dc8","primary_datacenter":"dc8","acl":{"enabled":true, #开启ACL"default_policy":"deny","enable_token_persistence":true,"enable_key_list_policy":true,"tokens":{"master":"9bfbe81f-2648-4673-af14-d13e0a170050" #上述生成}}
}[root@k8s-master consul]# kubectl create configmap global-config -n middleware --from-file ./consul-config.json

statefulset文件

apiVersion: apps/v1
kind: StatefulSet
metadata:name: consulnamespace: middleware
spec:serviceName: consul-serverreplicas: 1selector:  # 新增selector部分matchLabels:component: consul  # 必须与template.metadata.labels中的标签匹配template:metadata:labels:component: consul  # 这里已经正确设置spec:tolerations:  # 新增污点容忍- key: "node-role.kubernetes.io/control-plane"operator: "Exists"effect: "NoSchedule"affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:  # 改为软性要求- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: componentoperator: Invalues:- consultopologyKey: kubernetes.io/hostnamecontainers:- name: consulimage: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/hashicorp/consul:latestargs:- "agent"- "-server"- "-bootstrap-expect=1" #此处的数量要与副本数相同- "-ui"- "-data-dir=/etc/consul/data"- "-config-file=/etc/consul/config/consul-config.json"- "-bind=0.0.0.0"- "-client=0.0.0.0"- "-advertise=$(PODIP)"- "-retry-join=consul-server.$(NAMESPACE).svc.cluster.local" - "-domain=cluster.local"- "-disable-host-node-id"livenessProbe:httpGet:path: /v1/status/leaderport: 8500initialDelaySeconds: 20  # Consul启动较慢需要更长的初始延迟periodSeconds: 20failureThreshold: 3readinessProbe:httpGet:path: /v1/status/peersport: 8500initialDelaySeconds: 15periodSeconds: 5successThreshold: 1failureThreshold: 2resources:requests:memory: "256Mi"cpu: "100m"limits:memory: "512Mi"cpu: "500m"volumeMounts:- name: datamountPath: /etc/consul/data- name: configmountPath: /etc/consul/config- name: timemountPath: /etc/localtimeenv:- name: PODIPvalueFrom:fieldRef:fieldPath: status.podIP- name: NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespaceports:- containerPort: 8500name: ui-port- containerPort: 8400name: alt-port- containerPort: 53name: udp-port- containerPort: 8443name: https-port- containerPort: 8080name: http-port- containerPort: 8301name: serflan- containerPort: 8302name: serfwan- containerPort: 8600name: consuldns- containerPort: 8300name: servervolumes:- name: configconfigMap:name: consul- name: timehostPath:path: /usr/share/zoneinfo/Asia/Shanghai#- name: data#emptyDir: {}volumeClaimTemplates:  # 持久化存储声明模板- metadata:name: dataspec:accessModes: [ "ReadWriteOnce" ]storageClassName: "consul-storage"  # 根据实际存储类调整resources:requests:storage: 1Gi #自行调整

无头service文件和nodeport类型的svc文件

apiVersion: v1
kind: Service
metadata:name: consul-server  # 必须与 StatefulSet 的 serviceName 一致namespace: middleware
spec:clusterIP: None  # Headless Service 特征ports:- port: 8300name: server-rpcselector:component: consul  # 匹配 Pod 标签
---
apiVersion: v1
kind: Service
metadata:name: consul-uinamespace: middlewarelabels:component: consul
spec:type: NodePortports:- name: httpport: 8500targetPort: 8500nodePort: 30850  # 自定义端口范围 30000-32767selector:component: consul  # 必须匹配 StatefulSet 的 Pod 标签

2.consul部署验证

代码如下（示例）：

[root@k8s-master consul]# kubectl get pod,svc -n middleware 
NAME           READY   STATUS    RESTARTS   AGE
pod/consul-0   1/1     Running   0          11mNAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/consul-server   ClusterIP   None           <none>        8300/TCP         46h
service/consul-ui       NodePort    10.96.23.206   <none>        8500:30850/TCP   46h

访问验证

三、consulctl工具实现

1.核心功能

使用golang 调用consul包"github.com/hashicorp/consul/api" 实现pod的注册和销毁

注册逻辑包含以下关键步骤:1. 解析Pod环境变量（名称、IP、命名空间）2. 构造service名称 (采用pod所在的名称空间)3. 构造服务ID（采用 <pod-name>-<port> 格式）4. 添加丰富的元数据（监控类型、所属项目等）5. 调用Consul API完成注册

采用双重注销机制确保可靠性:1. 当执行delete pod或者delete -f xxx.yaml时 优先通过Agent接口注销2. 失败后尝试Catalog接口注销3. 收集所有错误信息统一处理

//核心功能定义
type ServiceManager struct {client *api.Client
}func NewServiceManager(consulAddr, token string) (*ServiceManager, error) {config := api.DefaultConfig()config.Address = strings.TrimSpace(consulAddr)config.Token = strings.TrimSpace(token)config.Transport.TLSHandshakeTimeout = defaultTimeoutclient, err := api.NewClient(config)if err != nil {return nil, fmt.Errorf("consul client init failed: %w", err)}return &ServiceManager{client: client}, nil
}func (sm *ServiceManager) Register(service *api.AgentServiceRegistration) error {return sm.client.Agent().ServiceRegister(service)
}func (sm *ServiceManager) Deregister(serviceID string) error {return sm.client.Agent().ServiceDeregister(serviceID)
}

2.注册到consul的标签及元数据

pod注册到consul中后tag是container和对应的pod名称，元数据标签包含pod名称、podIP、端口、所在节点名称、监控类型、标题

Tags: []string{"container",podName,},
Meta: map[string]string{"podName":     podName,"podIP":       podIP,"podPort":     port,"hostNode":    hostNode,"monitorType": monitorType,"project":     project,
}

3.consulctl工具使用示例

注册

[root@k8s-master consul]# ./consulctl register "http://consulIP:consulPort" "consul的ACL TOKEN" "微服务的容器端口" "monitorType(自行命名)" "project(自行命名)"

销毁

[root@k8s-master consul]# ./consulctl deregister "http://consulIP:consulPort" "consul的ACL TOKEN" "微服务的容器端口" "consul的节点名称(理解为conusl的pod名称)"

四、通过Dockerfile构建consulctl工具镜像

结合golang多阶段构建

FROM swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/golang:1.23.7 AS builder
WORKDIR /app
COPY go.mod go.sum ./
COPY vendor  ./
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -mod=vendor -o /usr/local/bin/consulctlFROM swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/alpine:3.18.6
WORKDIR /usr/local/bin/
COPY --from=builder /usr/local/bin/consulctl /usr/local/bin/
ENTRYPOINT ["consulctl"]

通过docker构建并保存为tar文件

[root@k8s-master consul]# docker build -t pod_registry:v1.0 .
[root@k8s-master consul]# docker save -o pod_registry_v1.tar.gz pod_registry:v1.0

通过containerd的ctr命令将其导入到harbor中

#导入
[root@k8s-master consul]# ctr image import pod_registry_v1.tar.gz#替换标签
[root@k8s-master consul]# ctr image tag docker.io/library/pod_registry:v1 harbor.ops.local/registry/pod_registry:v1.0#推送到harbor仓库
[root@k8s-master consul]# ctr image push harbor.ops.local/registry/pod_registry:v1.0 --user admin:"xxxx" -k

五、Kubernetes集成方案

以k8s部署minio为例,修改对应的yaml文件,实现minio pod启动时将其注册到consul

文件如下所示 ：只演示关键部分添加

global-config名称的configamap准备

apiVersion: v1
kind: ConfigMap
metadata:name: global-confignamespace: middleware  # 与微服务同命名空间
data:CONSUL_IP: "consul-server.middleware.svc.cluster.local:8500"  # Consul服务DNS名称

acl-token secretyaml准备

apiVersion: v1
kind: Secret
metadata:name: acl-tokennamespace: middleware
type: Opaque
data:ACL_TOKEN: OWJmYmU4MWYtMjY0OC00NjczLWFmMTQtZDEzZTBhMTcwMDUwCg==  #上述64位字符串进行base64加密

共享卷使用--使initContainer中的consulctl命令能在container中使用

关键设计点:1. 使用initContainer完成注册2. 通过共享卷将工具拷贝到业务容器3. 支持Consul节点参数动态配置和动态获取Consul服务地址和ACL令牌

      initContainers:- name: service-registrarimage: harbor.ops.local/registry/pod_registry:v1.0 #consulctl工具镜像env:- name: POD_NAME #获取pod名称，用于注册到consul中显示valueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACE  #用于将注册到consul的pod以自身的名称空间为目录valueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_IP #用于consul中meta元数据展示valueFrom:fieldRef:fieldPath: status.podIP- name: CONSUL_IP #用于注册时连接consulvalueFrom:configMapKeyRef:name: global-config #与上述configmap名称一致key: CONSUL_IP- name: ACL_TOKEN #用于注册时登录consulvalueFrom:secretKeyRef:name: acl-token #与上述secret名称一致key: ACL_TOKEN - name: NODE_NAME  #用于consul中meta元数据展示valueFrom: fieldRef:fieldPath: spec.nodeNamevolumeMounts:- mountPath: /shared-bin  # 共享卷 /shared-bin目录 挂载到 Container,使其也能使用这个工具，不用再次封装minio镜像name: shared-bincommand: ["sh", "-c"]args:- |cp /usr/local/bin/consulctl /shared-bin/ &&/usr/local/bin/consulctl register \"$CONSUL_IP" \"$ACL_TOKEN" \"9000" \    #微服务pod端口"容器监控" \ #自定义"k8s"       #自定义

preStop钩子函数

通过preStop钩子确保:1. 服务注销先于Pod终止2. 设置terminationGracePeriodSeconds为60秒3. 支持Consul节点参数动态配置和动态获取Consul服务地址和ACL令牌

.....containers:- env:- name: CONSUL_IP  # 必须显式声明valueFrom:configMapKeyRef:name: global-configkey: CONSUL_IP- name: ACL_TOKEN  # 必须显式声明valueFrom:secretKeyRef:name: acl-tokenkey: ACL_TOKEN- name: CONSUL_NODE_NAME #根据自己的consul名称替换value: "consul-0"- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.nameimage: harbor.jdicity.local/registry/minio:latestlifecycle:preStop:exec:command: ["sh", "-c", "/usr/local/bin/consulctl deregister $CONSUL_IP $ACL_TOKEN 9000(根据实际微服务端口进行替换) $CONSUL_NODE_NAME"]volumeMounts:- mountPath: /usr/local/bin/consulctl  # 挂载到 minio 容器的 PATH 目录name: shared-binsubPath: consulctlvolumes:- name: shared-bin  # 共享卷emptyDir: {}

六、 结果验证

1.注册验证

创建minio,查看conusl ui中是否已经按要求注册到minio pod

[root@k8s-master consul]# kubectl apply -f sts_minio.yaml[root@k8s-master consul]# kubectl get pod -n middleware  -owide
NAME       READY   STATUS    RESTARTS   AGE   IP             NODE                       NOMINATED NODE   READINESS GATES
minio-0    1/1     Running   0          52s   10.244.1.63    k8s-node.ops.local     <none>           <none>
minio-1    1/1     Running   0          52s   10.244.0.113   k8s-master.ops.local   <none>           <none>#查看sidecar边车容器日志 提示注册成功
[root@k8s-master consul]# kubectl logs -f -n middleware minio-0 service-registrar 
/usr/local/bin/consulctl
register
consul-server.middleware.svc.cluster.local:8500
9bfbe81f-2648-4673-af14-d13e0a1700509000
容器监控
k8s
Service registered successfully

2.销毁验证

现在模拟删除一个minio-0 pod,看新pod是否会注册

[root@k8s-master consul]# kubectl delete pod -n middleware  minio-0
pod "minio-0" deleted
[root@k8s-master consul]# kubectl get pod -n middleware  -owide
NAME       READY   STATUS    RESTARTS   AGE     IP             NODE                  NOMINATED NODE   READINESS GATES
minio-0    1/1     Running   0          18s     10.244.1.64    k8s-node.ops.local     <none>           <none>
minio-1    1/1     Running   0          8m44s   10.244.0.113   k8s-master.ops.local   <none>           <none>

现在模拟删除一个minio的statefulset yml文件,看pod是否会被剔除

[root@k8s-master consul]# kubectl delete -f sts_minio.yml 
service "minio-svc" deleted
secret "minio-secret" deleted
statefulset.apps "minio" deleted

至此pod在consul中的注册/销毁功能已全部演示完成,目前已接入维护的项目中,需要工具留言即可

---

总结

物来顺应，未来不迎，当时不杂，既过不恋

曾国藩的这十六字箴言意思是，凡事要顺其自然，坦然面对，活在当下，不过度担忧未来还未发生的事，要心无杂念地做好眼前的事，不要去留恋和纠结发生过的事。这些处世原则可以帮助我们更好地应对生活的挑战，也让我们拥有更为丰富、充实和幸福的人生。希望各位亦是如此!!!