eks节点的网络策略配置机制解析

网络策略涉及到的组件包括如下

• Network Policy Controller控制器会自动安装在 EKS Control Plane 上，监视 NetworkPolicy 对象并发送指令到 Node Agent

  When you create a new Amazon EKS cluster, the network policy controller is automatically installed on the EKS control plane. It actively monitors the creation of network policies within your cluster and reconciles policy endpoints. Subsequently, the controller instructs the node agent to create or update eBPF programs on the node by publishing pod information through the policy endpoints

• Network Policy Node Agent，通过创建 eBPF 程序在节点上实现网络策略

• AWS eBPF SDK for Go，提供了一个接口，用于与节点上的 eBPF 程序进行交互

• VPC Resource Controller，管理Kubernetes Pods的分支和中继网络接口

当需要涉及到网络策略相关操作时（例如创建netpl资源），np-controller会解析配置的网络策略，并通过自定义 CRD （PolicyEndpoints在np-controller安装的时候自动安装） 资源发布解析的endpoints。np-node-agent获取PolicyEndpoint资源，并通过附加到pod主Veth 接口的 eBPF 探针来执行策略。对应的日志为/var/log/aws-routed-eninetwork-policy-agent.log

np-node-agent会将主机的/sys/fs/bpf文件系统挂载在卷bpf-pin-path下

/sys/fs/bpf from bpf-pin-path (rw)

np-node-agent默认的启动参数为

Args:                                                        --enable-ipv6=false                                        --enable-network-policy=false                               --enable-cloudwatch-logs=false                             --enable-policy-event-logs=false                           --log-file=/var/log/aws-routed-eni/network-policy-agent.log--metrics-bind-addr=:8162                                  --health-probe-bind-addr=:8163                             --conntrack-cache-cleanup-period=300                       

np-node-agent启动日志中的主要逻辑如下

• 等待controller就绪

• 设置ConntrackTTL清理时间cleanupPeriod为300

• 校验并安装bpf程序到/opt/cni/bin/

  $ ls -al /opt/cni/bin| grep bpf
  -rw-r--r-- 1 root root    27136 Oct 23 10:51 tc.v4egress.bpf.o
  -rw-r--r-- 1 root root    27360 Oct 23 10:51 tc.v4ingress.bpf.o
  -rw-r--r-- 1 root root     3384 Oct 23 10:51 v4events.bpf.o
  

• 安装aws-eks-na-cli命令到/opt/cni/bin/

• 设置全局默认maps和加载probe

• 初始化Conntrack client

• 开启rpc server监听127.0.0.1:50052

• 启动metrics server

如果收到了来自np-controller的指令，则np-node-agent会开始为pod创建网络策略，

• 如果没开np直接返回success

  // EnforceNpToPod processes CNI Enforce NP network request
  func (s *server) EnforceNpToPod(ctx context.Context, in *rpc.EnforceNpRequest) (*rpc.EnforceNpReply, error) {if s.policyReconciler.GeteBPFClient() == nil {s.log.Info("Network policy is disabled, returning success")
  

• 同一个节点上的pod副本会使用同样的eBPF firewall maps

  if s.policyReconciler.ArePoliciesAvailableInLocalCache(podIdentifier) && isMapUpdateRequired {//Derive Ingress and Egress Firewall Rules and Update the relevant eBPF mapsingressRules, egressRules, _ :=s.policyReconciler.DeriveFireWallRulesPerPodIdentifier(podIdentifier, in.K8S_POD_NAMESPACE)err = s.policyReconciler.GeteBPFClient().UpdateEbpfMaps(podIdentifier, ingressRules, egressRules)
  

由于vpc-cni插件会在节点上安装eBPF SDK集合，因此可以使用 eBPF SDK 工具（aws-eks-na-cli）来识别网络策略的问题

• aws-eks-na-cli命令只有在开启--enable-network-policy=true后才会安装到节点

$ sudo  /opt/cni/bin/aws-eks-na-cli ebpf -h
Dump all ebpf related dataUsage:aws-eks-na-cli ebpf [flags]aws-eks-na-cli ebpf [command]Aliases:ebpf, ebpfAvailable Commands:dump-maps       Dump all ebpf maps related dataloaded-ebpfdata Dump all ebpf related datamaps            Dump all ebpf maps related dataprogs           Dump all ebpf program related data

具体的用法涉及到ebpf sdk的组件和api调用，需要进一步学习