K8s抓包神器:一条命令搞定 Pod 网络分析,效率翻倍!
在 Kubernetes 的世界里,Pod 是最小的部署单元,但它们之间的网络通信却常常让人头疼。当服务出现调用失败、响应延迟、数据异常等问题时,我们往往需要深入分析网络流量,看看“中间到底发生了什么”。
传统做法是登录 Pod 所在节点,然后 进入容器 netns,最后使用节点上 tcpdump 工具进行抓包。整个过程比较繁琐。有没有一种更优雅、高效、安全的方式来实现对 Pod 流量的实时抓包呢?答案就是:kubectl sniff 插件!
0
1
什么是 kubectl sniff?
ksniff 是一个专为 Kubernetes 设计的 kubectl 插件,用于在集群中的 Pod 上实现远程网络抓包,结合了 tcpdump 的捕获能力和 Wireshark 的图形化分析功能,极大简化了容器网络故障排查流程。
工作原理
1、非特权模式(默认)
上传工具:将本地预存的静态 tcpdump 通过 tar 打包上传至目标容器临时目录(如 /tmp/static-tcpdump)
命令执行:在容器内执行命令捕获流量(如 /tmp/static-tcpdump -i any -U -w -),输出通过 SPDY 协议流式传输到本地
启动 Wireshark:本地自动唤起 Wireshark 解析数据流
2、特权模式(-p 参数)
创建特权 Pod:在目标 Pod 所在节点启动一个拥有 NET_ADMIN 权限的 Pod,共享目标网络命名空间
隔离抓包:在特权 Pod 中运行 tcpdump,流量不经过原容器,避免安全风险
0
2
安装sniff插件
使用krew安装sniff插件。如果没有安装过krew的话,请参考《掌握Kubernetes:kubectl ingress-nginx插件轻松管理Ingress资源》文章的安装krew小结。
在线安装
$ kubectl krew install sniff
Tip:国内环境不一定可以直接安装成功,这里也提供一个离线安装方法
离线安装
1、插件信息文件
cat <<'EOF' | tee sniff.yaml
apiVersion: krew.googlecontainertools.github.com/v1alpha2
kind: Plugin
metadata:
name: sniff
spec:
version: v1.6.2
homepage: https://github.com/eldadru/ksniff
platforms:
- uri: https://github.com/eldadru/ksniff/releases/download/v1.6.2/ksniff.zip
sha256: c59b5c9ea84d6cb771096f1246c919b71389f9d4234e858f4929208957e561fd
bin: kubectl-sniff
files:
- from: kubectl-sniff-darwin
to: kubectl-sniff
- from: static-tcpdump
to: .
- from: LICENSE
to: .
selector:
matchLabels:
os: darwin
arch: amd64
- uri: https://github.com/eldadru/ksniff/releases/download/v1.6.2/ksniff.zip
sha256: c59b5c9ea84d6cb771096f1246c919b71389f9d4234e858f4929208957e561fd
bin: kubectl-sniff
files:
- from: kubectl-sniff-darwin-arm64
to: kubectl-sniff
- from: static-tcpdump
to: .
- from: LICENSE
to: .
selector:
matchLabels:
os: darwin
arch: arm64
- uri: https://github.com/eldadru/ksniff/releases/download/v1.6.2/ksniff.zip
sha256: c59b5c9ea84d6cb771096f1246c919b71389f9d4234e858f4929208957e561fd
bin: kubectl-sniff
files:
- from: kubectl-sniff
to: .
- from: static-tcpdump
to: .
- from: LICENSE
to: .
selector:
matchLabels:
os: linux
arch: amd64
- uri: https://github.com/eldadru/ksniff/releases/download/v1.6.2/ksniff.zip
sha256: c59b5c9ea84d6cb771096f1246c919b71389f9d4234e858f4929208957e561fd
bin: kubectl-sniff.exe
files:
- from: kubectl-sniff-windows
to: kubectl-sniff.exe
- from: static-tcpdump
to: .
- from: LICENSE
to: .
selector:
matchLabels:
os: windows
arch: amd64
shortDescription: Start a remote packet capture on pods using tcpdump and wireshark
caveats: |
This plugin needs the following programs:
* wireshark (optional, used for live capture)
description: |
When working with micro-services, many times it's very helpful to get a capture of the network
activity between your micro-service and it's dependencies.
ksniff use kubectl to upload a statically compiled tcpdump binary to your pod and redirecting it's
output to your local Wireshark for smooth network debugging experience.
EOF
Tip:上述内容是从github粘贴到文件的,GitHub上的 sniff插件信息 地址。
2、下载sniff压缩包
$ wget https://github.com/eldadru/ksniff/releases/download/v1.6.2/ksniff.zip
3、安装sniff
$ kubectl krew install --manifest sniff.yaml --archive ksniff.zip
Installing plugin: sniff
Installed plugin: sniff
\
| Use this plugin:
| kubectl sniff
| Documentation:
| https://github.com/eldadru/ksniff
| Caveats:
| \
| | This plugin needs the following programs:
| | * wireshark (optional, used for live capture)
| /
/
0
3
实战sniff抓包
分别验证两种模式
非特权模式
$ k sniff -n default client-84dc6fbbb-lsj6m -f "port 1234" -o test.pcap
INFO[0000] using tcpdump path at: '/home/jiaxzeng/.krew/store/sniff/v1.6.2/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'client'
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] sniffing on pod: 'client-84dc6fbbb-lsj6m' [namespace: 'default', container: 'client', filter: 'port 1234', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/home/jiaxzeng/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/home/jiaxzeng/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'client'
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'client', pod: 'client-84dc6fbbb-lsj6m', namespace: 'default'
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '1', stdErr :''
INFO[0000] file not found on: '/tmp/static-tcpdump', starting to upload
INFO[0000] verifying file uploaded successfully
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'client', pod: 'client-84dc6fbbb-lsj6m', namespace: 'default'
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: ''
INFO[0000] file uploaded successfully
INFO[0000] tcpdump uploaded successfully
INFO[0000] output file option specified, storing output in: 'test.pcap'
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - port 1234]' on container: 'client', pod: 'client-84dc6fbbb-lsj6m', namespace: 'default'
^C
Tip:不出意外的话,你们执行这个会报错(退出码139)。因为下载到static-tcpdump版本太低了。我已经编译好可用的版本了,如果有需要的话,请在后台回复“666”。
特权模式
1、下载镜像并推送到Harbor
$ sudo docker pull docker.io/hamravesh/ksniff-helper:v3
$ sudo docker tag docker.io/hamravesh/ksniff-helper:v3 core.jiaxzeng.com/library/ksniff-helper:v3
$ sudo docker push core.jiaxzeng.com/library/ksniff-helper:v3
$ sudo docker pull docker.io/maintained/tcpdump:latest
$ sudo docker tag docker.io/maintained/tcpdump:latest core.jiaxzeng.com/library/tcpdump:4.99.1
$ sudo docker push core.jiaxzeng.com/library/tcpdump:4.99.1
2、sniff抓包
$ k sniff -n default simple-7788cd4d7d-98rsx -p --image core.jiaxzeng.com/library/ksniff-helper:v3 --tcpdump-image core.jiaxzeng.com/library/tcpdump:4.99.1 -o test.pcap
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'simple'
INFO[0000] sniffing method: privileged pod
INFO[0000] sniffing on pod: 'simple-7788cd4d7d-98rsx' [namespace: 'default', container: 'simple', filter: '', interface: 'any']
INFO[0000] creating privileged pod on node: 'k8s-node02'
INFO[0000] pod: 'ksniff-vj8zq' created successfully in namespace: 'default'
INFO[0000] waiting for pod successful startup
INFO[0003] pod: 'ksniff-vj8zq' created successfully on node: 'k8s-node02'
INFO[0003] output file option specified, storing output in: 'test.pcap'
INFO[0003] starting remote sniffing using privileged pod
INFO[0003] executing command: '[/bin/sh -c
set -ex
export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
export CONTAINERD_NAMESPACE="k8s.io"
export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
crictl pull core.jiaxzeng.com/library/tcpdump:4.99.1 >/dev/null
netns=$(crictl inspect b5873b6f4c83fb1b1d2ebd4e103652bd2a7626b315ebe5dabc26a9d46011b4ae | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" core.jiaxzeng.com/library/tcpdump:4.99.1 ksniff-container-PQeingmL tcpdump -i any -U -w -
]' on container: 'ksniff-privileged', pod: 'ksniff-vj8zq', namespace: 'default'
^C
Tip:CTRL+C后创建的pod是没有删除掉,需要手工执行删除。
两种模式选择
非特权模式要求:
业务pod启动用户是root
业务pod需要有sh环境(/bin/sh)
特权模式要求:
主机能拉取到镜像
需要手删除抓包的pod
0
4
结语
在 Kubernetes 复杂的微服务架构中,网络问题往往是最难定位的一类故障。而 kubectl sniff 提供了一种轻量级、非侵入式的抓包方式,极大提升了我们的诊断效率。