问题描述
ingress-nginx-controller pod健康检查失败,导致pod一直重启,查看kubelet日志报错:
I1024 11:10:37.192392 7 reflector.go:219] Starting reflector *v1.Secret (0s) from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192406 7 reflector.go:255] Listing and watching *v1.Secret from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192403 7 reflector.go:219] Starting reflector *v1.IngressClass (0s) from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192414 7 reflector.go:255] Listing and watching *v1.IngressClass from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192405 7 reflector.go:219] Starting reflector *v1.Service (0s) from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192423 7 reflector.go:255] Listing and watching *v1.Service from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192434 7 reflector.go:219] Starting reflector *v1.Endpoints (0s) from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192454 7 reflector.go:255] Listing and watching *v1.Endpoints from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192465 7 reflector.go:219] Starting reflector *v1.ConfigMap (0s) from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:37.192478 7 reflector.go:255] Listing and watching *v1.ConfigMap from k8s.io/client-go@v0.23.6/tools/cache/reflector.go:167
I1024 11:10:51.104048 7 healthz.go:257] nginx-ingress-controller check failed: healthz
E1024 11:11:36.820725 7 store.go:202] timed out waiting for caches to sync
问题分析
Listing/watching Secret/IngressClass/Endpoints/ConfigMap timed out waiting for caches to sync 发现在查找这些资源对象时间较长,发生超时,排查发现集群中的configmap较多(10000+)。修改ingress-nginx deployment或daemonsets中ingress-nginx的启动参数。
问题解决
1.修改deployment
...
containers:
- args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/ingress-nginx-defaultbackend
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --watch-namespace=ingress-nginx
...
参考链接
[1] https://github.com/kubernetes/ingress-nginx/issues/8647 如果您有其他问题,欢迎您联系火山引擎技术支持服务