電子發(fā)燒友網(wǎng)>電子資料下載>電子資料>KubeEye Kubernetes集群自動巡檢工具

KubeEye Kubernetes集群自動巡檢工具

1854540 2022-05-10 | zip | 0.42 MB | 次下載 | 2積分

普通下載普通下載

資料介紹

授權(quán)協(xié)議 Apache

開發(fā)語言 Google Go

操作系統(tǒng) 跨平臺

軟件類型開源軟件

所屬分類云計算、 PaaS系統(tǒng)/容器

軟件簡介

KubeEye 旨在發(fā)現(xiàn) Kubernetes 上的各種問題，比如應(yīng)用配置錯誤（使用Polaris）、集群組件不健康和節(jié)點(diǎn)問題（使用Node-Problem-Detector）。除了預(yù)定義的規(guī)則，它還支持自定義規(guī)則。

架構(gòu)圖

KubeEye 通過調(diào)用Kubernetes API，通過常規(guī)匹配日志中的關(guān)鍵錯誤信息和容器語法的規(guī)則匹配來獲取集群診斷數(shù)據(jù)，詳見架構(gòu)。

怎么使用

機(jī)器上安裝 KubeEye
- 從?Releases?中下載預(yù)構(gòu)建的可執(zhí)行文件。
- 或者你也可以從源代碼構(gòu)建
```
git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye 
make install
```
[可選] 安裝 Node-problem-Detector
注意：這一行將在你的集群上安裝 npd，只有當(dāng)你想要詳細(xì)的報告時才需要。
ke install npd
KubeEye 執(zhí)行

root@node1:# ke diag
NODENAME        SEVERITY     HEARTBEATTIME               REASON              MESSAGE
node18          Fatal        2020-11-19T10:32:03+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node19          Fatal        2020-11-19T10:31:37+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node2           Fatal        2020-11-19T10:31:14+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node3           Fatal        2020-11-27T17:36:53+08:00   KubeletNotReady     Container runtime not ready: RuntimeReady=false reason:DockerDaemonNotReady message:docker: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

NAME            SEVERITY     TIME                        MESSAGE
scheduler       Fatal        2020-11-27T17:09:59+08:00   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0          Fatal        2020-11-27T17:56:37+08:00   Get https://192.168.13.8:2379/health: dial tcp 192.168.13.8:2379: connect: connection refused

NAMESPACE       SEVERITY     PODNAME                                          EVENTTIME                   REASON                MESSAGE
default         Warning      node3.164b53d23ea79fc7                           2020-11-27T17:37:34+08:00   ContainerGCFailed     rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
default         Warning      node3.164b553ca5740aae                           2020-11-27T18:03:31+08:00   FreeDiskSpaceFailed   failed to garbage collect required amount of images. Wanted to free 5399374233 bytes, but freed 416077545 bytes
default         Warning      nginx-b8ffcf679-q4n9v.16491643e6b68cd7           2020-11-27T17:09:24+08:00   Failed                Error: ImagePullBackOff
default         Warning      node3.164b5861e041a60e                           2020-11-27T19:01:09+08:00   SystemOOM             System OOM encountered, victim process: stress, pid: 16713
default         Warning      node3.164b58660f8d4590                           2020-11-27T19:01:27+08:00   OOMKilling            Out of memory: Kill process 16711 (stress) score 205 or sacrifice child Killed process 16711 (stress), UID 0, total-vm:826516kB, anon-rss:819296kB, file-rss:0kB, shmem-rss:0kB
insights-agent  Warning      workloads-1606467120.164b519ca8c67416            2020-11-27T16:57:05+08:00   DeadlineExceeded      Job was active longer than specified deadline
kube-system     Warning      calico-node-zvl9t.164b3dc50580845d               2020-11-27T17:09:35+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system     Warning      kube-proxy-4bnn7.164b3dc4f4c4125d                2020-11-27T17:09:09+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system     Warning      nodelocaldns-2zbhh.164b3dc4f42d358b              2020-11-27T17:09:14+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29


NAMESPACE       SEVERITY     NAME                      KIND         TIME                        MESSAGE
kube-system     Warning      node-problem-detector     DaemonSet    2020-11-27T17:09:59+08:00   [livenessProbeMissing runAsPrivileged]
kube-system     Warning      calico-node               DaemonSet    2020-11-27T17:09:59+08:00   [runAsPrivileged cpuLimitsMissing]
kube-system     Warning      nodelocaldns              DaemonSet    2020-11-27T17:09:59+08:00   [cpuLimitsMissing runAsPrivileged]
default         Warning      nginx                     Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing livenessProbeMissing tagNotSpecified]
insights-agent  Warning      workloads                 CronJob      2020-11-27T17:09:59+08:00   [livenessProbeMissing]
insights-agent  Warning      cronjob-executor          Job          2020-11-27T17:09:59+08:00   [livenessProbeMissing]
kube-system     Warning      calico-kube-controllers   Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing livenessProbeMissing]
kube-system     Warning      coredns                   Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing]

您可以參考常見FAQ內(nèi)容來優(yōu)化您的集群。

KubeEye 能做什么

KubeEye 可以發(fā)現(xiàn)你的集群控制平面的問題，包括 kube-apiserver/kube-controller-manager/etcd 等。
KubeEye 可以幫助你檢測各種節(jié)點(diǎn)問題，包括內(nèi)存/CPU/磁盤壓力，意外的內(nèi)核錯誤日志等。
KubeEye 根據(jù)行業(yè)最佳實(shí)踐驗(yàn)證你的工作負(fù)載 yaml 規(guī)范，幫助你使你的集群穩(wěn)定。

檢查項

是/否	檢查項	描述
	ETCDHealthStatus	如果 etcd 啟動并正常運(yùn)行
	ControllerManagerHealthStatus	如果 kubernetes kube-controller-manager 正常啟動并運(yùn)行
	SchedulerHealthStatus	如果 kubernetes kube-schedule 正常啟動并運(yùn)行
	NodeMemory	如果節(jié)點(diǎn)內(nèi)存使用量超過閾值
	DockerHealthStatus	如果 docker 正常運(yùn)行
	NodeDisk	如果節(jié)點(diǎn)磁盤使用量超過閾值
	KubeletHealthStatus	如果 kubelet 激活狀態(tài)且正常運(yùn)行
	NodeCPU	如果節(jié)點(diǎn) CPU 使用量超過閾值
	NodeCorruptOverlay2	Overlay2 不可用
	NodeKernelNULLPointer	node 顯示 NotReady
	NodeDeadlock	死鎖是指兩個或兩個以上的進(jìn)程在爭奪資源時互相等待的現(xiàn)象。
	NodeOOM	監(jiān)控那些消耗過多內(nèi)存的進(jìn)程，尤其是那些消耗大量內(nèi)存非?？斓倪M(jìn)程，內(nèi)核會殺掉它們，防止它們耗盡內(nèi)存
	NodeExt4Error	Ext4 掛載失敗
	NodeTaskHung	檢查D狀態(tài)下是否有超過 120s 的進(jìn)程
	NodeUnregisterNetDevice	檢查對應(yīng)網(wǎng)絡(luò)
	NodeCorruptDockerImage	檢查 docker 鏡像
	NodeAUFSUmountHung	檢查存儲
	NodeDockerHung	Docker hang住, 檢查 docker 的日志
	PodSetLivenessProbe	如果為pod中的每一個容器設(shè)置了 livenessProbe
	PodSetTagNotSpecified	鏡像地址沒有聲明標(biāo)簽或標(biāo)簽是最新
	PodSetRunAsPrivileged	以特權(quán)模式運(yùn)行 Pod 意味著 Pod 可以訪問主機(jī)的資源和內(nèi)核功能
	PodSetImagePullBackOff	Pod 無法正確拉出鏡像，因此可以在相應(yīng)節(jié)點(diǎn)上手動拉出鏡像
	PodSetImageRegistry	檢查鏡像形式是否在相應(yīng)倉庫
	PodSetCpuLimitsMissing	未聲明 CPU 資源限制
	PodNoSuchFileOrDirectory	進(jìn)入容器查看相應(yīng)文件是否存在
	PodIOError	這通常是由于文件 IO 性能瓶頸
	PodNoSuchDeviceOrAddress	檢查對應(yīng)網(wǎng)絡(luò)
	PodInvalidArgument	檢查對應(yīng)存儲
	PodDeviceOrResourceBusy	檢查對應(yīng)的目錄和 PID
	PodFileExists	檢查現(xiàn)有文件
	PodTooManyOpenFiles	程序打開的文件/套接字連接數(shù)超過系統(tǒng)設(shè)置值
	PodNoSpaceLeftOnDevice	檢查磁盤和索引節(jié)點(diǎn)的使用情況
	NodeApiServerExpiredPeriod	將檢查 ApiServer 證書的到期日期少于30天
	PodSetCpuRequestsMissing	未聲明 CPU 資源請求值
	PodSetHostIPCSet	設(shè)置主機(jī) IP
	PodSetHostNetworkSet	設(shè)置主機(jī)網(wǎng)絡(luò)
	PodHostPIDSet	設(shè)置主機(jī) PID
	PodMemoryRequestsMiss	沒有聲明內(nèi)存資源請求值
	PodSetHostPort	設(shè)置主機(jī)端口
	PodSetMemoryLimitsMissing	沒有聲明內(nèi)存資源限制值
	PodNotReadOnlyRootFiles	文件系統(tǒng)未設(shè)置為只讀
	PodSetPullPolicyNotAlways	鏡像拉策略并非總是如此
	PodSetRunAsRootAllowed	以 root 用戶執(zhí)行
	PodDangerousCapabilities	您在 ALL / SYS_ADMIN / NET_ADMIN 等功能中有危險的選擇
	PodlivenessProbeMissing	未聲明 ReadinessProbe
	privilegeEscalationAllowed	允許特權(quán)升級
?	NodeNotReadyAndUseOfClosedNetworkConnection	http 2-max-streams-per-connection
?	NodeNotReady	無法啟動 ContainerManager 無法設(shè)置屬性 TasksAccounting 或未知屬性

未標(biāo)注的項目正在開發(fā)中

添加自定義檢查規(guī)則

添加 npd 自定義檢查規(guī)則

安裝 NPD 指令?ke install npd
由 kubectl 編輯 configmap kube-system/node-problem-detector-config,

kubectl edit cm -n kube-system node-problem-detector-config

在 configMap 的規(guī)則下添加異常日志信息，規(guī)則遵循正則表達(dá)式。

自定義最佳實(shí)踐規(guī)則

準(zhǔn)備一個規(guī)則 yaml，例如，下面的規(guī)則將驗(yàn)證你的 pod 規(guī)范，以確保鏡像只來自授權(quán)的注冊處。

checks:
  imageFromUnauthorizedRegistry: warning

customChecks:
  imageFromUnauthorizedRegistry:
    promptMessage: When the corresponding rule does not match. Show that image from an unauthorized registry.
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          not:
            pattern: ^quay.io

將上述規(guī)則保存為yaml，例如，rule.yaml。
用 rule.yaml 運(yùn)行 KubeEye。

root:# ke diag -f rule.yaml --kubeconfig ~/.kube/config
NAMESPACE     SEVERITY    NAME                      KIND         TIME                        MESSAGE
default       Warning     nginx                     Deployment   2020-11-27T17:18:31+08:00   [imageFromUnauthorizedRegistry]
kube-system   Warning     node-problem-detector     DaemonSet    2020-11-27T17:18:31+08:00   [livenessProbeMissing runAsPrivileged]
kube-system   Warning     calico-node               DaemonSet    2020-11-27T17:18:31+08:00   [cpuLimitsMissing runAsPrivileged]
kube-system   Warning     calico-kube-controllers   Deployment   2020-11-27T17:18:31+08:00   [cpuLimitsMissing livenessProbeMissing]
kube-system   Warning     nodelocaldns              DaemonSet    2020-11-27T17:18:31+08:00   [runAsPrivileged cpuLimitsMissing]
default       Warning     nginx                     Deployment   2020-11-27T17:18:31+08:00   [livenessProbeMissing cpuLimitsMissing]
kube-system   Warning     coredns                   Deployment   2020-11-27T17:18:31+08:00   [cpuLimitsMissing]