kube-eventer事件监控
发布日期:2022-02-28 11:06:00 浏览次数:6 分类:技术文章

本文共 4042 字,大约阅读时间需要 13 分钟。

下载deployment

我这里保存成kube-event.yaml

# cat kube-event.yaml---apiVersion: apps/v1kind: Deploymentmetadata:  labels:    name: kube-eventer  name: kube-eventer  namespace: kube-systemspec:  replicas: 1  selector:    matchLabels:      app: kube-eventer  template:    metadata:      labels:        app: kube-eventer      annotations:        scheduler.alpha.kubernetes.io/critical-pod: ''    spec:      dnsPolicy: ClusterFirstWithHostNet      serviceAccount: kube-eventer      containers:        - image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.2.0-484d9cd-aliyun          name: kube-eventer          command:            - "/kube-eventer"            - "--source=kubernetes:https://kubernetes.default"            ## .e.g,dingtalk sink demo            #- --sink=dingtalk:[your_webhook_url]&label=[your_cluster_id]&level=[Normal or Warning(default)]            - --sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=355cf0156xxxxxxxxxxxxxxxxxx&level=Warning          env:          # If TZ is assigned, set the TZ value as the time zone          - name: TZ            value: "Asia/Shanghai"          volumeMounts:            - name: localtime              mountPath: /etc/localtime              readOnly: true            - name: zoneinfo              mountPath: /usr/share/zoneinfo              readOnly: true          resources:            requests:              cpu: 100m              memory: 100Mi            limits:              cpu: 500m              memory: 250Mi      volumes:        - name: localtime          hostPath:            path: /etc/localtime        - name: zoneinfo          hostPath:            path: /usr/share/zoneinfo---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: kube-eventerrules:  - apiGroups:      - ""    resources:      - configmaps      - events    verbs:      - get      - list      - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: kube-eventerroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: kube-eventersubjects:  - kind: ServiceAccount    name: kube-eventer    namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata:  name: kube-eventer  namespace: kube-system

钉钉群里创建自定义webhook

设置–智能群助手–添加机器人–选择WeebHook。定义机器人名称和安全设置

安全设置这里我定义了关键字,Waring。创建后复制webhook地址。然后更改上面deployment中的sink处。

我把上面的label删掉了,只留下了level=Waring,刚好对应了我关键字的Waring。只有带有关键字的才会触发告警。

测试告警

然后创建一个测试的Tomcat的deployment,故意把image镜像的tag写错,让他无法拉取镜像

[root@master allenjol]# kubectl apply -f deploy-tomcat-test.yamldeployment.apps/tomcat-deployment-allenjol createdservice/tomcat-service-allenjol created[root@master allenjol]# kubectl get poNAME                                        READY   STATUS             RESTARTS   AGEtomcat-deployment-allenjol-b6687f99-l5vj9   0/1     ImagePullBackOff   0          45s

部署kube-event.yaml并查看日志。可以看到隔30s去检测一次

]# kubectl apply -f kube-event.yaml]# kubectl get po -n kube-system | grep kube-event[root@master allenjol]# kubectl logs -f kube-eventer-648f64c985-zfkkg -n kube-systemI0708 09:26:36.409034       1 eventer.go:67] /kube-eventer --source=kubernetes:https://kubernetes.default --sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=355cf01569aef206dc6c05681aaf3ed0ea19ed3597db4c26c565dbeb69ce1303&level=WarningI0708 09:26:36.409191       1 eventer.go:68] kube-eventer version: v1.2.0 commit: 484d9cdI0708 09:26:36.411557       1 eventer.go:94] Starting with DingTalkSink sinkI0708 09:26:36.411596       1 eventer.go:108] Starting eventerI0708 09:26:36.411678       1 eventer.go:116] Starting eventer http serviceI0708 09:27:00.000163       1 manager.go:102] Exporting 5 eventsI0708 09:27:30.000130       1 manager.go:102] Exporting 9 eventsI0708 09:28:00.000147       1 manager.go:102] Exporting 1 eventsI0708 09:28:30.000150       1 manager.go:102] Exporting 4 eventsI0708 09:29:00.000138       1 manager.go:102] Exporting 1 events...

可以看到这里已经看到了钉钉的webhook地址了,并且还收集到了events。

查看钉钉群,就会看到已经出现了告警了。
cnsre运维博客|Linux系统运维|自动化运维|云计算|运维监控

其实这个告警当前还存在点问题。个人认为不应该这么频繁发送,应该像prometheus一样可以配置抑制和静默。然后监控时间可以更改。当然熟悉go语言可以自己改源码然后构建成镜像。

转载地址:https://blog.csdn.net/weixin_47430049/article/details/120667804 如侵犯您的版权,请留言回复原文章的地址,我们会给您删除此文章,给您带来不便请您谅解!

上一篇:Kubernetes 持久化数据存储 StorageClass
下一篇:记一次 Ubuntu 内核升级故障处理

发表评论

最新留言

留言是一种美德,欢迎回访!
[***.207.175.100]2024年04月09日 11时30分50秒

关于作者

    喝酒易醉,品茶养心,人生如梦,品茶悟道,何以解忧?唯有杜康!
-- 愿君每日到此一游!

推荐文章