基于 Kubernetes 1.5.2 的整容监控方案：从 Kubernets 到 Prometheus 与 Grafana

创建 monitoring 命名空间：

kubectl create namespace monitoring

创建服务账户及其相关角色：

kubectl create sa -n monitoringkubectl create rolebindings --role="read-only" --serviceaccount="something" -n monitoring

部署 Kub_state_metrics 并创建服务：

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: kub-state-metrics  namespace: monitoringspec:  replicas: 1  template:    metadata:      labels:        app: kub-state-metrics    spec:      containers:      - name: kub-state-metrics        image: quay.io/coreos/kub-state-metrics:kubestate-1.5.2        ports:        - containerPort: 8080

创建服务：

apiVersion: v1kind: Servicemetadata:  name: kub-state-metrics  namespace: monitoring  annotations:    prometheus.io/scrape: "true"spec:  ports:  - name: kub-state-metrics    port: 8080    protocol: TCP  selector:    app: kub-state-metrics

部署 Prometheus Sink：

apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: prometheus-node-exporter  namespace: monitoring  annotations:    prometheus.io/port: 9102spec:  template:    metadata:      name: prometheus-node-exporter      labels:        app: prometheus        component: node-exporter    spec:      containers:      - image: docker.io/prom/node-exporter:v0.14.0        name: prometheus-node-exporter        ports:        - name: prom-node-exp          containerPort: 9100          hostPort: 9100

创建服务：

apiVersion: v1kind: Servicemetadata:  name: prometheus-node-exporter  namespace: monitoring  annotations:    prometheus.io/scrape: "true"spec:  type: ClusterIP  ports:  - name: prometheus-node-exporter    port: 9100    protocol: TCP  selector:    app: prometheus    component: node-exporter

部署磁盘空间监控 DaemonSet：

apiVersion: extensions/v1beta1kind: DaemonSetmetadata:  name: node-directory-size-metrics  namespace: monitoringspec:  template:    metadata:      labels:        app: node-directory-size-metrics    spec:      containers:      - name: read-du        image: giantswarm/tiny-tools        volumeMounts:        - name: host-fs-var            mountPath: /mnt/var            readOnly: true        - name: metrics            mountPath: /tmp      - name: caddy        image: dockermuenster/caddy:latest        ports:        - containerPort: 9102        volumeMounts:        - name: metrics          mountPath: /var/www

创建服务：

apiVersion: v1kind: Servicemetadata:  name: node-directory-size-metrics  namespace: monitoringspec:  type: NodePort  ports:  - name: metrics    port: 9102    protocol: TCP

部署 Prometheus 服务器：

apiVersion: v1kind: ConfigMapmetadata:  name: prometheus-core  namespace: monitoringdata:  prometheus.yaml: |    global:      scrape_interval: 30s      scrape_timeout: 30s      evaluation_interval: 30s    rule_files:    - /etc/prometheus-rules/*.rules    scrape_configs:    - job_name: kubernetes-nodes      kubernetes_sd_configs:      - role: node      relabel_configs:      - source_labels: [__address__]          regex: '(.*):10250'          replacement: '${1}:10255'          target_label: __address__    - job_name: kubernetes-endpoints      kubernetes_sd_configs:      - role: endpoints      relabel_configs:      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]        action: keep        regex: true      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]        action: replace        target_label: __metrics_path__        regex: (.+)      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]        action: replace        target_label: __scheme__        regex: (https?)

部署 Prometheus 规则：

apiVersion: v1kind: ConfigMapmetadata:  name: prometheus-rules  namespace: monitoringdata:  cpu-usage.rules: |    ALERT NodeCPUUsage    IF (100 - (avg by (instance) (irate(node_cpu{name="node-exporter",mode="idle"}[5m])) * 100)) > 75    FOR 2m    LABELS { severity = "page" }    ANNOTATIONS {      SUMMARY = "{instance}: High CPU usage detected"      DESCRIPTION = "{instance}: CPU usage is above 75% (current value is: {value})"    }  ...

部署 Prometheus 服务：

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: prometheus-core  namespace: monitoringspec:  replicas: 1  template:    metadata:      name: prometheus-main      labels:        app: prometheus        component: core    spec:      serviceAccountName: prometheus-k8s      containers:      - name: prometheus        image: prom/prometheus:v1.7.1        args:        - '-storage.local.retention=12h'        - '-config.file=/etc/prometheus/prometheus.yaml'        - '-alertmanager.url=http://alertmanager:9093/'        ports:        - name: webui          containerPort: 9090

创建服务：

apiVersion: v1kind: Servicemetadata:  name: prometheus  namespace: monitoring  labels:    app: prometheus    component: coreannotations:  prometheus.io/scrape: "true"spec:  type: NodePort  ports:  - name: webui    port: 9090

部署 Grafana：

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: grafana-core  namespace: monitoringspec:  replicas: 1  template:    metadata:      labels:        app: grafana        component: core    spec:      containers:      - name: grafana-core        image: docker.io/grafana/grafana:latest        ports:        - name: grafana          containerPort: 3000        env:        - name: GF_AUTH_ANONYMOUS_ENABLED          value: "true"        - name: GF_AUTH_ANONYMOUS_ORG_ROLE          value: Admin

创建服务：

apiVersion: v1kind: Servicemetadata:  name: grafana  namespace: monitoring  labels:    app: grafana    component: corespec:  type: NodePort  ports:  - name: grafana    nodePort: 31000

使用 Grafana 模板创建监控仪表盘：

以下是 Grafana 模板的简要说明：

{  "annotations": {    "list": []  },  "editable": true,  "graphTooltip": 0,  "id": 21,  "rows": [     {      " Panels": [        {          " Targets": [            {              " expr": "sum(container_memory_usage_bytes{pod_name=\"$pod\", namespace=\"$namespace\"}) by (namespace, pod_name)"              " format": "time_series"              " interval": "30s"              " legendFormat": "total"            },            ....          ]        },        ...      ]    }  ],  "repeat": null}

可以通过 Grafana UI 直接导入此模板。