Kubernetes service discovery monitoring endpoints

monitor Pod

Previous apiserver It's actually a special kind of Endpoints, Now let's also configure a task to specifically discover common types of Endpoint, In fact, that is Service The associated Pod list , Because not all Endpoints Will provide metrics Interface , So we need to take the initiative to tell Prometheus To discover what Endpoints, Of course, there are many ways to tell , But one of the conventional ways is through annotations Note to inform , As shown below ：

- job_name: "endpoints"
  kubernetes_sd_configs:
    - role: endpoints
  relabel_configs:
    #  Retain  Service  The annotation of is  prometheus.io/scrape: true  Of  Endpoints
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    #  The index interface protocol passed  prometheus.io/scheme  This annotation gets  http  or  https
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    #  The endpoint path of the index interface passes through  prometheus.io/path  This annotation gets 
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    #  The direct interface address port passes through  prometheus.io/port  Annotation access 
    - source_labels:
        [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: ([^:]+)(?::\d+)?;(\d+) # RE2  Regular rule ,+ One or more times ,? yes 0 Time or 1 Time , among ?: Represents a non matching group ( It means not getting matching results )
      replacement: $1:$2
    #  mapping  Service  Of  Label  label 
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    #  take  namespace  Map to tags 
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    #  take  Service  The name is mapped to a label 
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: kubernetes_name
    #  take  Pod  The name is mapped to a label 
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

With this annotation , In fact, it is to tell Prometheus, Now the service offers metrics Interface . The port and address provided by the interface must be told , And what the agreement is , All this goes through annotation To provide .

    #  Retain  Service  The annotation of is  prometheus.io/scrape: true  Of  Endpoints
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true

Notice that we are here  relabel_configs  A lot of configuration has been done in the area , Especially the first reservation __meta_kubernetes_service_annotation_prometheus_io_scrape  by true To keep it , This means that you want to automatically discover Endpoint, We need to be in Service Of  annotations  Area add  prometheus.io/scrape=true  Annotations , We can also use  Relabeler - The playground for Prometheus relabeling rules  This tool helps us configure Relabel. Now let's update the above configuration , Check the effect ：

  We can see  endpoints  Under this task, only 5 A mission , This is because we are  relabel_configs  Filtered  annotations  Yes  prometheus.io/scrape=true  Of Service, Now only two such services in our system meet the requirements , such as  kube-dns  This Service Here are two examples , So there are two examples ：

 * kubectl get svc kube-dns -n kube-system -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9153"  # metrics  Port of the interface 
    prometheus.io/scrape: "true"  #  This annotation can make prometheus Auto discovery 
  creationTimestamp: "2021-10-25T12:33:14Z"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: kube-dns
  namespace: kube-system
  ......

Now we created before redis This Service To add on  prometheus.io/scrape=true  This annotation ：

# redis-svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: kube-mon
  annotations:
    prometheus.io/scrape: "true" #  Let the automatic discovery above get the service 
    prometheus.io/port: "9121" #  Appoint metrics Interface access port 
spec:
  selector:
    app: redis
  ports:
    - name: redis
      port: 6379
      targetPort: 6379
    - name: prom
      port: 9121
      targetPort: 9121

because redis Service metrics Interface in 9121 This  redis-exporter  Service , So we need to add another  prometheus.io/port=9121  In this way annotations, Update this Service：

 * kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/redis-svc.yaml

After the update is complete , Go to Prometheus see Targets route , You can see redis The service automatically appears in  endpoints  Under this task ：

In this way, we have a new service , If the service itself provides  /metrics  Interface , We don't need to configure it statically at all , Now we can put the previously configured redis Static configuration removes .

Empathy prometheus

[[email protected] prometheus]# cat prometheus-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitor
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
  labels:
    app: prometheus
spec:
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      protocol: TCP
  selector:
    app: prometheus
    component: server