Prometheus alarm record persistence (historical alarm saving and Statistics)

I came across a need some time ago ： take Prometheus Persistent storage of historical alarms , Analyze and count again . After trying some tools , And finally alertsnitch+grafana The plan .

alertsnitch: https://gitlab.com/yakshaving.art/alertsnitch

Briefly introduce the principle of this tool ,alertsnitch Implemented a webhook, By means of alertmanager Middle configuration receiver by alertsbnitch webhook Send the alarm to alertsnitch,alertsnitch The data is processed and stored in Mysql/Postgre, For detailed introduction, please refer to its gitlab Home page .

Here is a brief introduction to how to use （k8s install , Database usage Mysql）

1、Mysql establish alertsnitch Database used and corresponding account , A little .

2、 install alertsnitch, Modify the following ALERTSNITCH_DSN Value

# alertsnitch-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertsnitch
  namespace: alertmanager
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: alertsnitch    
  template:
    metadata:
      labels:
        app.kubernetes.io/name: alertsnitch
    spec:
      containers:
      - image: registry.gitlab.com/yakshaving.art/alertsnitch
        name: alertsnitch
        ports:
        - containerPort: 9567
          name: http
        env:
        - name: ALERTSNITCH_BACKEND
          value: mysql
        - name: ALERTSNITCH_DSN
          value: ${MYSQL_USER}:${MYSQL_PASSWORD}(${MYSQL_IP}:${MYSQL_PORT})/{$MYSQL_DATABASE}
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 9567
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /-/health
            port: 9567
          initialDelaySeconds: 60
          periodSeconds: 10

apiVersion: v1
kind: Service
metadata:
  name: alertsnitch
  namespace: alertmanager
spec:
  ports:
  - name: http
    port: 9567
    targetPort: http
  selector:
    app.kubernetes.io/name: alertsnitch

3、 To configure alertmanager

    route:
      receiver: all
      routes:
      # All alarm collection 
      - receiver: all
        continue: true

--------------------

    receivers:
    #alertsnitch
    - name: 'all'
      webhook_configs:
      - url: 'http://alertsnitch.alertmanager.svc:9567/webhook'
        send_resolved: true

4、 have access to grafana To display and count historical alarms , Make a ready-made for reference

https://grafana.com/grafana/dashboards/15833