Prometheus thanos Quick Guide

Thanos The system is built on Prometheus above , Used to extend Prometheus function , Provides global queries 、 High Availability Support 、 Core capabilities such as historical data backup to object storage .Prometheus Play the role of data collection in the architecture , The alarm function is still recommended Prometheus Local data and alarm capability of nodes .

Thanos Components

• Sidecar： Connect to Prometheus, Read its data and upload it to the object storage , And provide query API.
• Store-gateway： Provide the query ability of indicators in object storage .
• Compactor： Compress the data stored in the object 、 Downsampling and configuration retention time .
• Receiver： from Prometheus remote-write WAL receive data , Upload to object store , And provide query API.
• Ruler/Rule： Calculation Thanos Data recording rules and alerting rules, Upload to object store .
• Querier/Query： Realization Prometheus v1 API To aggregate the data of the underlying components .
• Query Frontend： As a query agent , Split query by day , And distribute the request to Querier, At the same time, cache the response results .

Use Sidecar The architecture is as follows ：

Use Receiver The architecture is as follows ：

Sidecar

Thanos adopt Sidecar Process and existing Promerheus Integrate ,Sidecar And Prometheus Running on the same host or Pod in .Sidecar The role of the Prometheus Back up your local data to the object store , And pass gRPC API For others Thanos Component provides index query capability .Sidecar Used Prometheus Of reload function , therefore Prometheus Parameters need to be added during startup -web.enable-lifecycle.

External storage

The following configuration , To configure Sidecar take Prometheus Backup data to object storage ：

thanos sidecar \
    --tsdb.path            /var/prometheus \          # TSDB data directory of Prometheus
    --prometheus.url       "http://localhost:9090" \  # Be sure that the sidecar can use this url!
    --objstore.config-file bucket_config.yaml \       # Storage configuration for uploading data

Storage profile bucket_config.yaml For an example of writing, see Thanos - Highly available Prometheus setup with long term storage capabilities

This component pair Prometheus The impact is negligible , If you just want to Sidecar As a query aggregation component , You can get rid of --objstore.config-file.

Store API

Sidecar Provides a gRPC Store API, Can be used to query Prometheus Index data in . Following configuration , take Sidecar Connect to Prometheus, And exposed it Store API.

thanos sidecar \
    --tsdb.path                 /var/prometheus \
    --objstore.config-file      bucket_config.yaml \       # Bucket config file to send data to
    --prometheus.url            http://localhost:9090 \    # Location of the Prometheus HTTP server
    --http-address              0.0.0.0:19191 \            # HTTP endpoint for collecting metrics on the Sidecar
    --grpc-address              0.0.0.0:19090              # GRPC endpoint for StoreAPI

Upload historical data

When Sidecar Using parameter --shipper.upload-compacted Startup time , Will scan Prometheus Locally stored blcok And upload to the object store .

Be careful ： Can no longer run Sidecar And has been block Upload to the storage node and run this parameter , This will cause duplication in the storage block, You must delete... Manually , Otherwise, an error will be reported .

External Labels

Prometheus You can configure the external_labels, To identify a Prometheus example . because Thanos Characteristics of aggregating data across multiple instances , Therefore, it is very important to provide a consistent set of labels .

stay Thanos in , Every Prometheus Must have a set of globally unique identification labels , for example ：

# Prometheus A
global:
  external_labels:
    region: eu-west
    monitor: infrastructure
    replica: A


# Prometheus B
global:
  external_labels:
    region: eu-west
    monitor: infrastructure
    replica: B

Querier/Query

Configure as described above for all Prometheus Good configuration Sidecar after , We want to use Thanos Global query layer to query multiple at one time Prometheus The data of .

Query Components are stateless , You can expand the number of copies at will .Query Connecting to Sidecar after , Will automatically detect a PromQL Query which... To use Prometheus.

Thanos Querier It's also achieved Prometheus The official HTTP API, Therefore, it can be directly used as Gafana Data source usage of .

thanos query \
    --http-address 0.0.0.0:19192 \                                # HTTP Endpoint for Thanos Querier UI
    --store        1.2.3.4:19090 \                                # Static gRPC Store API Address for the query node to query
    --store        1.2.3.5:19090 \                                # Also repeatable
    --store        dnssrv+_grpc._tcp.thanos-store.monitoring.svc  # Supports DNS A & SRV records
    --store        Thanos_Store_IP:port                           #  Connect Thnaos Store Components , Later, we will further introduce 

among 1.2.3.4:19090、1.2.3.5:19090 For two Sidecar Of gRPC Address .Query It will be called periodically API To get the latest metadata information and check API Health status . The metadata includes the time window and time of each node external labels.

Now access HTTP 19192, You will see a similar Prometheus Of UI Interface , If the components are configured correctly , You can query all Prometheus The data of the .

Query to duplicate

Query The component can query the data from Prometheus HA De duplication of data , This requires us to configure prometheus add global.external_labels To identify each Prometheus The identity of the , For example, use the configuration above ,replica The tag is used to distinguish a Prometheus HA Different instances in the pair .Query It can be configured as follows ：

thanos query \
    --http-address        0.0.0.0:19192 \
    --store               1.2.3.4:19090 \
    --store               1.2.3.5:19090 \
    --query.replica-label replica  # Replica label for de-duplication
    --query.replica-label replicaX #  Support the configuration of multiple tags 

Store Gateway

because Sidecar The data is backed up to the object store , So we will Prometheus Set the retention time of local data to be shorter to reduce disk space . So we also need a component to query the historical data in the object store , This component is Store Gateway, Same as Sidecar equally , It also provides StoreAPI Interface , And need to be Query Component calls .

thanos store \
    --data-dir             /var/thanos/store \   # Disk space for local caches
    --objstore.config-file bucket_config.yaml \  # Bucket to fetch data from
    --http-address         0.0.0.0:19191 \       # HTTP endpoint for collecting metrics on the Store Gateway
    --grpc-address         0.0.0.0:19090         # GRPC endpoint for StoreAPI

Store Gateway It takes up a small amount of disk space to cache the basic information of the data in the object storage , Usually not more than GB, Caching can provide Store Gateway Restart speed , But it's not necessary , It is also OK to delete the cache information before restarting .

Compactor

Prometheus It will compress regularly （compact, Merge ） Historical data to improve query speed ,Thanos You also need a component to perform the same process , This component is compactor.Compactor It will scan the object and store the data needed block perform compact operation . meanwhile ,Compactor You can also create downsampling data , And configure the retention time of data .

thanos compact \
    --data-dir             /var/thanos/compact \  # Temporary workspace for data processing
    --objstore.config-file bucket_config.yaml \   # Bucket where to apply the compacting
    --http-address         0.0.0.0:19191          # HTTP endpoint for collecting metrics on the Compactor

Compactor The component is not on the query and data backup path , It can be executed as a scheduled task , You can also stay in the background to process data in time . It is recommended to provide 100-300GB Local space for Compactor To process data .

Compactor Component can only run one , And if you manually modify the data in the storage , You can no longer run this program .

See ：Thanos Compactor Components use _ A sunny afternoon blog -CSDN Blog

Ruler/Rule

If you need a global view alerting rules and recording rules, You can use this component . Otherwise, it is not recommended , It is recommended to use Prometheus Self contained alert function .

The configuration method of this component is shown in ：Rule