"Write multi tenant" implementation of Prometheus and thanos receiver

Thanos Receiver So called “ multi-tenancy ”, It is mainly reflected in the allocation of different services to different tenants Receiver（ Restricted energy direction Receiver Source of sending data ）, And create different instances locally for different tenants TSDB.

Thanos Receiver Component reception Prometheus Of remote_write request , Write data locally TSDB, And regularly will TSDB block Upload to object store .Receiver The components are also exposed Store API to Querer Component to realize real-time query of indicators .

Thanos Receiver Multi tenant support , It is from remote_write Requested HTTP header（ Through parameters --receive.tenant-header="THANOS-TENANT" Set up ） Get the tenant's id, Each tenant has an independent TSDB, Data submitted to the tenant successfully TSDB, I.e. on behalf of remote_write The request is successful . Indicators will be tagged tenant_id（ Through parameters -receive.tenant-label-name="tenant_id" Set up ） To identify the tenant of the indicator .

Prometheus It can be configured in remote_write Attach... To the request header：

remote_write:
  - url: http://tenants:19291/api/v1/receive
    headers:
      THANOS-TENANT: a

If you want to achieve multiple Receiver Load balancing and data replication , You need to use the configuration file （ adopt --receive.hashrings-file=<path> Parameter assignment ） Will be multiple Receiver Form hash ring , The instance configuration file is as follows ：

[
   {
       "hashring": "tenant-a",
       "endpoints": ["tenant-a-1.metrics.local:19291/api/v1/receive", "tenant-a-2.metrics.local:19291/api/v1/receive"],
       "tenants": ["tenant-a"]
   },
   {
       "hashring": "tenants-b-c",
       "endpoints": ["tenant-b-c-1.metrics.local:19291/api/v1/receive", "tenant-b-c-2.metrics.local:19291/api/v1/receive"],
       "tenants": ["tenant-b", "tenant-c"]
   },
   {
       "hashring": "soft-tenants",
       "endpoints": ["http://soft-tenants-1.metrics.local:19291/api/v1/receive"]
   }
]

The above configuration file means ：THANOS-TENANT The value is a Write request for , Will be made by tenant-a-1.metrics.local:19291/api/v1/receive"、"tenant-a-2.metrics.local:19291/api/v1/receive" Two Receiver Handle ,THANOS-TENANT The value is b or c Write request for , Will be made by tenant-b-c-1.metrics.local:19291/api/v1/receive"、"tenant-b-c-2.metrics.local:19291/api/v1/receive Two Receiver Handle , No, THANOS-TENANT header Or its value is not a、b、c One of the write requests , By http://soft-tenants-1.metrics.local:19291/api/v1/receive This Receiver Handle .

There are two concepts involved here

• Soft tenancy： If the tenant is not explicitly specified in the hash ring configuration file , Then any tenant is considered to be a valid match . All not set HTTP header Or tenant ID Tenant write requests that do not match other hash rings , Will be assigned to the hash ring . Corresponding Receiver The default tenant tag and value will be added to the indicator （ You can use the parameters –receive.default-tenant-id Appoint ）.
• Hard tenancy： be-all remote_write The request must go through HTTP header Explicitly specify the tenant ID, The request will enter the matching hash ring for processing .

Be careful ：remote_write Requests can be sent to a group of Receiver Any one of , But the final request will be handed over to the matching Receiver To deal with . therefore , Can be in Receiver Add a load balancer to distribute requests randomly Receiver Load balancing of .

From the current documentation ,Thanos Read components such as Querier Etc. does not support multi tenancy , Only through labels （tenant_id） To filter tenant data . If you want to implement multi tenant read , You need to use a third-party program , Please see ： Use prom-label-proxy Realization Prometheus Thanos Label based multi tenant reading _ A sunny afternoon blog -CSDN Blog