Implementation of multi tenant read and write in Prometheus cortex

Cortex All components of the will be from each request header Of X-Scope-OrgID Get tenants from ID. The tenant here represents the owner of a set of data , It is to Cortex Write data , And have the query authority of the data . be-all Cortex Components will be unconditionally trusted X-Scope-OrgID Request , So if you want to protect Cortex Protection from malicious calls , You need to add a protective layer by yourself .

Note the tenants who query and write requests ID It has to be consistent , Otherwise, the required data cannot be queried .

Enable the multi tenant feature

By adding auth.enabled=true, Or the command line -auth.enabled=true Enable the multi tenant feature . for example ：

./cortex -target=distributor -auth.enabled=true

If you want to disable multi tenancy , You need to pass parameters to all components auth.enabled=false, In this case, all requests X-Scope-OrgID Will be set to "fake".

Prometheus To configure

The first method is to directly remote_write Add in configuration header Information .

remote_write:
  - url: http://<cortex>/prometheus/api/v1/push
    headers:
      X-Scope-OrgID: <org>

The second method is to use Cortex-Tenant, Add... Based on existing tag values header Information .

Cortex Tenant Placed in Prometheus and Cortex Between , When Prometheus When a write request for passes through the component ,Cortex Tenant Will search for the value of the predefined tag , And use it as X-Scope-OrgID Add the value of to the request header in , And then forwarded to Cortex.

See the detailed usage method ：

GitHub - blind-oracle/cortex-tenant: Prometheus remote write proxy that adds Cortex tenant ID based on metric labels

This component is a third-party component , Not Cortex Team maintenance .

Query side configuration

at present Cortex There is no front-end query page for multi tenant , In the use of Grafana As a client query , Multi tenant query can be implemented according to the following scheme .（ At present, it is only an idea , Not yet practiced and tested , The theory works ）

Scheme 1 ：

stay Grafana And Cortex Place a layer between Nginx Reverse proxy ,Nginx Add a configuration similar to the following ：

server {
    server_name  prod.com
    location / {
        proxy_pass  http://cortex;
        proxy_set_header X-Scope-OrgID <prod tenant ID>;
    }
}

server {
    server_name  ops.com
    location / {
        proxy_pass  http://cortex;
        proxy_set_header X-Scope-OrgID <ops tenant ID>;
    }
}

Grafana The side passes different tenants Org To separate from each other , Configure different data sources , In this way, different tenants can only query their own data .

Option two ：

stay Grafana When adding data sources to , To configure Custom HTTP Headers.