tsdbinfo

tsdbinfo - Understand the series and labels you store in Prometheus

tsdbinfo is a tool that looks into the data folder of Prometheus and gives you a basic understanding of what is stored there.

Install

Linux

curl -LO https://github.com/laszlocph/tsdbinfo/releases/download/v0.2.0/tsdbinfo-v0.2.0-linux-amd64
mv tsdbinfo-v0.2.0-linux-amd64 tsdbinfo
chmod +x tsdbinfo

Mac

curl -LO https://github.com/laszlocph/tsdbinfo/releases/download/v0.2.0/tsdbinfo-v0.2.0-mac-amd64
mv tsdbinfo-v0.2.0-mac-amd64 tsdbinfo
chmod +x tsdbinfo

Basic usage

First make a copy of the data in your Prometheus --storage.tsdb.path path.

Running tsdbinfo on the same path - in parallel - with your production Prometheus may cause race conditions with unpredictable results. More on Copying the Prometheus data folder

List all the blocks

  ➜  tsdbinfo blocks --storage.tsdb.path.copy=/my/prometheus/path/data-copy --no-prom-logs
  ID                            FROM                         UNTIL                        STATS
  01CZWK46GK8BVHQCRNNS763NS3    2018-12-22T13:00:00+01:00    2018-12-29T07:00:00+01:00    {"numSamples":3167899784,"numSeries":3070548,"numChunks":29336192,"numBytes":4419004512}
  01D1EFWJ44G9WGN7AQ9398G2W2    2019-01-11T01:00:00+01:00    2019-01-11T19:00:00+01:00    {"numBytes":8634}
  01D1EFWJRQ35VYNT2M4YYEJV3R    2019-01-16T07:00:00+01:00    2019-01-17T01:00:00+01:00    {"numBytes":8634}

Identify the largest metrics

  ➜  tsdbinfo metrics --storage.tsdb.path.copy=/my/prometheus/path/data-copy --block=01CZWK46GK8BVHQCRNNS763NS3 --no-bar  --no-prom-logs --top=3
  METRICSAMPLES        SERIES     LABELS
  solr_metrics_core_errors_total                              164,291,959    4,229      core: 99, handler: 32, collection: 16, replica: 9, instance: 5
  solr_metrics_core_time_seconds_total                        164,291,959    4,229      core: 99, handler: 32, collection: 16, replica: 9, instance: 5
  solr_metrics_core_timeouts_total                            164,291,959    4,229      core: 99, handler: 32, collection: 16, replica: 9, instance: 5

Investigate label explosion

  ➜  tsdbinfo metric --storage.tsdb.path.copy=/my/prometheus/path/data-copy --block=01CZWK46GK8BVHQCRNNS763NS3 --metric=http_server_requests_total --no-prom-logs
  Metric        http_server_requests_total
  Samples       5,365,975
  TimeSeries    582
  Label         path                    172
  Label         kubernetes_pod_name     14
  Label         instance                14
  Label         code                    10
  Label         pod_template_hash       7
  Label         method                  6
  Label         version                 5
  Label         app                     5
  Label         feature                 2
  Label         kubernetes_namespace    2
  Label         k8scluster              2
  Label         __name__                1
  Label         job                     1
  LabelValue    pod_template_hash       4120792344
  LabelValue    pod_template_hash       1980135293
  LabelValue    pod_template_hash       102934907
  LabelValue    pod_template_hash       1006272702
  LabelValue    pod_template_hash       3602012261
  LabelValue    pod_template_hash       3571852123
  LabelValue    pod_template_hash       3513057117
  LabelValue    code                    400
  LabelValue    code                    401
  LabelValue    code                    406
  LabelValue    code                    200
  LabelValue    code                    301
  LabelValue    code                    302
  LabelValue    code                    304
  LabelValue    code                    404
  LabelValue    code                    422
  LabelValue    code                    500
  LabelValue    method                  get
  LabelValue    method                  post
  LabelValue    method                  head
  LabelValue    method                  put
  LabelValue    method                  patch
...

Uncover the sources of cardinality explosion in Prometheus

tsdbinfo is best used to understand what labels you store and spot cardinality explosion that is bad for your Prometheus: https://prometheus.io/docs/practices/naming/#labels

Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

Design

It follows the Prometheus timeseries database (tsdb) notions: blocks, series, labels and samples. Data is organized into blocks that hold all samples of a time period. Each metric has a number of series associated with it and the series has samples. Each label combination defines a new timeseries. You can find more on blocks here: https://fabxc.org/tsdb/

Copying the Prometheus data folder

If you don't have the space to duplicate your Prometheus data files, just take a sample of the folders as each subfolder of the --storage.tsdb.path is self containing.

This is what I do:

$ ls -l /data/prometheus
total 88
drwxr-xr-x 3 prometheus prometheus 4096 Nov  5 09:02 01CVHHMPR9E7FANBKZZX7WVN2Y
drwxr-xr-x 3 prometheus prometheus 4096 Nov 12 03:02 01CW2XTTVMBR0PYDZ0KZ7C8HMG
drwxr-xr-x 3 prometheus prometheus 4096 Nov 18 21:02 01CWMA0JF9PZABN8JQ3HAR7K22
drwxr-xr-x 3 prometheus prometheus 4096 Nov 25 15:02 01CX5P6DY7RK8XFWY37PB4T1SY
drwxr-xr-x 3 prometheus prometheus 4096 Dec  2 09:01 01CXQ2C1YDJTRZATV8XYDG3W5N
drwxr-xr-x 3 prometheus prometheus 4096 Dec  9 03:01 01CY8EJ3T6670KC3Q9EQ8PV6DB
drwxr-xr-x 3 prometheus prometheus 4096 Dec 15 21:02 01CYSTR2E1M23DEAXM0C808BSR
drwxr-xr-x 3 prometheus prometheus 4096 Dec 22 15:02 01CZB6Y6922TDXYTR6GVRW3ADV
drwxr-xr-x 3 prometheus prometheus 4096 Dec 29 09:02 01CZWK46GK8BVHQCRNNS763NS3
drwxr-xr-x 3 prometheus prometheus 4096 Jan  5 03:02 01D0DZ9Q7JP7VHA1T00PA3ZJ71
drwxr-xr-x 3 prometheus prometheus 4096 Jan 11 21:01 01D0ZBF8C2DD5QC8WD8V765SEX
drwxr-xr-x 3 prometheus prometheus 4096 Jan 18 15:01 01D1GQNBDP2NXH6BVN1E022VPS
drwxr-xr-x 3 prometheus prometheus 4096 Jan 25 09:02 01D223V9N0YPN673FZPTXPDME8
drwxr-xr-x 3 prometheus prometheus 4096 Jan 27 15:00 01D27X71GWJE0SAB2P6H98FE4W
drwxr-xr-x 3 prometheus prometheus 4096 Jan 28 09:00 01D29V089NE4VF321Q2TA8FF3T
drwxr-xr-x 3 prometheus prometheus 4096 Jan 29 03:00 01D2BRSQCV29G3SFTB5GAXMXV1
drwxr-xr-x 3 prometheus prometheus 4096 Jan 29 09:00 01D2CDCPWR58D42P8MVGKT0B1S
drwxr-xr-x 3 prometheus prometheus 4096 Jan 29 15:00 01D2D1ZJ7F2ZZP2H2PREEFFEFX
drwxr-xr-x 3 prometheus prometheus 4096 Jan 29 15:00 01D2D2006A2HK3V8A4SSZR92XS
drwxr-xr-x 3 prometheus prometheus 4096 Jan 29 17:00 01D2D8V9E2YVRW1BE884YKKW27
-rw-r--r-- 1 prometheus prometheus    0 Aug  9 12:37 lock
drwxr-xr-x 2 prometheus prometheus 4096 Jan 29 17:43 wal

Then copying recent chunks:

$ mkdir /data/prom-analysis
$ cd /data/prom-analysis
$ cp -r ../prometheus/01D*/ .

Then running tsdbinfo:

$ tsdbinfo blocks --storage.tsdb.path.copy=/data/prom-analysis