meters

Overview ¶

Package meters implements lightweight metrics for internal use.

A meter is a name, a set of key/value pairs (set by the underlying logger), and a value. A value can be an absolute value ("gauge"), an incremental value ("counter"), or a set of samples (like "histogram", though lossless in this implementation). Meters are most useful when additional code analyzes the entire log of a program's run; this is called "analysis code" or "anaylsis software" throughout the documentation.

Values are stored by writing the set of operations to build the value to the logs. For example, each time a sample is added to a sampler with Sample, a log line is produced. Reading the logs will allow analysis software to recover the entire set of samples. Counters are similar; each increment event emits a log line, and and analysis code can add the deltas to see the final value, or the value at a particular point in time.

Values are logically associated with key=value metadata. Each value is uniquely identified by the set of key=value metadata; a meter foo with field foo=bar is logically a different value from the meter foo with field foo=baz. The metadata is defined by the fields applied to the underlying logger. For example, if you declare a counter `tx_bytes` on each HTTP request you handle, each time the count of transmitted bytes is updated, the log message will include fields inherited from the default HTTP logger, which includes a per-request ID. Therefore, analysis code can calculate a tx_byte count for every request handled by the system by observing the value of the x-request-id field on log lines matching the tx_byte counter format. And, of course, it can ignore the extra fields and add everything up to show a program-wide count of bytes transmitted.

Because pctx.Child allows the caller to define a namespace for logs and metrics, meter names do not need to be namespaced. Prefer a meter named `tx_bytes` in the `chunk_storage.Upload` logger over one named `chunk_storage.upload.tx_bytes`.

Normally it's considered "too expensive" to store metrics with such a high cardinality (per-user, per-IP, per-request), but this system has no such limitation. Cardinality has no impact on write performance. Analysis code can make the decision on which fields to discard to decrease the cardinality for long-term storage, if desired. Be aware that the usual log sampling rules apply to logged meters.

Aggregating each operation on a meter over time recovers the value of the meter at a particular time. Any sort of smartness or validation comes from the reader, not from this writing code. If you want to treat a certain meter name as a string gauge, integer counter, and sampler of proto messages, that is OK. The analysis code that processes the logs will need to be ready for that, or at least ready to ignore values it doesn't think are valid.

To emit meters, simply call these public functions in this package:

Set(ctx, "helpful_name", value) // Set the current value of the meter to value.
Inc(ctx, "helpful_name", delta) // Increment the current value of the meter by delta.
Sample(ctx, "helpful_name", sample) // Add sample to the set of samples captured by the meter.

It is safe to write to the same meter from different goroutines concurrently.

Some minimal aggregation can be done in-process. This is a compromise to reduce the "noise" in the logs. If you were uploading a 1GB file, it would make sense to increment the byte count meter 1 billion times, as each byte of the file is passed to the TCP stack. This, however, would be very noisy and make the logs difficult to read. So, we offer aggregates to flush the value of gauges (Set) and counters (Inc) periodically, grouping many value-changing operations into a single log message.

An aggregator can be registered on a context (with pctx.Child), causing all future writes to that meter on that context to be aggregated. (The code that writes the meter need not be aware of the aggregated nature; the public Set/Inc/Sample API automatically does the right thing.)

Aggregated meters are emitted to the logs based on a time interval or value delta threshold set at registration time. If a write occurs, and the meter hasn't been logged for that interval, then a log line will be produced showing the current value of the meter. If unflushed data exists when the controlling context is Done, a log line representing the final value is also emitted. From an analysis standpoint, nothing changes; the emitted aggregated meters are indistinguishable from immediately emitted meters. It just is a little bit easier on the eyes of the human reader.

Aggregated meters can only be one type of meter (gauge/counter) with one type of value; if you create a counter named tx_bytes and then call the gauge operatoin "Set" on it, no aggregation will be done on that data. (Inc calls continue to be aggregated; it doesn't "break" the aggregation to accidentally call the wrong value-changing function.) Similarly, the Go type of the value is set at registration time; calls writing values of a different type will not be aggregated, but again, do not break any ongoing aggregation.

Finally, a couple of implementation notes. This package never panics or returns an error; every possible argument you can pass to any public function is valid. Also, functions must never create or contend on any global locks (except the lock on the logger to prevent goroutines from writing partial lines). This is so that heavy use of meters in one component does not delay other more crticial code. Locks scoped to one context chain are OK, but a global lock on something popular like `tx_bytes` is not desirable.