Metrics in software systems are important tools that offer visibility into your system. They are the main tool for observation and measuring performance. Metrics help you form a baseline for your system’s behaviour, plan for growth and detect drifts and deviations.

What’s a software metric? Simply put, it’s a measurement captured at software runtime. It’s a number that describes your application at a point in time - think CPU usage, memory usage, processed requests per second, request latency.

How do you collect metrics? Obviously, you need metrics to collect first. The operation of adding code in your application that generate metrics during runtime is called code instrumentation - it’s not a feature that your users will use, it’s an instrument for you to monitor the application.

For example, if you want to measure the total number of requests served by an endpoint in your application, you can wrap its handler in a helper function that increments a counter. To make this counter accessible during runtime, you expose it by implementing an HTTP handler that returns the value of the counter:

package main

import (
 "fmt"
 "log"
 "net/http"
 "sync"
)

var (
 mu           sync.Mutex
 requestCount int64
)

func withCount(h http.HandlerFunc) http.HandlerFunc {
 return func(w http.ResponseWriter, r *http.Request) {
  mu.Lock()
  requestCount++
  mu.Unlock()
  h(w, r)
 }
}

func helloHandler(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintln(w, "hello")
}

func metricsHandler(w http.ResponseWriter, r *http.Request) {
 mu.Lock()
 defer mu.Unlock()
 fmt.Fprintf(w, "request_count %d\n", requestCount)
}

func main() {
 http.HandleFunc("/hello", withCount(helloHandler))
 http.HandleFunc("/metrics", metricsHandler)
 log.Fatal(http.ListenAndServe(":8080", nil))
}

Now you can simply call the endpoint to check the value of the counter during runtime.

Adding metrics to your application will stack up quickly, but there are mature libraries and tools for instrumenting your code in a structured manner. Don’t reinvent the wheel.

The standard used for observability at the time of this writing is OpenTelemetry (OTel). It is defined as an open source observability framework for cloud native software. It provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application.

Basically, it’s a convention of APIs to follow for observability: you instrument your application and expose the metrics according to this standard (i.e. use an OTel compatible library as a dependency to your application and call its functions to instrument your code). Then you can connect observability tools that follow this standard in a plug-in manner (i.e. tell them to pull your API endpoints that expose metrics) .

This enables developers and operators to simply deploy a tool and connect it to their app without writing custom code for wiring observability services. It also allows swapping a tool with another without any code changes as long as the new tool follows the OpenTelemetry standard.

After you instrument the code and you expose the metrics, you need to collect them. That means calling the endpoints in your service that expose metrics every now and then to get those metrics. Typically this is done by a collector like the OTel Collector, Alloy or even a full monitoring system like Prometheus.

The frequency of collecting metrics from an instrumented application is called a scrape interval: it’s a very important knob in observability and can make or break metrics:

• If it’s too low, you’ll collect metrics very frequently:
  • You’ll put load on your service and decrease availability of the features intended for your users
  • You’ll have a lot of requests to process and you’ll need more storage to store the metrics which costs more money.
• if it’s too high, you’ll collect metrics rarely:
  • You’ll lose insights into your data - spikes that happen intermittently and recover fast won’t be seen.

You must think deep about the behaviour of your service and adjust accordingly - if you want more real time insights, you can set a smaller scrape interval, but prepare to handle the increased load on your service and storage.

Now the metrics are collected. Finally. Next step is to view them. Let’s say you have a scraping interval of 15s and you want to plot your metric over an hour. How many points will be plotted? That would be:

3600 (one hour in seconds) / 15 (scrape interval) = 240

Not bad. But what if you zoom out? And you want the metric for 30 days? In addition, what if you have a full dashboard with multiple services and for each service you gather 5 metrics?

For 20 services that would be:

30 * 24 * 3600 (30 days in seconds) * 20 (services) * 5 (metrics) = 259,200,000

That’s around 260 million raw points. That is a lot of data. If you tried to send 260 million raw points from your database to your browser, the network request would timeout, and if it didn’t, your browser tab would freeze and crash trying to render those pixels.

There has to be a way to prevent the database, network, and browser from collapsing under the weight of millions of points and at the same time not lose the information provided to us by this data.

To achieve this, a compression strategy is used. Instead of sending raw points, these points are grouped together into buckets and summarised using a specific strategy. This is where the resolution window comes in (in Prometheus, it is also called a range vector). The resolution window gives the size of that bucket. The resolution window basically says: “For every single point you plot on the graph, don’t look at just one sample; look at a window of data spanning 1 minute leading up to that point.”

For example, Envoy proxy exposes a metric called envoy_http_downstream_rq_total. This is a counter. A counter means it is a value that only increases, but never decreases. If I would like to compute the requests per second, I would do

rate(envoy_http_downstream_rq_total[1m])

What this does:

• Look: get the value of the counter at the current timestamp.
• Look back: get the value of the counter exactly 1 minute ago
• Calculate delta: subtract the old value from the new value. This tells it how many requests happened in one minute.
• Convert to Rate: divide that delta by 60 seconds (the number of seconds in 1 minute).

The Result: You get a “Per-Second” value. If your graph shows 100, it means “In the 1 minute leading up to this point, the server was averaging 100 requests per second.”

In this case, the summarisation strategy is the rate. So instead of plotting each raw point, you only plot one which represents an aggregate of 1 minute.

The summarisation strategy depends entirely on the metric. For software observability, there are three main types of metrics:

• Counters - a value that only increases. The total number of occurrences of an operation does not offer insight into what’s happening in a system. This is where rate and irate are used to check the rate of change.
• Gauges - a value that can go up or down, e.g. CPU usage, memory usage, temperature, number of active threads. Use:
  • Average (avg) to see the general trends.
  • Max (max) to check worst case scenarios.
  • Min (min) to create baselines.
• Histograms - a distribution of values bucketed by size, e.g. request latencies grouped as “under 10ms”, “under 50ms”, “under 100ms”… Each bucket is a counter. Use: histogram_quantile to compute percentiles — e.g. p95 answers “95% of requests finished faster than X ms”.

One important note: there is a risk with choosing a resolution window that’s too low or too high:

• A resolution window that’s too low won’t have enough data to show the correct behaviour in your system -> if your scrape interval is 1 minute, you cannot use a resolution window for irate that’s lower than 2 minutes -> you need two points at least to compute irate.
• A resolution window that’s too high can smooth the plotted graph and hide spikes in the data.

Zoom in or out according to your needs, but don’t break the math of the summarisation functions. Zooming in will be more responsive, you can see things happening closer to real-time. Zooming out will help you see the overview and the trends of metrics.

So we saw two important configurations:

• The scrape interval, which answers “How often do we collect?”
• The resolution window, which answers “How do we group the data?”

But there is one more configuration to worry about, that is the chart step. The chart step answers the question: “How do we draw the dots?“. This one is for the observability tools used to view the data, e.g. Grafana. The chart step decides how many points to draw on the screen, e.g., “I want a point every 5 minutes”.

If you are looking at a 1-hour graph, your dashboard might only ask for a data point every 15 seconds. If you use a 1-second or 5-second resolution window on a graph that only plots a point every 15s, you are “skipping” data. You’ll see spikes, but you might miss the ones that happened between the 15-second intervals. The graph step should ideally be equal to or smaller than the resolution window.

Luckily, professional visualisation tools have built in functionalities that automatically check your scrape interval and the graph step, then pick the smallest possible window that ensures no data is ever skipped and no math breaks. In Grafana, for example, this is the [$__rate_interval] dynamic variable.

To sum up, this is the trifecta configuration of software metrics:

Practical Cheat Sheet

This Practical Cheat Sheet is mostly AI generated.

A few rules of thumb and queries to keep handy.

Rules of thumb

• rate() needs at least 2 samples in its window. Make the window ≥ 4× your scrape interval so counter resets and missed scrapes don’t break the math.
• irate() uses only the last 2 samples regardless of window size. Window ≥ 2× scrape interval is the floor — go a bit higher to absorb scrape jitter.
• Use rate() for dashboards and trends; irate() for spike-detection alerts; increase() when you want a total count over a window (e.g. “sign-ups in the last hour”).
• The resolution window should be ≥ the chart step, otherwise the graph skips data between pixels.
• In Grafana, prefer [$__rate_interval] over a hardcoded window — it adapts to the scrape interval and the dashboard zoom.
• Keep label cardinality bounded. Never use user IDs, request IDs, or raw paths as label values — every unique combination is a new time series in RAM.

Window picks by scrape interval

Common PromQL queries

Requests per second:

sum(rate(http_requests_total[5m]))

Error rate (5xx as a ratio of total):

sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

p95 latency from a histogram:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Total count over the last hour:

sum(increase(http_requests_total[1h]))