Chapter 3 of the O’Reilly book I’m co-authoring with Kentik CEO Avi Freedman is out today (you can download it here) and I wanted to share a bit that didn’t make it through the editing process, because even with the eBook edition, space (and reader attention) isn’t infinite.

Nevertheless, I thought the longer version was worth a look, for some of the points it made.

In the book, we wrote:

In blog articles you’ll see the phrase “reason about”, as in “its more data than one person can reason about”, which is a fancy way of saying “keep it all in your head enough to be able to find patterns”. I mention this because one differentiations between “observability” from “monitoring” is not just the sheer volume of data, but also the disparity of that day that makes it difficult, if not impossible, to reason about it.

But here’s the longer version that we had to cut down:

In order to really appreciate how Machine Learning (ML) and AI can improve observability, it’s necessary to (briefly) differentiate between it (meaning: “observability”) and “monitoring”.

Monitoring is mostly concerned with “known unknowns”, by which I mean the things we know COULD potentially fail, but we don’t know WHEN. It’s comparatively trivial to understand what data needs to be collected, reported, and alerted on because we know where to look. Monitoring isn’t concerned with how unique the data is (i.e. “cardinality”) – whether we’re talking about an individual data point or a collection/grouping/slice of that data. And because of the two previous points (known unknowns and cardinality), correlation is usually manual – meaning humans who understand the combination of elements that indicate something is wrong are needed to craft the trigger for an alert or other action.

Observability, on the other hand, is more biased toward “unknown unknowns” – we don’t know what could go wrong and we also have no idea when that failure might occur. That ties into observability’s focus on “high cardinality” data – when specific data points or various permutations have never (or rarely) been observed. To achieve this, massive amounts of data need to be collected, normalized, parsed, and pivoted to find those unique combinations. And because of those two things (unknown unknowns and cardinality), correlation cannot be trusted to a human. There is simply too much data, and too many possible combinations of that data, to “reason about” it (to use phrase popularized by Charity Majors and Liz Fong-Jones).

This is a comparison I make often when I’m giving conference talks – NOT because I think I have the One True Definition (far from it) but rather because I want folks to have the proper context.

One additional point I make on both sides is “signals” – the data elements used to determine if the object of monitoring is healthy or not.

In “traditional” monitoring, signals are domain-specific. The data points that tell you a network device is not performing well are measurably different from those that indicate a database is unhappy. Or a block of code. Or even a wifi environment.

Meanwhile the philosophy of observability tries to standardize on the “Four Golden Signals” as established in the SRE Handbook – namely Latency, Traffice, Errors, and Saturation.

To be clear: the title of this blog is a misnomer. It’s not (and never was) “versus”. Monitoring and Observability should be seen as a BOTH-AND rather than EITHER-OR proposition.

With all that said, here’s a screenshot of the slide I use in my presentations, which lays it out (without the talk track). I hope you find it useful.