Why the AI era needs cheap, sovereign, scale-to-zero log search and why ELK and Loki can’t give it to you

For years, the smartest thing you could do with old logs was to delete them.

A human was only going to look at the last few hours anyway. Keeping a month of logs searchable meant paying for hot storage and for nodes that almost nobody used. So retention windows shrank to 7, 14, maybe 30 days. The rest went to a bucket that nobody could search, and everyone moved on. It was the rational choice.

It is not anymore. Three things changed at the same time. Together, they turn long and searchable retention from a niche compliance problem into a general engineering problem. And the two default open source answers, ELK and Grafana Loki, are not designed to solve it at a reasonable cost.

Three forces are breaking your logging bill

AI multiplies the logs and the reasons to read them. Every agent, copilot, and generated service is one more thing producing telemetry, and volume grows fast. But the more interesting change is on the consumer side. Old logs were not worthless because they had no signal. They were worthless because no human had time to dig through six months of history looking for a weak pattern. An AI analyst removes exactly that limit. An agent doing threat hunting or root cause analysis does not get tired. It will look into the past, because looking is cheap for it. To be clear: AI does not make your logs valuable as training data. Raw logs are noisy and full of PII, and almost nobody trains a model on them. AI makes them valuable because it is the first analyst with enough time to mine the history we used to delete.

Security needs that history, and the numbers are hard to ignore. When you detect a breach, the evidence you need is usually old. Mandiant’s M-Trends 2026 report puts the global median attacker dwell time at 14 days. For the dangerous categories, espionage and insider operations, the median is 122 days, and some intrusions stay hidden for more than a year. IBM’s Cost of a Data Breach 2025 report measures the full breach lifecycle at around 241 days. And there is a worse number: in about a third of the intrusions, Mandiant could not determine the entry point, in part because the logs were already gone. The most useful security question today is: “this indicator was published this morning, have we seen it anywhere in the last twelve months?” You cannot answer it with two weeks of retention.

Sovereignty is closing the easy way out. The path of least resistance was to send everything to a US SaaS. That path now collides with GDPR, Schrems II, the Cloud Act, and the European digital sovereignty initiatives. For many organizations, sending their most sensitive operational data, full of secrets and infrastructure details, to a third party vendor running on someone else’s cloud is no longer acceptable.

Put the three together: more logs, a real reason to keep them searchable for months, and the obligation to do it on infrastructure you control.

ELK and Loki fail here, in different ways

The two usual options fail, and they fail differently. That difference matters.

Cost to keep a log queryable: rising steeply for an Elasticsearch hot cluster, nearly flat for object storage with stateless search

On a hot cluster, the cost of keeping a log searchable grows with every day you keep it. On object storage, it stays almost flat.

Elasticsearch links retention cost to compute. In an ELK or OpenSearch cluster, the index and its replicas live on local SSD, on nodes that are always on. The data is the cluster. Keeping a log searchable for a year means paying for a year of hot or warm nodes. The cost of retention grows with every day you keep the data. You can move old indices to cold snapshots, but then they are not really searchable: you have to restore complete indices back to the nodes before you can query them, and that is a slow batch operation.

Loki is cheap because it does not really index. Grafana Loki stores chunks in object storage and only indexes labels, not the content of the logs. It is a good design for cheap storage and for tailing logs by label. But to search the content, Loki has to scan every chunk inside the label and time window you select. For full text search over high cardinality data and long time ranges, which is exactly the security and forensic case, that becomes an expensive grep.

So the choice today is bad: an always-on search cluster with a bill that grows with retention, or cheap storage that you cannot really search. Neither one matches the three forces above.

The third option: there is no database

The way out is the same change that transformed data warehousing ten years ago, when Snowflake and later the lakehouses separated storage from compute. That change is now arriving to observability, and the open source pieces to build it yourself already exist.

The core idea: stop running a log database. Your logs become immutable files in your own object storage, in an open format that you own. Search runs on stateless compute that reads those files directly and scales to zero between queries. Storage and compute are fully separated. Years of logs stay one query away, for more or less the price of a bucket, and you pay for query compute only when someone asks a question.

Reference architecture: sources to Vector to Quickwit indexers to object storage to stateless Quickwit searchers to Grafana, with a CloudNativePG metastore underneath, runnable on-prem or in any cloud

Every box is open source. The same design runs as Lambda plus S3 in the cloud, or k3s plus SeaweedFS on bare metal.

Piece by piece, all of it open source or CNCF:

• Collect and classify with Vector. Vector (the open source pipeline that also powers Datadog Observability Pipelines) parses and reshapes events, removes PII and secrets before anything is stored, and routes each category of log to a different destination and retention tier. Security logs go to one place, debug noise to another.
• Store in your own bucket. S3 in the cloud, or SeaweedFS or Ceph on premise. The cheapest durable storage available, and portable: the same architecture runs as Lambda plus S3 in a public cloud, or as k3s plus SeaweedFS on bare metal, and the data never has to move between them.
• Index and search with Quickwit. Quickwit is an open source search engine designed for object storage from the start, unlike Elasticsearch. It writes a real inverted index as immutable “splits” into the bucket. Its searchers are stateless and read byte ranges directly from object storage, so you can scale them to zero with KEDA and start them only on demand.
• Keep the catalog in Postgres. A small PostgreSQL metastore, easy to run on Kubernetes with CloudNativePG, keeps the catalog of splits: which ones exist and what data they cover. It is small, it avoids the consistency problems of object storage, and it is always on. The question “what data do we have?” is always instant, even when the data itself is cold.
• Visualize with Grafana. Quickwit has an official Grafana data source, so the front end is normal Grafana: Explore, dashboards, alerts. If you run Grafana plus Loki today, this is almost a drop-in change. Same UI, with a real search index below it.

You probably keep these logs already

The strongest objection is always “why change something that works?”. The honest answer makes the change much smaller than it looks: you probably keep this data already.

Most organizations with some compliance pressure already send old logs to S3 or Glacier, just in case. The data sits there, write-only, never queried, because restoring it is too painful. This architecture does not ask you to retain more. It makes the dead archive you already pay for searchable, at a similar cost. The first step is simple: point a stateless search engine at the bucket you already have.

Where this still loses, honestly

This is not an ELK killer, and it would be dishonest to present it as one. Search over object storage has a higher latency floor, and there is a short delay before new data becomes searchable. The design is made for low query rates over enormous volumes, not for fifty analysts refreshing dashboards on the last five minutes. And the Kibana ecosystem for SIEM, detection rules and RBAC is more mature than Grafana on top of a younger engine.

The correct framing is not “remove ELK”. It is tiered: keep your hot, interactive, last-week data where it already works, and use this pattern for the long, searchable, sovereign retention tier that the current tools make too expensive.

The part that is not solved yet

There is one open problem, and it is a good one to share with the community.

Two-speed search: the instant tier reads directly for sub-second results, while deep archive requires a targeted restore of only the splits a query needs

Most of the classic “rehydration” disappears. Only the real offline archive needs a restore, and the metastore makes it targeted.

Because searchers read object storage directly, most of the classic rehydration simply disappears. Anything stored in an instant access class, like Standard, Infrequent Access, or even Glacier Instant Retrieval, stays searchable in milliseconds with no restore step. “Cold” only means cheaper per gigabyte and a bit more expensive per query.

The exception is the real offline archive: Glacier Flexible Retrieval and Deep Archive, where a read fails until you request a restore and wait. The elegant solution is a targeted one. The metastore knows exactly which splits a query needs, so you restore only those objects, not complete indices, and then you search. Building this two-speed path, instant by default and restore on demand for the deep tail, is still open work. If someone wants to build it, it is a very good project.

Closing

The summary of the argument is short. Searchable long-term logging is no longer a luxury for regulated industries. Attackers hide for months, AI is finally the analyst with time to find them, and sovereignty rules do not allow outsourcing the problem. The architecture that makes it affordable does not look like ELK or Loki: it is a bucket you own, an open data format, and stateless search that scales to zero. Even the query engine is a replaceable piece. The data stays yours in any case.

I am preparing a conference session based on this idea, so this post is also a way to put the argument in order before building the slides. If you work on log management and you see a weak point in it, I am happy to hear it.