The Logs You Don't Keep Can't Tell You What Happened

The quarterly cloud spend review always finds a target. This quarter, it is the SIEM ingestion bill. The infrastructure team points out that DNS query logs and S3 data events trigger zero alerts and account for sixty percent of the volume. The proposal to drop them passes to save forty thousand dollars a month. Eight months later, a database export is exfiltrated from a bucket. The incident response team needs the IP address that pulled it. There is no record.

Detection and evidence are two different budgets, but they get merged into the same line item. The SIEM bill arrives monthly. The cost of an absent log arrives with the investigation.

A SIEM is an alerting engine, not an archive. Treating it like an archive is why the ingest bill is out of control. Pouring every VPC flow log and S3 data event into a SIEM will bankrupt an IT department. Dropping them entirely blinds the incident response team. The middle ground is routing the noise to cheap, immutable cold storage.

The default is not a posture.

The problem often starts before the SIEM renewal. Platform defaults are product decisions designed to protect the vendor’s storage capacity, not your incident timeline. Entra sign-in logs sit at 30 days on the free tier. CloudTrail is off by default. S3 data events are off because the per-event charge scared the finance team three years ago. Nobody chose that posture; it arrived with the account.

When the external forensics lead joins the day-zero bridge and asks for Active Directory object modifications from six months ago, the security engineer opens the tenant configuration. The retention is set to the default 30 days. The call goes silent. The team will have to guess how the threat actor escalated privileges.

Threat actors use your retention policy as a cloaking device. If identity logs roll over every 30 days, an attacker only needs to compromise a credential, establish a backdoor, and wait 31 days to move laterally. The initial vector ages out of existence. You cannot reconstruct a 200-day dwell time with 30 days of evidence.

The question set is knowable in advance.

The investigator will ask who authenticated, from where, to what, when, with what credential, and what changed afterward. You can design retention around those six questions instead of a gigabyte ingestion limit.

A search that returns nothing cannot distinguish a quiet account from an unlogged one. “We found no record of access” is a fundamentally different statement from “we did not collect access records.” Proving a negative requires intact records. Fourteen months after an offboarding, an investigator might ask what a contractor accessed in the six weeks before they left. If the access logs rolled at 90 days, the answer is that you can tell them the contractor had access, but not what they used.

The practical judgment is not to keep every byte in an expensive search tier. It is to separate collection, search, and preservation.

Alerting requires a fraction of the telemetry that forensic reconstruction requires. Build two paths. Keep 30 to 90 days of data in the hot SIEM for search. Route the high-volume, low-alert logs directly into an object storage bucket configured with immutability, like S3 Object Lock in compliance mode or Azure immutable blobs. Set a lifecycle policy to move them to a cold archive after 90 days, retaining them for 12 to 24 months or whatever the regulator demands.

The audit trail must live in an account or subscription that the production team does not administer. If the credentials used in an intrusion belong to the team that runs production, and they can delete the audit trail, you have a governance problem. The blast radius of the audit trail should be smaller than the blast radius of the thing it is auditing. Break-glass access to the archive must be logged, alerted, and reviewed.

Store the events raw and parse them later. Ingest-time parsing locks you into a schema and a vendor. Raw events plus a parser you can rewrite in two years is how you avoid re-litigating the decision when the SIEM contract comes up for renewal.

Turn on the organization-level trail before anything else. One trail per organization, aggregated to a central bucket, with per-account trails as a backup. The failure mode is two hundred accounts, each with a trail owned by a different team, rotating on different schedules.

A cold tier with an untested restore path is a plan, not a capability.

During a tabletop exercise, the first ninety minutes often go to figuring out who has the permissions to read the archive, and whether the restore job has been tested since it was written. Define an evidence recovery time objective. Pick an event from several months ago and retrieve its original record. Confirm its timestamp, actor, target, and source environment. If retrieval depends on someone remembering an undocumented bucket name, the archive is not ready for an investigation.

A year of logs sounds excessive when every month’s invoice arrives on time. It sounds different when the first unauthorized login falls outside the retention window. Dropping a log source to save money is a valid business choice, provided the business knows exactly what it will not be able to prove later.