S3 Monitoring Beyond CloudWatch: Object-Level Visibility for Data Engineering Teams

CloudWatch tells you how much S3 storage you have and how many requests you're making. It doesn't tell you which tables are degrading, which pipelines have stopped writing, or which IAM roles are behaving unexpectedly. For data engineering teams, that missing layer is where most of the real problems live.

What CloudWatch actually gives you

CloudWatch S3 metrics,BucketSizeBytes, NumberOfObjects, AllRequests,are aggregated at the bucket level. They're useful for broad cost tracking and capacity planning, but they tell you nothing about what's happening inside your data lake at the table or pipeline level.

You can add more granularity with request metrics per prefix, but this requires manual configuration per bucket and still doesn't give you the access pattern context needed to understand data flow, pipeline health, or table degradation.

What data engineering teams actually need to see

• Which Iceberg, Delta Lake, or Hudi tables are accumulating orphaned files or snapshot bloat
• Which ETL pipelines have stopped writing,and when the last write happened
• Which IAM roles are accessing which prefixes, with what SDK version, and at what frequency
• Which Athena queries are scanning cold partitions that haven't been accessed in months
• Which prefixes have data that should be in GLACIER but is sitting in STANDARD

None of this comes from CloudWatch. All of it is present in S3 access logs and S3 inventory data,it just needs to be parsed and correlated correctly.

S3 access logs as a monitoring substrate

S3 server access logs record every request made against your buckets: the requester identity, operation type, object key, response size, and timestamp. At scale, this is hundreds of millions of log lines per day,but the signal density is high.

From access logs alone you can infer: pipeline write cadence and health, query engine access patterns, IAM role behavior, cold vs hot data by prefix, and anomalous access events. Combined with S3 inventory (which gives you object sizes, storage classes, and last-modified dates), you have a complete picture of your data lake health without touching any catalog or query engine.

The gap between raw logs and actionable monitoring

The challenge isn't that the data isn't there,it's that processing hundreds of millions of S3 access log lines per day, correlating them with inventory data, and mapping them to table and pipeline abstractions is a significant engineering problem. Most teams either don't do it at all or build partial solutions that answer one specific question.

reCost processes this layer continuously and surfaces findings as structured insights: which tables need attention, which pipelines are dead, which roles are behaving unusually. No infrastructure to deploy, no agents, no code changes. Just your existing S3 logs pointed at the platform.

What this enables in practice

• Data lake health reviews in minutes, not hours of SQL queries
• Pipeline failure detection from write pattern deviation,no instrumentation needed
• IAM security review across your entire S3 environment from one view
• Storage cost attribution at the table and prefix level, not just the bucket
• Athena cost investigation without running EXPLAIN on every query