Your ML Model Is Wrong and You Don't Know It: Building Feature Monitoring into Feast

• The silent killer of ML systems: When a feature's null rate shifts from 2% to 25% due to an upstream bug, your credit-scoring model starts approving bad loans. Your fraud detector misses patterns. Your recommendation engine goes off the rails. And nobody notices — because feature stores traditionally have zero visibility into data quality. This isn't hypothetical; it happens routinely in production ML.

• Feast is the most widely adopted open-source feature store (4,000+ GitHub stars), used in production by teams running ML on PostgreSQL, Snowflake, BigQuery, Redshift, Spark, and more. Until now, it had no built-in monitoring — teams either built custom solutions or flew blind.

• I designed and built a comprehensive data quality monitoring system for Feast that computes statistical profiles (mean, stddev, percentiles, null rates, histograms) for every registered feature — automatically. The key architectural challenge: Feast supports 8 offline store backends with completely different SQL dialects. The solution uses SQL push-down with a PyArrow/NumPy fallback for backends that don't support it — zero data transfer for supported backends.

• Beyond data quality, I added operational observability: Offline store RED metrics (request rate, error rate, latency) via Prometheus, and structured SOX audit logging for compliance in regulated industries — answering "who accessed what features, when, and how much data."

• In this talk, I will walk through:

  • The architecture: OfflineStore abstraction with 6 new methods, SQL push-down vs Python fallback dispatch, and the DQM job engine for async computation

  • How auto-baseline on feature registeration and multi-granularity scheduling (daily/weekly/monthly) work

  • SOX audit logging design: PII-minimized, dedicated logger, never breaks the serving path

  • Live demo: monitoring run, REST API, and the React monitoring UI with histogram overlays

  • Why this matters more with agentic AI — when agents autonomously consume features for real-time decisions, drifted data has a larger blast radius

Attendees will understand why feature quality monitoring is a critical missing layer in production ML, and how to detect silent data failures before they corrupt model predictions. Also they will learn Why feature monitoring becomes a safety mechanism (not just observability) when AI agents autonomously consume features for real-time decisions like loan approvals or fraud detection.