Narration: This presentation covers the full Cost Event Consumer project —
a Go service that integrates with OSAC fulfillment-service for capacity-based
and consumption-based cost tracking. We'll walk through the architecture,
what we built, and what's proven.
Narration: This is the core pipeline. An operator creates a resource in OSAC —
a VM, cluster, or bare metal instance. The Watch stream delivers events in
sub-second to our Watcher, which stores them in inventory. Every 60 seconds
the metering sweep reads all RUNNING resources and produces usage entries —
CPU core-seconds, memory GiB-seconds, uptime. Every 30 seconds the rating
sweep multiplies usage by rates to produce dollar costs. The raw event log
is forwarded to Splunk via HEC for audit.
Narration: This is the consumption-based side — MaaS token metering.
The IPP external-metering plugin in the OSAC AI gateway calls our
checkBalance and reportUsage endpoints. On every inference request,
we check the tenant's budget. On every response, we receive a
CloudEvent with token counts and feed it through the same metering
and rating pipeline. Verified against the upstream IPP source code
and OpenAPI spec.
Narration: We have two recorded demos available. Demo 1 shows the full
pipeline end-to-end from the terminal — creating a VM in OSAC and watching
cost entries appear. Demo 3 shows the same pipeline through the live
dashboard with MaaS traffic, per-tenant pricing, and CSV export.
Narration: Here's everything we built. The core is OSAC integration with
capacity-based and consumption-based metering. On top of that: a rating
engine with tiered pricing and per-tenant overrides, custom metric
extraction that handles new resource types with zero code changes,
quota and reporting APIs, bare metal costing, audit trail forwarding
to Splunk, full observability, CI with two integration test suites,
and developer tooling. Four rounds of adversarial review with zero
critical or high findings open, and the stress test proved 850 req/s
sustained. 12 of 18 requirements are done.
Narration: The built-in dashboard shows the pipeline in action — cost
split across tenants, Infrastructure vs Supplementary classification.
The Environment tab shows OSAC connection, database (credentials masked),
processing intervals. All served from the binary itself.
Narration: Prometheus metrics on a separate port without auth so
Prometheus can scrape without a JWT — same pattern as Koku. Counters
for events, metering entries, cost entries. Histograms for sweep
duration. The Grafana dashboard is pre-provisioned and auto-refreshing.
Narration: LOG_FORMAT=json for production. Every request gets a request
ID. Probe endpoints are exempt from JWT auth so Kubernetes can reach
them. Graceful shutdown drains in-flight requests. If a goroutine panics,
the error propagates to the errgroup and the pod restarts.
Narration: Every PR runs 6 CI jobs. Integration test 1 deploys the full
OSAC stack on k3s — gRPC, REST gateway, OIDC mock, two PostgreSQL
instances, and our consumer — then runs 12 end-to-end checks.
Integration test 2 deploys the MaaS pipeline on k3d — Istio, IPP
external-metering plugin, echo LLM, and our consumer — and verifies
balance checks and usage reporting at load.
Narration: This is the most important functional feature we added.
OSAC is evolving — new resource types, new metrics. Without custom
metrics, every new dimension means a code change. With it, an operator
drops a JSON config file and the system meters it automatically. No
code changes, no recompile, no redeploy.
Narration: Left side is the CloudEvent that arrives. Right side is the
config that tells the system which fields to extract as meters. The
field names in the config point at the field names in the event data.
Rating, reporting, and quotas all work on free-text meter names — so
custom metrics flow through the entire pipeline with zero code changes.
Narration: Bruno is a local HTTP client like Postman but file-based —
the collection is committed to git. Each request has documentation with
valid parameter values. Click to fire, see the response. Great for demos
and for developers exploring the API.
Narration: The main open item is MaaS tenant attribution — we've
researched the IPP pipeline and found that TokenMetadata on the
MaaSSubscription CRD has the fields we need but they're not wired
through to the CloudEvent. Project-level quotas and catalog pricing
are new requirements from Pau's review. The gRPC Watch PoC is done
and tested — ready to switch if needed.
Narration: Three spikes are in progress. Koku integration explores how
to feed our cost data into Koku's existing daily summary tables — the
path to using Koku's reporting and UI. GoRules/Zen evaluates a
programmable rules engine for complex rating logic beyond flat and
tiered rates. FOCUS export adds a cost export endpoint in the FinOps
Foundation's standard format for interoperability with third-party
cost tools.