FinOps & Cloud Economics · Sep 2026 · 17 min read

Open Cloud Cost Intelligence: A Real FinOps Pipeline, Built and Run Against Real Billing Data

Ingestion, normalization, cost allocation, and an optimization engine — not against sample data, but against a real Azure resource group's real Cost Management API response. The result: a genuine finding (one service is 75% of daily spend) and a genuine gap (100% of that cost is currently unallocated by tag).

A FinOps Pipeline Against Real Numbers, Not a Mockup

"Cloud cost intelligence" architecture diagrams all look roughly the same: ingest cost data, normalize it, allocate it by team/product/environment, run an optimization engine, render a dashboard. What almost never gets shown is what that pipeline actually produces when pointed at a real, currently-running cloud environment instead of a sample CSV. This project builds all five layers and points them at MYRIX's real Azure resource group — the hub-spoke VNets, Bastion, Load Balancer, and VM from an earlier article in this series, genuinely running and genuinely accruing cost.

Live demo: cloud-cost-intelligence.pages.dev

Before Step 1, one term this walkthrough leans on:

Step 1 — Ingestion: a real call to Azure's Cost Management Query API

No SDK to install here — no @azure/arm-costmanagement, no service principal, no client secret. az rest is a pass-through built into the Azure CLI itself: if az login already works on your machine, this works too. There is a real catch worth knowing before you check az costmanagement --help: the az costmanagement extension only exposes an export subcommand, not query — the query API has to be called directly via az rest against the REST endpoint, which is what this function does.

export function fetchAzureCostData(resourceGroupFilter?: string): NormalizedCostRecord[] {
  const subscriptionId = execSync("az account show --query id -o tsv").toString().trim();

  const raw = execSync(
    \`az rest --method post --url "https://management.azure.com/subscriptions/\${subscriptionId}/providers/Microsoft.CostManagement/query?api-version=2023-11-01" --body @/tmp/cci-query.json\`
  ).toString();

  // ...parse columns/rows into NormalizedCostRecord[]
}

Why az rest instead of a separate SDK or stored credential: this reuses the exact same authenticated az CLI session every other Azure article in this series already relies on — no service principal, no client secret, no separate auth flow to build or document. The tradeoff is real, and worth stating: this makes the pipeline a CLI tool run by a human (or a CI job already authenticated to Azure), not a client-side dashboard that could run in a browser with its own credentials.

A real operational constraint hit immediately: Azure's Cost Management Query API enforces an aggressive per-subscription rate limit — repeated calls during development returned:

ERROR: Too Many Requests({"error":{"code":"429","message":"Too many requests. Please retry."}})

roughly one successful query per several minutes. This isn't documented prominently anywhere obvious, and it's a genuine design constraint for anything built against this API: a dashboard that tries to poll it live, per-request, from multiple users would exhaust that budget almost immediately. That's the actual reason this project's dashboard reads a pre-generated snapshot file rather than calling the API live from the browser — not just a security choice (keeping credentials off the client), but a rate-limit one too.

Step 2 — Normalization: one shape, regardless of cloud

export interface NormalizedCostRecord {
  provider: 'azure' | 'aws' | 'gcp';
  date: string;
  resourceGroup: string;
  service: string;
  cost: number;
  currency: string;
  tags: Record<string, string>;
}

Why this matters even with only one provider implemented: Azure's raw Cost Management response is a columns-array-plus-rows-array shape that has nothing in common with AWS Cost Explorer's ResultsByTime structure or a GCP billing export's BigQuery row format. Every layer downstream of ingestion — allocation, optimization, the dashboard — only ever touches NormalizedCostRecord. Adding AWS or GCP later means writing one more ingestion adapter that produces this same shape; it doesn't mean touching allocation, optimization, or the dashboard at all.

Step 3 — Allocation: a real, honest gap

az resource list --resource-group myrix-net-rg --query "[].{name:name,tags:tags}" -o json

[
  { "name": "vnet-hub", "tags": null },
  { "name": "bastion-myrix", "tags": {} },
  { "name": "vm-myrix", "tags": {} }
]

Zero cost-allocation tags existed on any resource in this group before this project started. Rather than skip the allocation step or fabricate tags to make a demo look complete, real team/product/environment/workload tags were applied to the resource group and its top cost-driving resources as part of building this:

az resource tag --ids <resource-id> \
  --tags team=platform-engineering product=myrix environment=lab workload=game-networking-demo \
  --is-incremental

Why the dashboard still shows 100% unallocated right now: Azure's cost data has a real lag before newly-added tags appear in billing records — tags applied today don't retroactively populate into cost rows already recorded for today. The honest snapshot, taken minutes after tagging, correctly shows every dollar as unallocated. That's not a bug in the allocation code; it's the allocation code correctly reporting a gap that will close in Azure's own next billing cycle, not before.

Step 4 — Optimization: one real finding, from real data

const ALWAYS_ON_SERVICES = ["Azure Bastion", "Load Balancer", "VPN Gateway", "NAT Gateway"];
if (share > 0.5 && ALWAYS_ON_SERVICES.includes(service)) {
  // flag it
}

Why "always-on cost concentration" was the first heuristic built, not rightsizing or idle-detection: it's the one the actual data justified. Querying myrix-net-rg's real cost breakdown for real:

[
  { "service": "Azure Bastion", "cost": 2.5111 },
  { "service": "Virtual Machines", "cost": 0.3224 },
  { "service": "Load Balancer", "cost": 0.2202 },
  { "service": "Container Registry", "cost": 0.1190 }
]

Azure Bastion alone accounts for 75% of this resource group's total daily spend — more than the VM, the Load Balancer, and the Container Registry combined. Bastion bills a fixed hourly rate for existing, whether or not anyone is actively using it to SSH into anything. At $2.51/day, that's roughly $75/month for a jump host in a lab environment — a completely ordinary, completely real FinOps pattern: an always-on management service quietly outspending the workload it exists to manage.

Step 5 — The dashboard: a real snapshot, not a live poll

The dashboard is a single static HTML page that fetches data/snapshot.json — a file the pipeline writes, not an API the browser calls directly:

fetch('./data/snapshot.json')
  .then((r) => r.json())
  .then((snapshot) => { /* render totals, allocation, findings */ });

Why this is the correct architecture, not a shortcut: an Azure credential capable of querying Cost Management data has real read access to billing information — putting that credential in client-side JavaScript would expose it to anyone who opens the browser's network tab. Combined with the rate-limit constraint from Step 1, a static, periodically-regenerated snapshot is the honest architecture here, not a live dashboard with a spinner that happens to be slow.

Closing Thoughts

Every layer in the original architecture diagram — ingestion, normalization, allocation, optimization, dashboard — is real and runs against real data in this project. What real data adds that a mockup can't: a genuine 429 from an undocumented rate limit, a genuine 100%-unallocated number because tags were added after the fact, and a genuine finding that a jump host costs more than the thing it's guarding. None of those three things would exist in a version of this project built against sample CSVs — they're specifically what shows up when the pipeline points at something that's actually running and actually billing.

GitHub Repository: cloud-cost-intelligence — the full pipeline, the real snapshot this article quotes, and an honest "what's not built yet" section covering the AWS/GCP adapters and remaining optimization checks the architecture supports but doesn't yet implement.

Reviewed against the Azure Cost Management Query API (2023-11-01) as of September 2026.

FinOps · Azure · Cost Management · Cloud Economics · Cost Allocation