AI Operations & Monitoring in Treasury

Building the model was the easy part. Keep it accurate, tested and auditable in production.

Building the model was the easy part

An often-quoted industry estimate says most machine learning models never make it into production (87% is the number you will see). The ones that fail rarely fail because the model is bad. They fail because operations are missing.

The treasury AI operations cycle

Model drift: when your forecast goes stale

All three end in retraining, but the fix is different: data drift may mean fixing the source, concept drift a new model, feature drift a repaired pipeline.

The treasury AI KPI framework

For net flow, which crosses zero, track the average error in EUR instead of MAPE (Module 9).

Automation: Power Automate for treasury AI

Four patterns. Each is a trigger and an action, built visually.

Power BI: the model performance dashboard

The same four panels appear in the live monitor later in this module, so you can see what they tell you before you build them in Power BI.

Testing framework: what to test, when

import pandas as pd

FLOWS = ['receipts', 'supplier_payments', 'payroll', 'tax', 'intercompany', 'other']

def load(path='treasury_cash_flows_clean.csv'):
    return pd.read_csv(path, parse_dates=['date'])

def test_no_missing_values():
    assert load()[FLOWS].isnull().sum().sum() == 0

def test_no_duplicate_weeks():
    assert not load()['week_number'].duplicated().any()

def test_weeks_are_consecutive():
    assert (load()['date'].diff().dropna() == pd.Timedelta(days=7)).all()

def test_net_flow_reconciles():
    df = load()
    assert ((df[FLOWS].sum(axis=1) - df['net_flow']).abs() < 1).all()

def test_payments_are_negative():
    assert (load()['supplier_payments'] <= 0).all()

The AI audit trail: what to document

import hashlib, json, os
from datetime import datetime, timezone
import pandas as pd
import prophet

def sha256(path):
    with open(path, 'rb') as f:
        return hashlib.sha256(f.read()).hexdigest()

fc = pd.read_csv('forecast_13_weeks.csv')
record = {
    'run_at': datetime.now(timezone.utc).isoformat(timespec='seconds'),
    'model_version': 'cash-forecast v1.3',
    'library': f'prophet {prophet.__version__}',
    'input_file': 'treasury_cash_flows_clean.csv',
    'input_sha256': sha256('treasury_cash_flows_clean.csv'),
    'parameters': {'changepoint_prior_scale': 0.05, 'regressors': ['is_payroll_week', 'is_tax_week'],
                   'train_weeks': 52, 'one_offs_excluded': [38]},
    'output_file': 'forecast_13_weeks.csv',
    'output_sha256': sha256('forecast_13_weeks.csv'),
    'breach_weeks': int((fc['pessimistic_status'] == 'BREACH').sum()),
    'reviewed_by': None,          # filled in at sign-off, never by the script
    'approved_at': None,
}
with open('audit_log.jsonl', 'a') as f:
    f.write(json.dumps(record) + '\n')
print(json.dumps(record, indent=2))

Retraining: when and how

from prophet import Prophet

# A major customer was won on 2 March 2026: tell the model the trend may change there
m = Prophet(changepoints=['2026-03-02'])

Prophet takes changepoints as a list when you create the model. There is no separate "add changepoint" method (checked on Prophet 1.1.7 and 1.4). Never set and forget: retraining is maintenance, not failure.

Build a monitoring dashboard with vibe coding

The Module 5 method, applied to AI operations. Paste the prompt into Claude (Artifacts) or any AI tool that builds interactive apps, then refine.

Build me an interactive AI Model Monitoring Dashboard as a React artifact.

PURPOSE: A treasury team monitors the performance of its cash flow forecasting model.

DATA: Pre-fill 12 weeks of simulated monitoring data. Each week has: week, forecast_receipts, actual_receipts, data_age_hours, exceptions_count. Make the actuals drift away from the forecast in the last 4 weeks.

CALCULATIONS:
- Weekly error % = |actual - forecast| / actual
- Running 4-week average error
- Bias % = sum(forecast - actual) / sum(actual)
- Status per week: GREEN (error < 7%), AMBER (7-10%), RED (> 10%)

VISUAL:
- Top row: 4 KPI cards (current error, 4-week average, bias, data freshness)
- Error trend line chart with green/amber/red zones
- Forecast vs actual comparison chart
- Data quality table with status indicators
- Alert section: weeks with error > 10% or data older than 24 hours

COLOR: Teal accent. Red/amber/green for status.
EXPORT: A CSV button for the full monitoring log.

Add a banner at the top: 'RETRAIN RECOMMENDED' in red if the error has been above 10% for 3 consecutive weeks. Otherwise 'MODEL HEALTHY' in green. Make the actual values editable so I can test the banner.

The monitor, live

12 weeks of receipts forecast vs actual (EUR K). Edit any actual and watch the KPIs, statuses and banner update.

What it shows: actual receipts fall from week 6 while the forecast keeps rising. That is drift, most likely a customer lost or paying later. The bias confirms it: the model over-forecasts. Diagnose before you retrain.

Microsoft Fabric for treasury AI at scale

When you outgrow Excel + Python + Power BI, Fabric brings it together. You do not need it today. You will when you run several models across many entities with frequent retraining.

What you take away

You started as a user. You finish as a builder.

Ten modules, from where AI works in treasury to building and operating AI systems.