Pipewright - Governed MLOps on Google Cloud by Fluxor Software

For every team

One platform. Four people who need it to work.

Clear ownership is the point: the platform team runs the cloud, ML teams own their models, approvers decide, auditors read the record. Nobody does someone else's job.

Describe the model. Skip the plumbing.

A use case is a folder: your pipeline code, plus YAML for data, training, gates and monitoring. Pipewright does the rest.

Validate offline before anything costs money
Release in one command: build, pin, verify, go live
Champion comparison and gates on every training run
Drift and performance monitoring with email alerts
Features by name, point in time: from feature tables or Feature Store groups, never from the future
Ray when a step needs a cluster: one per run, made and removed for you

usecases/churn/config/evaluation.yaml

evaluation:
  primary_metric: auc
  gates:
    - { metric: auc, min: 0.75 }
    - { metric: calibration_error, max: 0.05 }
  champion: { enabled: true, min_improvement: 0.005 }
  slices: [contract_length_months]

$ pipewright release --env prod --usecase churn
verified: contract v3 is valid
verified: training and inference templates match
live: churn -> 67d2e88a

One environment for every team, from one file.

A Terraform blueprint stands up a private environment; teams join by name. One shared launcher serves them all, each with its own identity and data.

CMEK everywhere, a key per service, rotated
No public endpoints: internal launcher, Private Service Connect
Guardrails on: VPC Service Controls and organization policies
CI without keys: GitHub Actions through Workload Identity Federation, let in by identity
Policies per environment: manual promotion, required approvers
Feature access per team: the datasets each team may read, granted, never copied

platform.yaml

spec:
  blueprint: private-managed
  region: europe-west2
  policies:
    promote_model: manual
    retraining: { required_approvers: [model-risk] }
  approver_groups:
    model-risk: { members: [group:model-risk@yourco.com] }
  tenants:
    retention: { landing_prefix: incoming/retention/ }
    fraud:     { landing_prefix: incoming/fraud/ }
  guardrails:
    vpc_sc: { mode: enforce }

Decide with the facts, under your own name.

When a model drifts you get a short decision pack by email. The email never approves anything: you decide with your own identity, and the record keeps your reason.

Is it real? PSI per feature, the trend, live performance
Is it the data? Data checks on the latest batches
Quorum: one approval per required group, never the requester
Snoozes expire, and cases re-open when things get worse

pipewright retrain approve --usecase churn

Case rc-20261007-c3d3ac: churn (prod), drift
Drifted: tenure_months 3.035, monthly_charges 0.853
Data checks passed. Champion v1, 1 day old.
Approvers: ml-leads (done), model-risk

$ pipewright retrain approve --case rc-20261007-c3d3ac \
    --reason "real shift after the migration"
approve sent as approver group model-risk
recorded: case is approved
training submitted on the chosen window

The record, not a reconstruction.

Every approval and deployment produces an evidence pack: what was trained, on what, how it was judged, who approved it, how it runs and how it is watched. Hashed, and stored write-once.

Fifteen controls, each with the facts it rests on
Gaps are reported, never hidden
SHA-256 for every file; tampering shows
Retention locks for years, in your project

gs://.../evidence/packs/churn/v4/

evidence.json       controls, lineage, approvals
report.md           for people
contract.json       the release that trained it
model_card.md       intended use, limits
evaluation.json     metrics, gates, champion
events.json         every lifecycle event
audit.json          who changed or used it
SHA256SUMS          written last

12 pass · 1 info · 2 n/a · 0 gaps

The platform

Built for teams who answer to someone.

The core does the MLOps. Enterprise adds what regulated customers need around it: private networking, audit evidence, and Fluxor beside you.

Enterprise

Private by default

The private-managed blueprint stands up an environment with every guardrail on. Pipelines, models and endpoints all run in your own project, encrypted with your keys.

CMEK per serviceno public IPsinternal launcherVPC-SCorg policiesCMEK logsRay over PSC

Core

A person decides

Drift opens a decision case, not a retrain. Training starts only when every required group has approved.

ml-leadsapproved

model-riskapproved

same person twicerefused

requestercannot approve

snoozed case, worsere-opened

Core

Metadata-driven

Contract, features, training, gates, triggers and monitoring declared, and checked against each other. Features are named, not queried.

Core

Replayable releases

Write-once contracts, digests everywhere, rollback in one command.

Core

Events your tools can use

Lifecycle and decision events on Pub/Sub and in structured logs: route them to Slack, PagerDuty, a SIEM, or your own apps.

Enterprise

Evidence packs

Produced automatically on every approval and deployment, and kept write-once for as long as your regulator asks.

What approvers receive

A decision pack in the inbox. The decision stays with a person.

When drift or a performance drop opens a case, the people your team names get the facts in a few lines: what moved, by how much, the trend, the data checks, the champion and what retraining would use. Every step after that is sent again, so nothing is decided in silence.

@Email to the addresses in retraining.decisions.notify

⇄Pub/Sub feed for your own apps, opt-in per use case

!Escalation when a case waits too long, and a weekly digest

Inbox · ml-team@yourco.com

[prod] churn: retrain decision needed (rc-20261007-c3d3ac)

InboxML decisionsAction needed

Cloud Monitoring <alerts@monitoring.example>to ml-team, model-risk

Tue 21:17

churn (prod) drifted, and a person needs to decide what happens next. Nothing will retrain until the required groups approve.

Drifted	tenure_months 3.035, monthly_charges 0.853
Predictions	PSI 1.253
Trend	worst PSI 0.01, 1.04, 3.03 over recent runs
Performance	not judged yet: 0 labelled rows
Data checks	passed: 1,000 rows, most nulls total_charges 1.5%
Champion	version 1, trained 2026-10-06
Approvers	ml-leads, model-risk

Review the full decision pack, then decide with your own identity:

pipewright retrain show --env prod --usecase churn --case rc-20261007-c3d3ac View case rc-20261007-c3d3ac

This email never approves anything. Approve, decline or snooze with pipewright retrain; every decision is recorded with your name and reason. You receive this because you are in retraining.decisions.notify for churn.

Evidence

Fifteen questions, answered for every model.

Each control states what was checked and the facts it rests on: the job, the key, the identity, the approval, the decision. When a source cannot be read, the pack says so.

See a sample pack

churn-classifier v4

12 pass1 info2 n/a

EV-01Model artifacts encrypted with a customer-managed keypass

EV-02Training ran encrypted, in the private networkpass

EV-03Training ran as the tenant's runner identitypass

EV-04Trained from a released, pinned contractpass

EV-05Evaluated against acceptance criteria and the championpass

EV-06Promotion to the default version is approvedpass

EV-07Deployment recorded and encryptedpass

EV-08Monitored for drift and performancepass

EV-09Key enabled and rotated at least every 90 dayspass

EV-10Changes to the model and its endpoints are audit-loggedpass

EV-11Data subject erasure requests handledinfo

EV-12Every drift alert has a recorded human decisionn/a

EV-13Ray clusters ran encrypted, in the private network, as the runnerpass

EV-14Features came from the team's own feature datasets, as of each row's cut-offpass

EV-15A promoted model is the tested, approved model, unchangedn/a

Promotion

What you tested is what ships. Byte for byte.

pipewright promote carries a tested release, and the model you approved, from test to production. Nothing is rebuilt. Every artifact passes four checks at the border, and anything that differs from what you tested is turned back.

component image test sha256:6d978a1f617ef908a46d2703fe48bf51… prod sha256:6d978a1f617ef908a46d2703fe48bf51… identical model files test 6 of 6, the approved v5 prod 6 of 6, unchanged since promotion (EV-15) verified

Turned back at the border (real refusals from our tests)

✕ A pipeline that compiles differentlytraining: container.resources only in the destination. Nothing was promoted.

✕ A model nobody approvedv6 is not the default (approved) version in test: approve it there first.

✕ A setting production forbidsapprovals.promote_model: auto is not allowed in prod (platform policy)

Fleet View · Enterprise

Every model, every environment. One read-only screen.

Platform teams, ML leads and model risk see the whole fleet at once: what is live where, which models wait for a decision, what drifted, what failed, and whether every control holds. Fleet View changes nothing; it shows what the platform already records.

test, prod, …→collector, hourly, read-only→snapshots: metadata only→Fleet View behind Identity-Aware Proxy

https://fleet.acme.example/

IAP

Fleet View, the Drift and decisions page

Fleet. Each use case across test and production: the live release, the champion, what waits for approval, drift, and how far production is behind.Full size ↗

Reads, never writesReleases, approvals and rollbacks stay in the CLI and CI. Fleet View cannot change a thing.

Metadata, never dataNames, versions, digests, states and drift scores. No row of your data is ever collected.

Your identity, your teamsSign-in through Identity-Aware Proxy. Each viewer sees only their teams, and every page view is logged.

Inside your cloudRuns in your own admin project, with your keys, beside the environments it watches.

Editions

One core. Enterprise where it counts.

Pipewright is available today through early access, running on your own Google Cloud with Fluxor alongside you. The core will be released as open source later. Enterprise adds private networking, audit evidence and support from Fluxor.

	CoreOpen source later	Enterpriseby Fluxor Software
CLI, configuration model and examples	✓	✓
Shared launcher and write-once release contracts	✓	✓
Monitoring, alerts and lifecycle events	✓	✓
Retrain decisions with approver quorum	✓	✓
Ray on Agent Platform steps: a cluster per run, within platform limits	✓	✓
Promotion between environments: the tested release and approved model, by digest	✓	✓
Features from feature tables and Feature Store groups, point in time	✓	✓
Private-managed blueprint (CMEK, VPC-SC, internal launcher)	-	✓
Approver groups, decision feeds and policy limits, provisioned	-	✓
Ray provisioned: Private Service Connect, CMEK and the identities, from one switch	-	✓
Feature governance: datasets granted per team, features required, evidence of every read	-	✓
Promotion across private projects: grants, perimeter rules, evidence it arrived unchanged	-	✓
Keyless CI into private environments: Workload Identity Federation, admitted through the perimeter by identity	-	✓
Evidence packs and the evidence worker	-	✓
Fleet View: every environment, team and model in one read-only view	-	✓
SIEM and observability exports	-	✓
Support, onboarding and platform build-outs	Fluxor (early access)	Fluxor

FAQ

Questions teams ask first.

Do we have to rewrite our pipelines?

No. Pipelines stay plain Kubeflow Pipelines code. Pipewright compiles them, supplies the parameters, identities and outputs, and releases them. The examples show the conventions.

Does our data leave our Google Cloud project?

No. Pipewright runs inside your project: pipelines, models, endpoints, monitoring and evidence all stay there, encrypted with your keys. In the private blueprint nothing is reachable from the internet.

Will it retrain models on its own?

Not unless you ask it to. By default drift opens a decision case for people to approve, decline or snooze. A "candidate" mode trains a challenger automatically but never promotes it without a person.

How does a model reach production?

By promotion, not by building it again. pipewright promote takes the release you tested, and the model you approved, to production: images copied by their digest, pipelines checked against the tested ones, the model's files verified one by one. Production's own policies apply first, and a production approver still decides when the model starts serving. Anything that differs from what was tested is refused, and every promotion is recorded and can be rolled back without rebuilding.

Does it work with a feature store?

Yes. A use case names the features it needs, from BigQuery feature tables or groups registered in the Agent Platform Feature Store, and Pipewright reads each value as it was at the row's cut-off, so nothing from the future reaches training. Batch scoring reads features the same way. Your features stay with their owners: Pipewright copies nothing, and your platform team decides which feature datasets each team may read. Every model's evidence records exactly which features it trained on.

Can we use Ray?

Yes, in the environments where you train. A step asks for a Ray on Agent Platform cluster in its YAML and gets one for that run only, sized within limits your platform team sets. Pipewright creates it with your keys and network, runs the step's Ray job on it and deletes it afterwards, so nothing is left running. Ray needs the VPC Service Controls perimeter in dry run, so it runs in development and test, while production keeps its perimeter enforced and receives the models you trained there by promotion. While developing, data scientists keep their own standalone clusters for notebooks; pipelines never depend on them.

Can we see every model across environments in one place?

Yes, with Fleet View in Enterprise. One read-only screen shows every use case in every environment: the live release and its digest, the champion and the candidates waiting, run failures, drift and retrain decisions, the evidence controls and each environment's security posture. A collector reads metadata only, never your data, and each viewer signs in with their Google identity and sees only their own teams.

Is Pipewright open source?

Not yet. Today Pipewright is available through early access, set up on your own Google Cloud with Fluxor. The core will be released as open source later; Enterprise features stay with Fluxor.

Which clouds does it support?

Google Cloud only, on Agent Platform Pipelines today. A GKE-based blueprint with Kubeflow Pipelines is planned.

How do approvers prove who they are?

With their own Google identity. Approver groups are registered by the platform team; membership is checked by Google Cloud IAM, and every decision is recorded with the person's name and reason.

Governed machine learning,
described in YAML.

Configure

Release

Run

Watch

Decide

Prove

One platform. Four people who need it to work.

Describe the model. Skip the plumbing.

One environment for every team, from one file.

Decide with the facts, under your own name.

The record, not a reconstruction.

Built for teams who answer to someone.

Private by default

A person decides

Metadata-driven

Replayable releases

Events your tools can use

Evidence packs

A decision pack in the inbox. The decision stays with a person.

Fifteen questions, answered for every model.

What you tested is what ships. Byte for byte.

Turned back at the border (real refusals from our tests)

Every model, every environment. One read-only screen.

One core. Enterprise where it counts.

Questions teams ask first.

See your first use case go live.