Case 05 Shipped

A migration built from a full inventory

A legacy Kubeflow estate, from 51 pipelines to more than 1,500 registered models, inventoried and scored before anything moved, then migrated to Databricks in waves.

Context
AI platform of a global mobility company, 2026
My part
I audited the legacy estate, wrote the migration plan, shipped the platform groundwork it depends on, built the pilots and ran the move.
Team
Data and ML teams own their pipelines and models. The platform team owns the target platform.
Status
Delivered. The estate now runs on Databricks.

Stack: Kubeflow · GCP · Databricks · Unity Catalog · Asset Bundles · MLflow · Delta Sharing · Harbor · Vault · Trivy · release-please · Terragrunt

  • 51 Kubeflow pipelines inventoried and scored Measured
  • ~25% of the estate retired instead of moved Shipped
  • 141 catalogs moved to a new data-governance group through reviewed changes Shipped
On this page

Context

The platform’s ML workloads ran on a self-managed Kubeflow estate in GCP, and the target was Databricks on AWS. I audited the legacy estate, wrote the migration plan, built the platform groundwork it depended on, and ran the move in waves.

Problem

A migration plan is as good as its inventory. Nobody had a complete list of what ran on Kubeflow, what each pipeline depended on, or which of the registered models still mattered.

Constraints

  • Two clouds during the move: GCP as the source, AWS as the target.
  • Services load models by name, so model references had to survive the move.
  • Teams in several workspaces needed the same data.

Options

For each asset there were two paths: migrate it to the target platform, or retire it. The complexity score set the order and the effort.

Engineering decisions

  1. Score every asset for migration complexity.
  2. Map Kubeflow Pipelines primitives to Databricks Jobs and Asset Bundles.
  3. Move more than 1,500 registered MLflow models to Unity Catalog model aliases, one mapping per model family, with Delta Sharing across workspaces.
  4. Retire about a quarter of the estate instead of moving it.
  1. Inventory
    • 51 Kubeflow pipelines
    • 41 image directories, about 22 unique
    • 25 package directories, about 22 unique
    • 1,500+ registered models, mostly per-city variants
  2. Score
    • Migration complexity one score per asset
  3. Decide
    • Migrate mapped to the target platform
    • Decommission about 25% of the estate
  4. Map
    • Databricks Jobs and Asset Bundles from Kubeflow Pipelines primitives
    • Unity Catalog model aliases from MLflow models
    • Delta Sharing data across workspaces
From inventory to decisions. Every asset gets a score and a path before a pipeline moves.Plan view. Counts come from the inventory. The figure does not show the migration as done.

Implementation

The plan needed a target platform that teams could trust, so the groundwork shipped first.

Custom containers for Databricks jobs. A cluster policy accepts images from the corporate registry and nothing else, and registry credentials stay hidden from users. A versioned base-image line uses release-please for versions and Trivy for scanning, and a self-service guide walks teams through the flow.

Unity Catalog governance. I extended the account group model and created the production data-governance admin group. Reviewed Terragrunt changes moved ownership of 141 catalogs, across Unity Catalog, BigQuery federation and Delta Sharing, to the new group, with declarative import blocks bringing drifted resources under code. An artifact-allowlist module with input validation and destroy guards runs on five workspaces, and a governance watchdog detects access and ownership drift in dev, stage and prod, and corrects it in dev.

Validation

I built the first migration pilots as Databricks Asset Bundles. One of them collapsed 75 per-city Kubeflow jobs into one parameterised job. The pilots proved the mappings on real pipelines, and the complexity scores then set the order of the waves that followed.

Outcome

  • MeasuredAn inventory of 51 Kubeflow pipelines and the images and packages behind them (about 22 unique of each), every asset scored for migration complexity.
  • ShippedKubeflow Pipelines primitives moved to Databricks Jobs and Asset Bundles.
  • ShippedMore than 1,500 registered MLflow models moved, family by family, to Unity Catalog model aliases.
  • ShippedAbout 25% of the estate retired instead of moved.
  • ShippedThe first migration pilots as Asset Bundles, with 75 per-city jobs collapsed into one parameterised job.
  • ShippedCustom-container support with a Harbor-only policy, Vault-injected credentials and scanned base images.
  • ShippedCatalog ownership moved to a data-governance group through reviewed changes, an artifact allowlist on five workspaces, and a governance watchdog.