The metadata-driven pipeline decisions I would revisit before moving ADF patterns into Fabric Data Factory

The expensive version of an ADF-to-Fabric migration is the one where the control table keeps working just long enough to hide what changed.

A metadata-driven ADF framework can reduce duplicated pipelines. It can also turn source selection, connection behavior, dataset shape, schedules, validation, and ownership into a second programming language. When that happens, the migration plan starts preserving the abstraction before the team has decided whether the abstraction is still honest.

My thesis is simple: keep metadata for durable operating decisions, and delete or make explicit the ADF-specific indirection before rebuilding the pattern in Fabric Data Factory.

Problem

The migration trap is treating a metadata-driven ADF framework as one reusable asset.

In ADF, a team may have used one control table to drive linked services, parameterized datasets, target paths, table names, watermarks, schedules, and validation rules through a reusable copy pipeline. That pattern can be useful. It keeps repeated pipeline objects under control.

It also makes some decisions too easy to miss. If server_name, database_name, dataset_name, and schedule_name are just fields in a row, the team may forget that Fabric Data Factory does not model those pieces the same way.

As of my 2026-04-27 source check, Microsoft’s migration planning guidance says Fabric Data Factory defines dataset properties inline within activities, replaces linked services with Fabric connections, uses variable libraries instead of ADF global parameters, and handles scheduling differently from ADF. The same guidance says manual migration is necessary for complex environments and low-parity patterns (Microsoft Learn, migration planning, last updated 2026-04-11).

That does not make metadata-driven pipelines bad. It means I would stop asking, “Can we migrate the framework?” and start asking, “Which fields still describe how we operate this data path?”

Default approach

My default is to turn one old ADF control-table row into a migration decision record before rebuilding anything in Fabric.

• Inventory each field by job: source identity, connection behavior, dataset or path behavior, load pattern, target shape, schedule, validation, ownership, and migration status.
• Keep metadata that the team still needs to review deliberately: source identity, load pattern, watermark policy, validation evidence, owner, escalation path, and migration status.
• Move platform-specific choices into Fabric-owned artifacts or documented conventions: connections, inline activity settings, variable libraries, workspace rules, and deployment mappings.
• Choose the target shape before preserving the orchestration pattern. The path might land in a Lakehouse table, feed a Warehouse table, use Copy job, call a notebook for merge behavior, or stay in ADF for now.
• Rebuild schedules and validation as production controls, not as leftovers from the old trigger framework.
• Give each pipeline a migration status such as migrate, needs-review, redesign-required, keep-mounted-adf-for-now, or delete-abstraction.

The source notes matter because this is platform behavior, not a timeless design law. As of 2026-04-27, Microsoft’s ADF upgrade page categorizes pipelines as Ready, Needs review, Coming soon, or Not compatible. It also says dynamic linked services, highly dynamic linked-service patterns, and dataset-driven metadata patterns cannot migrate as-is through the UX-based path (Microsoft Learn, upgrade ADF pipelines to Fabric, last updated 2026-04-21). The comparison page frames Fabric as connection-based and dataset-free, with data properties defined inline in activities (Microsoft Learn, ADF/Fabric differences, last updated 2026-03-31).

So the decision is not “metadata or no metadata.” The decision is where the metadata earns its keep.

Example

Here is the kind of ADF control-table row I would not migrate blindly.

ADF control-table row before migration
source_system: erp_sql
linked_service_name: ls_sql_dynamic
server_name: erp-prod-sql-01
database_name: operations
schema_name: dbo
table_name: PurchaseOrders
dataset_name: ds_sql_table_dynamic
target_path: raw/erp/purchase_orders/
watermark_column: LastModifiedUtc
load_type: incremental
schedule_name: daily_0500_eastern
validation_rule: row_count_plus_watermark
owner_team: analytics_platform

That row mixes four different jobs.

First, it names the data path: source_system, schema_name, table_name, and watermark_column tell me what source is moving and how change is detected. Those fields still belong in a migration discussion.

Second, it hides connection behavior: linked_service_name, server_name, and database_name may have been useful ADF indirection, but I would not carry them forward as runtime-switchable strings unless the team can explain the operational reason. In Fabric, I want the governed connection reference to be visible.

Third, it blurs target shape: target_path says where raw data landed in the old pattern, but it does not answer whether the Fabric target is a Lakehouse table, a Warehouse table, both, or neither.

Fourth, it under-specifies production ownership. validation_rule and owner_team are a start, but I also want the failed-run owner, source-contract owner, reporting sign-off owner, rerun note, and escalation path. Migration is where I would make those fields boring and explicit.

As of 2026-04-27, Microsoft’s global-parameter migration guide says ADF global parameters need manual steps, expression references must be updated, and workspace variables should not be overloaded with run-time values (Microsoft Learn, global parameters to variable libraries, last updated 2026-04-11). That is the kind of boundary I want the record to expose, not hide.

The Fabric-ready record is less clever and more operational.

Fabric migration decision record
source_identity: erp_sql / PurchaseOrders
fabric_connection_reference: ops-sql-prod-connection
connection_strategy: explicit connection per governed source, not dynamic server/database strings
load_pattern: incremental copy to Lakehouse staging, with notebook or Warehouse merge only where needed
target_shape: Bronze Lakehouse table -> curated Warehouse table if reporting needs SQL serving
watermark_policy: LastModifiedUtc plus replay/backfill note
schedule_policy: pipeline-local schedule; no reusable central trigger assumption
validation_evidence: row count, max watermark, schema drift check, failed-run owner, rerun note
owner: analytics_platform owns orchestration; source owner owns source contract; reporting owner signs off curated output
escalation_path: source connectivity -> platform; source data change -> source owner; reporting mismatch -> analytics owner
migration_status: redesign-required because dynamic linked-service and dataset-driven pattern cannot migrate as-is

The new record does not pretend every old field deserves a new home.

It keeps source_identity, load_pattern, watermark_policy, validation_evidence, owner, and escalation_path because those decisions survive the platform move. It moves connection choice into an explicit governed Fabric connection reference. It makes target shape a first-class decision instead of burying the answer in a path string. It records schedule policy and migration status so no one confuses a successful assessment with production readiness.

That last point matters. As of my 2026-04-27 source check, Microsoft’s upgrade guidance says post-migration work includes validating connections, re-enabling and configuring triggers, running end-to-end tests, and validating in nonproduction before production cutover (Microsoft Learn, upgrade ADF pipelines to Fabric, last updated 2026-04-21). Microsoft’s global-parameter guide separately says ADF global parameters are not automatically migrated and need deliberate re-authoring into variable libraries (Microsoft Learn, global parameters to variable libraries, last updated 2026-04-11). I would rather keep those checks on the decision record than treat migration as proof that the pipeline is safe.

Tradeoffs

• Default: Keep a control table for source identity, load pattern, watermark policy, validation evidence, owner, escalation path, and migration status. Breaks when: the table starts storing every platform-specific knob. Mitigation: move connection, schedule, deployment, and target-shape conventions into Fabric-owned artifacts or documented workspace rules.
• Default: Use explicit Fabric connections for governed sources. Breaks when: the old ADF pattern relied on parameterized linked services to swap server, database, or authentication context at runtime. Mitigation: create separate governed connection references and keep the source-selection decision visible in metadata instead of hiding it inside dynamic connection strings.
• Default: Define file, table, schema, and copy settings close to the Fabric activity that uses them. Breaks when: the team tries to recreate ADF reusable dataset objects as a parallel metadata language. Mitigation: keep reusable naming conventions, but let Fabric inline properties and activities own the concrete shape.
• Default: Use variable libraries for environment constants and pipeline parameters for run-specific values. Breaks when: old global parameters are treated as an automatic migration. Mitigation: record each global parameter, decide whether it is an environment constant or a runtime decision, and rewrite references deliberately.
• Default: Keep scheduling boring and pipeline-local unless there is a strong reason to centralize orchestration elsewhere. Breaks when: the ADF framework depended on reusable triggers, tumbling windows, dependency triggers, or backfill semantics. Mitigation: rebuild schedules, backfills, and trigger metadata explicitly after assessment rather than assuming parity.
• Default: Validate after migration with row counts, watermarks, schema checks, owner review, and a rerun note. Breaks when: the migration assessment says a pipeline is supported and the team treats that as production proof. Mitigation: run nonproduction end-to-end validation and keep the result in the migration decision record.

Close

Next step: Take one metadata-driven ADF control-table row and mark every field as keep, make explicit in Fabric, or delete; only then decide whether the pipeline should migrate, be redesigned, stay mounted for now, or be retired.

If the row cannot name the owner, validation evidence, target shape, and migration status, I would fix the row before I trusted the migration plan.